The spin fluctuations of electron and hole doped self-assembled quantum dot ensembles are measured optically in the low-intensity limit of a probe laser for absence and presence of longitudinal or transverse magnetic fields. The experimental results are modeled by two complementary approaches based either on a semiclassical or quantum mechanical description. This allows us to characterize the hyperfine interaction of electron and hole spins with the surrounding bath of nuclei on time scales covering several orders of magnitude. Our results demonstrate (i) the intrinsic precession of the electron spin fluctuations around the effective Overhauser field caused by the host lattice nuclear spins, (ii) the comparably long time scales for electron and hole spin decoherence, as well as (iii) the dramatic enhancement of the spin lifetimes induced by a longitudinal magnetic field due to the decoupling of nuclear and charge carrier spins.