We report an observation of the optical polarization of hole spins in the negatively charged self-assembled InAs/GaAs quantum dots when the dot is occupied by 2 or 5 electrons on average. It was found that the spin relaxation time of holes is comparable with their lifetime. It was demonstrated that the hole spin polarization can be destroyed by both the transverse magnetic field (the Hanle effect) and the electric bias.