The mechanism of formation of the polarimetric signal observed in the spin noise spectroscopy (SNS) is analyzed from the viewpoint of the light scattering theory. A rigorous calculation of the polarimetric signal (Faraday rotation or ellipticity) recorded in the SNS is presented in the approximation of single scattering. We show that it is most correctly to consider this noise as a result of scattering of the probe light beam by fluctuating susceptibility of the medium. Fluctuations of the gyrotropic (antisymmetric) part of the susceptibility tensor lead to appearance of the typical for the SNS Faraday rotation noise at the Larmor frequency. At the same time, fluctuations of linear anisotropy of the medium (symmetric part of the susceptibility tensor) give rise to the ellipticity noise of the probe beam spectrally localized at the double Larmor frequency. The results of the theoretical analysis well agree with the experimental data on the ellipticity noise in cesium vapor.