We demonstrate theoretically that in gyrotropic semiconductors and semiconductor nanosystems the Brownian motion of electrons results in temporal fluctuations of the polarization plane of light passing through or reflected from the structure, i.e., in stochastic Faraday or Kerr rotation effects. The theory of the effects is developed for a number of prominent gyrotropic systems such as bulk tellurium, ensembles of chiral carbon nanotubes, and GaAs-based quantum wells of different crystallographic orientations. We show that the power spectrum of these fluctuations in thermal equilibrium is proportional to the ac conductivity of the system. We evaluate contributions resulting from the fluctuations of the electric current, as well as of spin, valley polarization, and the spin current to the noise of the Faraday/Kerr rotation. Hence all-optical measurements of the Faraday and Kerr rotation noise provide an access to the transport properties of the semiconductor systems.