Rayleigh backscattering produced by an incident fundamental mode in a multimode optical fiber is analyzed using a diffraction technique, with a full set of backward-propagating modes being taken into consideration. Explicit formulas are derived for mode excitation efficiency via radial distributions of the mode fields, and it is proved that only half-azimuthal modes are backscattered by the incident wave of a fixed polarization. Advanced analytical expressions are developed for fibers with a quadratic refractive-index profile, and mode groups of even numbers, composed of modes with equal propagation constants, are stated to be excited with equal efficiencies. (C) 2016 Optical Society of America