Quantum memories can be used not only for storage of quantum information but also for a substantial manipulation of ensembles of quantum states. Therefore, the speed of such manipulation and the ability to write and retrieve signals of relatively short duration become important. One approach towards enhancing the performance of a quantum memory is to combine an active medium with an optical cavity. Previous works investigating cavity-enhanced memories concentrated on noise processes in the bad cavity limit, that is, for signals that are much longer than the cavity field lifetime. In this work we investigate four-wave mixing noise that arises from the retrieval of relatively short signals from cavity-assisted memories, thus complementing recent works by other authors. We propose an approach that allows one to account for noise sources of different frequencies and different physical origin by using two-band spectral filtering of the noise sources in the Heisenberg-Langevin picture. We demonstrate that in these spectrally selective memories the sideband atomic noise sources contribute to the four-wave mixing noise on par with the sideband quantized field entering the cavity.