Quantum efficiency is an important characteristic of quantum memory devices that are aimed at recording the quantum state of light signals and its storing and reading. In the case of memory based on an ensemble of cold atoms placed in an optical cavity, the efficiency is restricted, in particular, by relaxation processes in the system of active atomic levels. We show how the effect of the relaxation on the quantum efficiency can be determined in a regime of the memory usage in which the evolution of signals in time is not arbitrarily slow on the scale of the field lifetime in the cavity and when the frequently used approximation of the adiabatic elimination of the quantized cavity mode field cannot be applied. Taking into account the effect of the nonadiabaticity on the memory quality is of interest in view of the fact that, in order to increase the field–medium coupling parameter, a higher cavity quality factor is required, whereas storing and processing of sequences of many signals in the memory implies that their duration is reduced. We consider the applicability of the well-known efficiency estimates via the system cooperativity parameter and estimate a more general form. In connection with the theoretical description of the memory of the given type, we also discuss qualitative differences in the behavior of a random source introduced into the Heisenberg–Langevin equations for atomic variables in the cases of a large and a small number of atoms.