Introducing magnetic exchange interaction into topological insulators is known to break the time-reversal symmetry and to open a gap at the Dirac point in the otherwise gapless topological surface states. This allows various novel topological quantum phenomena to be attained, including the quantum anomalous Hall effect and can lead to the emergence of the axion insulator phase. Among the different approaches, magnetic doping is an effective, but still experimentally challenging pathway to provide the magnetic exchange interaction. Here we demonstrate that epitaxial deposition of Co and Mn magnetic atoms onto the (0001) surface of the BiSbTeSe2 topological insulator with a coverage between 0.6 and 3 atoms per surface cell performed in a finely tuned temperature range of 300°-330°C leads to the substitution of pnictogen atoms in the surface layer with magnetic atoms and to the formation of a two-dimensional magnetic phase with out-of-plane magnetization as proved by SQUID magnetometry. This magnetic layer is responsible for the appearance of a gap in the Dirac surface state as revealed by laser-based microfocused angle-resolved photoelectron spectroscopy. Our measurements have shown that the gap exists within the temperature range of 15-100 K, where the out-of-plane magnetization persists. The presented experimental results are supported by relativistic ab initio calculations.