In this paper we consider mathematical and computer modeling of nonlinear dynamics of particle beams in cyclic accelerators in terms of the matrix representation of the corresponding nonlinear differential equations. The proposed approach is different from the usual presentations of non-linear equations in the form of Taylor series. In the paper, we use the coefficients representation in the form of two-dimensional matrices. The similar approach allows us not only to significantly reduce the time spent on modeling beam dynamics but use symbolic mathematics to calculate the necessary two-dimensional matrices. This method demonstrates the effectiveness when solving problems of dynamics problems and optimization of control systems, and for evaluating the influence of various effects on the dynamics of the beam (including taking into account the spin). Using the tools of symbolic computations not only significantly increases the computational efficiency of the method, but also allows you to create databases of "ready-made" transformations (Lego-objects), which greatly simplify the process of modeling particle dynamics. Examples of solving practical problems are given.