The paper discusses the connection of Taylor maps and polynomial neural networks (PNN) for numerical solving of the ordinary differential equations (ODEs). Having the system of ODEs, it is possible to calculate weights of PNN that simulates the dynamics of these equations. It is shown that proposed PNN architecture can provide better accuracy with less computational time in comparison with traditional numerical solvers. Moreover, neural network derived from the ODEs can be used for simulation of system dynamics with different initial conditions, but without training procedure. Besides, if the equations are unknown, the weights of the PNN can be fitted in a data-driven way. In the paper, we describe the connection of PNN with differential equations theoretically along with the examples for both dynamics simulation and learning with data.