In this paper mathematical modeling of drillstring dynamics in the hole in drilling simulator is discussed. The longitudinal, torsional and bending column vibrations are described by a system of nonlinear partial differential equations. Model coefficients are set according to received incomplete experimental data. There are different regimes of drilling, depending on the control parameters, namely the top drive rotation speed and force on bit: longtime stopping, uniform and stick-slip rotation.