Study of drilling systems is a topical problem in drilling industry due to the amount of failures which lead to considerable cost and time loses for mining companies. This paper presents mathematical model of drilling system actuated by induction motor. The model consists of two discs connected by a no-mass string. The upper disc is actuated by the rotor of the motor and the lower disc is connected with a brake device which causes friction torque. Set-valued force laws are used in order to model the friction. The obtained model is described by system of ordinary differential equations (ODE's) with discontinuous right-hand side. A bifurcation analysis of the model is done. Presence of so-called hidden oscillations (represented by stickslip stable vibrations) along with stable equilibrium is demonstrated. Such system requires accurate numerical modeling in the neighborhood of the discontinuous surface. Here the method of modeling such systems based on Filippov definition of solution for systems of ODE's with discontinuous right-hand side is used for analysis of dynamics of the model.