© Published under licence by IOP Publishing Ltd. The nonlinear features of the vibrational motion of a single dust particle trapped in a standing strip are investigated. The excitation of the nonlinear oscillations with the large amplitude of the order of 1.5 mm is due to the periodic movement of the standing striation caused by the square-wave modulation of the discharge current. The frequency responses are investigated depending on value of the modulation depth. The anharmonic effects of the dust particle oscillations such as parametric instabilities and hysteresis are obtained. The theory of the anharmonic oscillator provides a good quantitative description of the experimental data. The values of the thresholds of excitation of parametric instabilities, the anharmonic coefficients and the critical values of the oscillation amplitude for the hysteresis are calculated. The potential energy curve of the single dust particle trapped in the striation is calculated using the values of anharmonic coefficients.