In a Hilbert space H \mathfrak {H} , a family of selfadjoint operators (a quadratic operator pencil) A ( t ) A(t) , t ∈ R t\in \mathbb {R} , of the form A ( t ) = X ( t ) ∗ X ( t ) A(t) = X(t)^* X(t) is considered, where X ( t ) = X 0 + t X 1 X(t) = X_0 + t X_1 . It is assumed that the point λ 0 = 0 \lambda _0=0 is an isolated eigenvalue of finite multiplicity for the operator A ( 0 ) A(0) . Let F ( t ) F(t) be the spectral projection of the operator A ( t ) A(t) for the interval [ 0 , δ ] [0,\delta ] . By using approximations for F ( t ) F(t) and A ( t ) F ( t ) A(t)F(t) for | t | ≤ t 0 |t| \leq t_0 (the so-called threshold approximations), approximations in the operator norm on H \mathfrak {H} are obtained for the operators cos ⁡ ( τ A ( t ) 1 / 2 ) \cos ( \tau A(t)^{1/2}) and A ( t ) − 1 / 2 sin ⁡ ( τ A ( t ) 1 / 2 ) A(t)^{-1/2}\sin ( \tau A(t)^{1/2}) , τ ∈ R \tau \in \mathbb {R} . The numbers δ \delta and t 0 t_0 are controlled explicitly. The next object of study is the behavior for small ε > 0 \varepsilon >0 of the operators cos ⁡ ( ε − 1 τ A ( t ) 1 / 2 ) \cos ( \varepsilon ^{-1} \tau A(t)^{1/2}) and A ( t ) − 1 / 2 sin ⁡ ( ε − 1 τ A ( t ) 1 / 2 ) A(t)^{-1/2}\sin (\varepsilon ^{-1}\tau A(t)^{1/2}) multiplied by the “smoothing factor” ε q ( t 2 + ε 2 ) − q / 2 \varepsilon ^q (t^2 + \varepsilon ^2)^{-q/2} with a suitable q > 0 q>0 . The obtained approximations are given in terms of the spectral characteristics of the operator A ( t ) A(t) near the lower edge of the spectrum. The results are aimed at application to homogenization of hyperbolic equations with periodic rapidly oscillating coefficients.