Problem of mathematical modeling high-rate processes in condensed medium is considered by using new nonlocal hydrodynamic approach based on the results of non-equilibrium statistical mechanics and cybernetical physics. Interrelationships between the spatiotemporal correlations in the integral thermodynamic relationships between forces and fluxes and the system internal structure made it possible to describe the self-organization of new dynamic structures in the open system. The temporal structure evolution is described by methods of the control theory of adaptive systems. The proposed approach to the structure evolution allows a new insight into the system state stability. The proposed approach is used to describe high-rate shear flow in the Couette formulation. Explicit approximate solutions to the problem show that steady pure shear flow far from equilibrium looses its stability due to dynamic structure evolution. Near the boundaries there appear layers where continuum mechanics becomes invalid and non-equilibrium interfacial interaction with the walls forms vortex structures. In transient modes a meta-stable state can occur where the system evolution can change its direction due to any weak impact.