In this paper the problem of a multidimensional optimization in unsupervised learning and clustering is studied under significant uncertainties in the data model and measurements of penalty functions. We propose a modified version of SPSA-based algorithm which maintains stability under conditions such as a sparse Gaussian mixture model. This data model is important because it can be effectively used to evaluate the noise model in many practical systems. The proposed algorithm is robust to external disturbances and is able to process data sequentially, “on the fly”. In this paper provides a study of this algorithm and its mathematical justification. The behavior of the algorithm is illustrated by examples of its use for clustering in various difficult conditions.