The notion of parametrically splitting algorithms is introduced, which are characterized by their capability of solving a given problem on a large number of processors with few data transfers. These algorithms include many numerical integration algorithms, Monte Carlo and quasi-Monte Carlo methods, and so on. It is shown that parametrically splitting algorithms include, in particular, stochastic and quasi-stochastic algorithms for solving linear algebraic equations, which are particularly efficient when the number of variables is large. The parametrically splitting version of preconditioning methods is analyzed. The grid analogue of the Laplace equation in the upper relaxation method is considered in detail.