It is shown by using the example of solution methods for systems of linear algebraic equations that algorithms possessing the important properties of being parallel and asynchronous can be constructed by reducing the problem under consideration to computing a path integral. Earlier, it was shown by the author and his colleagues that, with increasing the dimension n of the system, parallel and asynchronous Monte Carlo algorithms may become better than the corresponding iterative methods. A qualitative explanation of this phenomenon is suggested. The solution of a system of linear algebraic equations of the form X = AX + F admits a simple representation in form of a path integral only under the condition λ ₁(A) < 1, where λ ₁(A) is an eigenvalue of A with maximum absolute value. Otherwise, a recursive solution procedure with (coarse) parallelism properties can be constructed. However, in this case, additional conditions for synchronizing the algorithm are needed. Finally, it is shown how to obtain efficient analogues of stochastic algorithms (algorithms of the quasi-Monte Carlo method), which exhibit better rate of convergence than those in the Monte Carlo method, on the basis of results obtained by the author and Wagner in [10]. Similar approaches apply to a large class of problems of mathematical and theoretical physics in which integral representations of solutions are known.