In this paper we introduce a new notion of a sequence of symmetry groups of an infinite word. Given a subgroup of the symmetric group, it acts on the set of finite words of length n by permutation. We associate to an infinite word w a sequence of its symmetry groups: For each n, a symmetry group of w is a subgroup of the symmetric group such that is a factor of w for each permutation and each factor v of length n of w. We study general properties of the symmetry groups of infinite words and characterize the sequences of symmetry groups of several families of infinite words. We show that for each subgroup G of there exists an infinite word w with
. On the other hand, the structure of possible sequences is quite restrictive: we show that they cannot contain for each order n certain cycles, transpositions and some other permutations. The sequences of symmetry groups can also characterize a generalized periodicity property. We prove that symmetry groups of Sturmian words and more generally Arnoux-Rauzy words are of order two for large enough n; on the other hand, symmetry groups of certain Toeplitz words have exponential growth.