In this paper we introduce a new notion of a symmetry group of an infinite word. Given a subgroup G_n of the symmetric group S_n, it acts on the set of finite words of length n by permutation. For each n, a symmetry group of an infinite word w is a subgroup G_n of the symmetric group S_n such that g(v) is a factor of w for each permutation g∈ G_n and each factor v of w. We study general properties of symmetry groups of infinite words and characterize symmetry groups of several families of infinite words. We show that symmetry groups of Sturmian words and more generally Arnoux-Rauzy words are of order two for large enough n; on the other hand, symmetry groups of certain Toeplitz words have exponential growth.