A novel formalism for the evaluation of the Casimir-Polder potential in an arbitrary gauge of vector potentials is introduced. The ground state energy of a neutral atom in the presence of an infinite two-dimensional plane with Chern-Simons interaction is derived at zero temperature. The essential feature of the result is its dependence on the antisymmetric part of a dipole moment correlation function.