This paper presents isolated photon-hadron correlations using pp and p-Pb data collected by the ALICE detector at the LHC. For photons with |η|<0.67 and 12<pT<40 GeV/c, the associated yield of charged particles in the range |η|<0.80 and 0.5<pT<10 GeV/c is presented. These momenta are much lower than previous measurements at the LHC. No significant difference between pp and p-Pb is observed, with Pythia 8.2 describing both data sets within uncertainties. This measurement constrains nuclear effects on the parton fragmentation in p-Pb collisions, and provides a benchmark for future studies of Pb-Pb collisions.