Narrow-gap semiconductors with strong spin-orbit coupling such as bismuth tellurohalides have become popular candidates for spintronic applications. But driving spin-polarized photocurrents in these materials with circularly polarized light requires picosecond lifetimes of the photoexcited carriers and low spin-flip scattering rates. In search of these essential ingredients, we conducted an extensive study of the carrier dynamics on the Te-terminated surface of BiTeI, which exhibits a giant Rashba splitting of both surface and bulk states. We observe a complex interplay of surface and bulk dynamics after photoexcitation. Carriers are rapidly rearranged in momentum space by quasielastic phonon and defect scattering, while a phonon bottleneck leads to a slow equilibration between bulk electrons and lattice. The particular band dispersion opens an inelastic decay channel for hot carriers in the form of plasmon excitations, which are immanent to Rashba-split systems. These ultrafast scattering processes effectively redistribute excited carriers in momentum and energy space and thereby inhibit spin-polarized photocurrents.