Semiconductor quantum dots are excellent candidates for ultrafast coherent manipulation of qubits by laser pulses on picosecond timescales or even faster. In inhomogeneous ensembles a macroscopic optical polarization decays rapidly due to dephasing, which, however, is reversible in photon echoes carrying complete information about the coherent ensemble dynamics. Control of the echo emission time is mandatory for applications. Here, we propose a concept to reach this goal. In a two-pulse photon echo sequence, we apply an additional resonant control pulse with multiple of 2π area. Depending on its arrival time, the control slows down dephasing or rephasing of the exciton ensemble during its action. We demonstrate for self-assembled (In,Ga)As quantum dots that the photon echo emission time can be retarded or advanced by up to 5 ps relative to its nominal appearance time without control. This versatile protocol may be used to obtain significantly longer temporal shifts for suitably tailored control pulses.