Aims. Sunspot distribution in the northern and southern solar hemispheres exibit striking synchronous behaviour on the scale of a Schwabe cycle. However, sometimes the bilateral symmetry of the Butterfly diagram relative to the solar equatorial plane breaks down. The investigation of this phenomenon is important to explaining the almost-periodic behaviour of solar cycles. Methods. We use cross-recurrence plots for the study of the time-varying phase asymmetry of the northern and southern hemisphere and compare our results with the latitudinal distribution of the sunspots. Results. We observe a long-term persistence of phase leading in one of the hemispheres, which lasts almost 4 solar cycles and probably corresponds to the Gleissberg cycle. Long-term variations in the hemispheric-leading do not demonstrate clear periodicity but are strongly anti-correlated with the long-term variations in the magnetic equator.