Despite superior properties of ultrafine-grained (UFG) materials processed by severe plastic deformation (SPD), their thermal stability is a concern because of the supersaturated fractions of lattice defects. In this study, the microstructural stability of various UFG materials (2 alloys and 15 pure metals) after SPD processing through the high-pressure torsion (HPT) were investigated at room temperature for up to 10 years. While most of the metals with high melting temperatures remained stable, a softening by self-annealing occurred in pure silver, gold and copper (with moderate melting temperatures), and an unusual hardening occurred in pure magnesium, Al-Zn alloy and Mg-Li alloy (with low melting temperatures). These softening/hardening behaviors by grain coarsening were attributed to the contribution of grain boundaries to dislocation activity or grain-boundary sliding, respectively. It was shown that the self-annealing was accelerated by increasing the processing pressure and strain and by decreasing the processing temperature and stacking fault energy, due to the enhancement of stored energy and/or atomic mobility.