Influence of annealing on the microstructure and mechanical properties has been studied for Al-Zr (0.4 wt.%) alloy with the ultrafine-grained (UFG) structure formed by high-pressure torsion (HPT) at room temperature. A drastic hardening effect by short-term annealing in the temperature range of 90-280 degrees C was observed for the HPT-processed Al-Zr alloy. The effect of hardening by annealing for the HPT-processed Al-Zr alloy is compared with that for the HPT-processed commercial purity (CP) Al. It was shown that addition of 0.4 wt.% Zr in Al does not cause a significant impact on the magnitude of hardening by annealing up to 150 degrees C, however it leads to a shift of its maximum to higher annealing temperatures and expansion of the thermal stability range of strength up to 280 degrees C. The kinetics of hardening by annealing for CP Al and Al-Zr alloys in the UFG state has been studied for the first time. It was shown that in both materials the strength first increases linearly with the duration of annealing and then reaches saturation. The temperature dependence of the rate of hardening by annealing was analysed through an Arrhenius law, and apparent activation energy was extracted for both systems. The addition of Zr results in the reduction of the activation energy of annealing-induced hardening by similar to 2 times. Possible physical mechanisms controlling the kinetics of hardening by annealing are discussed for the ultrafine-grained CP Al and Al-Zr alloy.