Current evidence suggests that severity and mortality of COVID-19 is higher in men than in women, whereas women might be at increased risk of COVID-19 reinfection and development of long COVID. Differences between sexes have been observed in other infectious diseases and in the response to vaccines. Sex-specific expression patterns of proteins mediating virus binding and entry, and divergent reactions of the immune and endocrine system, in particular the hypothalamic–pituitary–adrenal axis, in response to acute stress might explain the higher severity of COVID-19 in men. In this Personal View, we discuss how sex hormones, comorbidities, and the sex chromosome complement influence these mechanisms in the context of COVID-19. Due to its role in the severity and progression of SARS-CoV-2 infections, we argue that sexual dimorphism has potential implications for disease treatment, public health measures, and follow-up of patients predisposed to the development of long COVID. We suggest that sex differences could be considered in future pandemic surveillance and treatment of patients with COVID-19 to help to achieve better disease stratification and improved outcomes.