Background The trial “Stress Echo (SE) 2020” evaluates novel applications of SE beyond coronary artery disease. The aim of the study was control quality and harmonize reading criteria. Methods One reader from 78 centers of the SE 2020 network asked for credentials to read a set of 20 SE video-clips selected by the core lab. All aspiring centers met the pre-requisite of high-volume and the years of experience in SE ranged from 5 to 31 years (mean value 18 years). The diagnostic gold standard was a reading by the core lab. The a priori determined pass threshold was 18/20 (≥ 90%). Results Of the initial 78 who started, 57 completed the first attempt: individual readers' score on first attempt ranged from 07/20 to 20/20 (accuracy from 35% to 100%, mean 78.7 ± 13%) and 44 readers passed it. There was a very poor correlation between years of experience and the reader's score on first attempt (r = − 0.161, p = 0.231). Of the 13 readers who failed the first attempt, 12 took it again after the web-based session and their accuracy improved (74% vs. 96%, p < 0.001). The kappa inter-observer agreement before and after web-based training was 0.59 on first attempt and rose to 0.91 on the last attempt. Conclusions In SE reading, the volume of activity or years of experience is not synonymous with diagnostic quality. Qualitative analysis and operator-dependence can become a limiting weakness in clinical practice, in the absence of strict pathways of learning, credentialing and audit.