We analyze radiation produced by a single TM mode incident on the open end of a cylindrical waveguide with uniform dielectric filling. This open-ended waveguide is placed inside concentric vacuum waveguide with a larger radius. Rigorous theory describing excitation of transmitted and reflected modes in each domain is developed. We also perform direct numerical simulation of described structures using Comsol Multiphysics code and show that the results obtained by these approaches for millimeter-sized dielectric waveguide coincide with less-than-percent accuracy. The analytical approach is more efficient for calculation of mode structure at high frequencies (up to Terahertz) and high permittivity. We also consider the situation where generated radiation is extracted into free space through the open end of the outer waveguide. We calculate radiation patterns in the far-field zone using both our algorithm and direct simulations and show that these results are in very good agreement too. (C) 2017 Elsevier B.V. All rights reserved.