We consider the mode transformationin a circular waveguide with a transversal boundarybetweena vacuum part and a part with a cylindrical dielectric layer and a vacuum channel. It is assumed that an incident mode can be both propagating and evanescent. Analysis is carried out with the using the mode decomposition technique. Numerical algorithm for calculating the mode transformationat an arbitrary channel radius is also developed. Typical dependences for the reflection and transmission coefficients on the channel radius are presented and discussed.