We consider the mode transformation in a circular waveguide with a transverse boundary between a vacuum part and a part with a cylindrical dielectric layer and a vacuum channel. Two situations are analyzed: an incident mode is launched on the boundary from the dielectric part of the waveguide or from the vacuum one. The incident mode is assumed to be both propagating and evanescent. Analysis is performed using the mode decomposition technique. Approximate solutions are obtained for the cases of the narrow channel and the case of the thin dielectric layer. A numerical algorithm for calculating the mode transformation at an arbitrary channel radius is developed. Typical dependences of the reflection and transmission coefficients on the channel radius are presented and discussed. It is shown, in particular, that the interaction of the evanescent mode with the boundary can result in excitation of the propagating modes in the reflected and transmitted fields.