The change point detection problem in the time series arises in a wide variety of fields. In some situations, we lack the necessary resources to apply complex techniques, so lightweight approaches are needed.In this study, we consider lightweight approaches to the change point detection problem, namely, classification-based barrier methods. We want to investigate the use of various classifiers and classification evaluation metrics for change point detection, so we are creating a framework that makes it easy to build methods from different components. To study a large number of constructed methods, we create a flexible benchmarking system that allows one to evaluate methods using different metrics.We conduct an empirical study of the methods, present the results and compare them with existing methods and with each other. Our implementation of the KNN based method shows high-quality results. However, we see potential in using at least one more tested classifier as well.