The study focuses on building an informational Russian language chatbot, which aims to answer various possible questions about the inclusion of people with Asperger syndrome. The idea of our project came from the problem of lack of awareness of people about the process of inclusive education and work. We believe that such a problem might cause communication difficulties or even conflicts between pupils, university and college students, or co-workers. A chatbot, which can be customized according to the age of a user, is an alternative informal way to provide information. This format should be suitable for children, teenagers, and young adults, who are not likely to read monotone articles to find out the needed information.

In the study, we implement, evaluate, and then compare two models of transfer learning to find out the most efficient approach to build our chatbot. The goal is to make the chatbot customizable. To reach this, we have decided to use transformer neural network architectures: BERT and GPT-2. Our customization involves three modes according to the age of potential users: younger schoolchildren, teenagers, young adults. A user can choose his or her category manually or use the automatic detection. The detection will be based on supervised learning method: multi-class CNN classification.

OpenAI GPT-2 and BERT by Google are both unsupervised transformer models. BERT is a bidirectional encoder-based pre-trained transformer that was trained with masked language modeling (MLM). MLM might cause difficulties in text generation because masked tokens used for training are conditionally independent in sentence structure. As a result, the input data distribution in the model might not correspond to real-world data. GPT-2 is a decoder-based transformer that uses a self-attention mechanism. The model is considered to be quite efficient in question answering and text generation, however, our study focuses on building a Russian language model, which might become challenging even for such a powerful architecture. We made a hypothesis that GPT-2 will present better results, however, it is still important for us to analyze mistakes and disadvantages of both models to get a greater understanding of capabilities of transfer learning techniques. To fine-tune both models, we will use a database provided to us by a project, which supports people with Asperger syndrome and autism in Russia and maintain its informational website.

The challenge and the major problem of our study are the linguistic features of conversation with people of different ages. To implement our study, we need to analyze linguistic features of the question asking of people of different ages, find out and classify their syntactic, lexical, and discourse features, and by means of surveys and explorations find out the best ways of answering according to the age of an asking person. To evaluate the chatbot, we plan to ask three focus groups of people of different ages according to three age categories of our customizable model to test the chatbot, ask it some questions and evaluate its quality during the survey.