To receive calls, we used the platform Voximplant , and to recognize the questions and answers - DeepPavlov . The voice assistant was launched in two and a half weeks, and it helped process 5,000 calls. We managed to roll out a product that helped the residents of Tatarstan receive reliable information from the authorities, and simply go out into the street. Below we will tell you how we did it.
What was the task
We originally planned to make a voice assistant that would help people receive government services and answer frequently asked questions. But when the whole gimmick with coronavirus began, we realized that Lilia would help unload the call center: for example, advise on the hotline, help get digital passes and unemployment benefits. Everything that a person can find out on the website of state services can be found out from Lilia, and you can also just chat with her.
. 1 -: , . : , , , . 12.
. , . , . , , , — « ».
In addition to issuing digital passes, Lilia had to answer questions from Tatarstan residents. We conceived a voice assistant that would answer common questions like "What you need to do to get a pass" and "Does ginger help you with coronavirus?"
To solve this problem, we could make a regular IVR. But IVR takes a long time, and if you do it normally, then you need to lay a complex architecture. In general, there was no time for this. We thought it would be easier to make a voice assistant (which can translate speech into text, process it, classify intents and provide a voice response). Hehe.
How it looks from the user's perspective
Everything is simple here.
- The man calls the hotline, he is offered to choose between a conversation with the operator and Lilia. At any stage, you can switch to a live operator.
- .
- , . , -.
To receive calls and implement scripts, we used the Voximplant platform: wrote a script and connected an answering machine signal. Lily greeted the man and asked what he wanted.
The caller asks a question over the phone. Then Lilia started listening. At VoxImplant, we used the ASR module to translate speech into text, it calls the Yandex.SpeechKit model under the hood. Thus, the audio stream is translated into text, which then undergoes tokenization and stemming. We also tried to extract features: NER, POS and Chunk for basic ML methods, but all this took a very long time.
For example, initially, from the question "Please tell me how to protect yourself from coronavirus", we stem "protection" and "coronavirus" and transform them into vector representation. The vector representation of entities is included in the classifier for the coronavirus (and later - for public services). Now DeepPavlov is doing all this.
Further, several options are possible, depending on the level of confidence (confidence):
- If the network classified the question with sufficient confidence (based on the study, the thresholds were selected individually, by class), then Lilia will answer the question.
- If the network has a low confidence response, we assume that this is a question that we did not cover within our dataset (but the question still refers to the coronavirus domain) or the person just wanted to talk about another topic. For example, he asked "Who is Elon Musk."
For such questions, we used the BERT model trained on the Wikipedia dump for the knowledge base question answering problem.
At the end, Lilia necessarily verifies the correctness of the recognition of the intent and the quality of the answer by asking: "Did I answer your question?" If the user answers yes, then Lilia will wait for the next question. If not, then we will ask you to reformulate the question and go through the whole cycle again. It so happens that this does not work. Then live operators enter the battle.
Now Lilia speaks in a pleasant voice of a speech synthesizer from Yandex - we slightly changed the key and increased the speed. Sometimes Lily confuses accent, but this can be fixed with markup. Of course, I would like to add Tatar, but so far it's difficult.
In total, the project took two and a half weeks, taking into account datasets: an idea came up, we discussed the project with the minister and, as we say, alga. A week was spent on estimation and research, development took another 10 days, then we finalized and screwed on additional functions. The main horses were Nvidia RTX2070. BERTs required about 12-16 GB of video memory.
From LSVM and catboost to DeepPavlov
During the development process, we used different classifier models. First, we tried machine learning models such as random forest, LSVM, catboost, logreg. In general, the accuracy of machine learning models was not very high. Why is that? Because many of the user questions are very similar to each other: the question “my child is sick with coronavirus what to do” is quite similar to the question “is it possible to walk with the child during the coronavirus”, although these are different categories and different answers are required.
We could do entity extraction, sampling, research. But we were in a hurry. Therefore, we decided to use the DeepPavlov library from MIPT in our work, which gave an accuracy of 78% with logistic regression and BERT - 84%.
The accuracy of the answers depends on the markup of the datasets. We had a list of 200 questions people ask on the hotline, but they were not properly segregated. For example, people asked what to do if one of their relatives got sick, and the question was what are the symptoms of the coronavirus. ML models got confused.
Results and plans for the future
Lilia worked for 2 weeks and processed 5000 calls. During this time, Lilia greatly facilitated the work of the hotline operators - they did not have to answer trivial and repetitive questions. Thanks to Lilia, users received passes, answers to questions and just talked. Of course, there were users who swore at her and asked to transfer to the operator.
The self-isolation regime was canceled, digital passes are no longer valid, but Lilia is still in the ranks. She continues to answer questions about the coronavirus, but now the ability to answer questions related to public services has been added to her.
We are a ministry, and we, in fact, have two tasks: so that other departments normally use technologies and so that residents of Tatarstan can easily and simply communicate with the state. The second task is perfectly handled by the portal of state services - ours, local, not federal. But this portal is a site and an application, which is still difficult for some to reach. And if residents do not go to the portal, then the portal will go to the resident, that is, we are moving towards simplifying the interaction with the portal for people who do not particularly use the Internet.
Now the Ministry of Digital Affairs of Tatarstan is working to ensure that people can receive government services by voice and via chat. We want to make a universal assistant who you can call / write and get answers to all important questions.
So far, Lilia can tell how long the registry office is open, and in the future we plan that she will be able to take meter readings (but this is only when we decide the issue with the security of the transfer of this data). In general, we are turning Lilia into a separate product.
If you would be interested to teach Lilia, welcome to our team.