We will tell you about our new product in detail in this article. The plan is this:
- why you need a Medical Voice Assistant;
- main use cases;
- what components the solution consists of;
- how tasks are processed and assigned.
How IHL works
Why you need a Medical Voice Assistant (IHP)
IHL has two main objectives.
- . , : , ( ) , , .
- . , , . / , .
Here we will look at basic scenarios for using a voice assistant that are suitable for almost any hospital. Of course, there can be many more options - it all depends on the needs of a particular clinic.
Doctor / nurse call . Suitable for cases when the patient needs the help of the medical staff. As soon as the patient utters a key word - for example, "pain reliever", "painful", "need an injection" - the system assigns the task to the necessary group of specialists and sets the deadline for the decision.
Automatic query execution... The patient asks for help information that is already in the knowledge base - for example, the schedule of the cafeteria. IHL finds the relevant article by keywords and voices it to the patient. The patient is happy, and so are the nurses: they can do more important tasks.
Two-way communication . It is used when the patient needs a specialist consultation - for example, if you need to ask your doctor for details about the dosage and schedule for taking a new medicine.
Components of IHL
The medical voice assistant can solve a variety of tasks, from the simplest to the most complex. However, its versatility is combined with ease of implementation. At the component level, this is a rather minimalistic software and hardware solution - we deliberately aimed for this.
IHL includes 4 basic modules.
- Patient terminal. It consists of a microphone for receiving sound, speakers for playing messages from the system (and doctors) and a microcomputer that processes patient information and interacts with other components.
- Speech recognition and synthesis system. The system can be used in the cloud or installed locally.
- Accounting and request processing system. The web application in which the medical staff works. Implemented on the basis of the ESM platform for automating business processes SimpleOne .
- Terminals for doctors and nurses. Tablets or PCs that are connected to the accounting and processing system via Wi-Fi or LAN.
Processing voice requests
The terminal in the ward acts as a communication center between the patient, the Speech Recognition and Synthesis System (SRS) and the Accounting and Request Processing System (CPS).
In general, the request processing process looks like this:
- the terminal is constantly waiting for a keyword;
- if a keyword is spoken, the terminal records a short audio (4–5 s);
- audio is sent to the CPC;
- the terminal sends a response from the CPC to the control system;
- the terminal announces the information to the patient depending on the preset CPS logic (template).
The screenshot shows an example of processing a request with an X-ray recording
For a more comfortable interaction, in addition to the standard responses “Your request has been accepted, wait for help,” IHL reproduces special signals - for example, about the beginning and end of the patient's message recording.
Assigning tasks based on templates
In the system, you can configure unique query templates, each of which can be assigned activation conditions - the required keywords. As soon as the client utters the keyword specified in the template, the system activates the request and assigns the task to a specific group of specialists, or, for example, announces the necessary information from the knowledge base.
The system has a table "Messages", which stores all phrases pronounced by the patient. The system monitors the table of incoming messages and creates a query if the message matches the template condition.
Example
For the request template "Pain reliever for a patient", select the keyword "painful". We assign a high priority to the request, select the “Nursing staff” method and connect the required group of performers - nurses. Now, if the patient says “it hurts,” IHL will automatically assign the task to the Nurses group. Group members will receive a notification that the task has been assigned; the patient's message will be reflected in the "Description" field of the request.
View of the request template
In parallel, the task is fixed in the response time control (SLA) counter. In our example, we have 10 minutes to solve the problem. If the specialist does not meet the deadline, the problem is escalated to the superior doctor or group (escalation scenarios are also configurable).
SLA counter
An unlimited number of request templates can be created in the system. This can be done by employees with administrator rights.
Query templates
Why we chose Pocketsphinx, Python and SQLite3
In the prototype solution, we used a microcomputer of the Raspberry Pi family and the Raspbian GNU / Linux base OS. Terminal is a simple application written in Python using REST requests to auxiliary systems and Pocketsphinx (LiveSpeech) library.
The Pocketsphinx library is good because it helps to improve the performance of the first response - the search for a keyword. The speech recognition and synthesis system uses complex mechanisms and dictionaries for recognition. Roughly speaking, Pocketsphinx speeds up the keyword recognition process to avoid negative customer experiences. Pocketsphinx is also easy to configure and can work in several modes.
Using Python and Pocketsphinx greatly expands the functionality of the terminal. To keep patients from getting bored, games can be added to IHL. In the prototype, for example, we have implemented a simple game "Cities".
The standard REST API is used to integrate the CDS and KMS.
Below is an example of a CPS address (we send a POST message towards the CPS, parse the response, and so on in a circle):
url = 'https://user:pass@mva.simpleone.ru/rest/v1/table/mva_itguild_inquiry'
payload = {«description»: text, «subject»: «mva_inquiry»}
header = {'Accept': 'application/json;charset=UTF-8','Content-Type': 'application/json;charset=UTF-8'}
response = requests.post(url,data=json.dumps(payload), headers=header)
i_json = response.json()
Continuous speech synthesis using CPC is not a quick process, because it is quite capacious: authorization, request, processing. To speed it up, a local SQLite3 database is used to store previously generated patient responses. This solution is well suited for processing requests with a constant scenario - for example, when voicing information about the location of doctors' offices and treatment rooms, the schedule of the canteen.
Another important point is logging. Without it, it is impossible to work on improvements and bug fixes. Therefore, the terminal implements logging of both the general process of operation of all adjacent systems, and individual components.
What is the result
Assigning tasks to medical staff using voice control is not only about convenience (for doctors and for patients), it is also about the possibility of serious optimization of work processes in hospitals. The more opportunities hospitals have to make their caregivers more comfortable and productive, the better the quality of patient care will ultimately be.
We wanted to create a really useful, multifunctional solution for hospitals, while using a minimum set of ready-made services. But perhaps the main advantage of IHL is its versatility. The voice assistant can be quickly deployed in a hospital of any size and profile. You can optimize software on the fly, taking into account the tasks of a particular medical institution and rapidly changing conditions.