Excursion into the past
I have been working for ICL Services for over 4 years and started as a technical support specialist on a large project. Now I lead the ITSM group in the direction of Service Desk - my competencies are concentrated in the deployment and configuration of ITSM systems, IT process management (Incident, Change, Problem Management). In fact, without exaggeration, I can say: the most skillful people in the field of SD work in my group.
The peculiarity of a large company, which I discovered after working in it for about six months from the moment of employment, is the following: people rarely shared their best practices. Not because they were greedy, but simply because they were deeply too lazy to explain to colleagues what the benefits were. After all, as it often happens, for the most part, everyone here was techies with long-standing relationships.
And then I come - and I see that one has gorgeous scripts that allow some of the work to be done automatically, the other keeps a schedule of his shifts in a cloud application and never confuses the shifts (and in those dense times we still used Excel), the third wrote a simple a web application that displays all fields of the application on one screen so as not to waste time switching tabs in the ITSM system and opening them when checking applications.
But these developments remained individual and were not shared with the entire project team, not to mention any implementations on similar projects. As my leader said, we have always reinvented our bicycle on our own - and I was right in the case. Of course, there was also a human factor: for example, there was someone more experienced, and he could not use the best practices of the "young". And a special classic - "but with him we have a personal conflict, I will not use the tools that he uses . "
It was successful for me, and in general for our entire department, new approaches to the processes were introduced and tested. Tons of documentation, howls from users and support groups - everything is as usual. I managed to push through the improvements that I saw, but most importantly, I managed to involve almost all the guys in working together on the service. After all, you know what a team that united against a problem is capable of? The problem has no chance. The customer introduced a new indicator - the percentage of errors when filling out applications. At the start, it was about 10% per team, that is, about 200 applications had both minor errors and significant ones.
The line and project leaders said, “it is impossible to reduce the percentage of errors even more - people are people, and they will always be wrong . " We replicated a web application written by our leading specialist, set control points and regularly took information on these indicators, worked out specific cases: when the ITSM system itself substitutes the necessary fields from Active Directory, what to look for. The result was not long in coming - in six months we brought the% of errors per month to the critically low level of 0-0.6% per team.
And around this point in my career, I realized two things:
- I want to work in a cool team , and its integral work result is much better than what even the most ingenious employee can do. This decision will definitely lead me to management later.
- I want to combine the achievements of talented, cool guys into a single system that is better, more reliable, faster than the one that you had before.
Project start
In 2019, I was offered to lead the Digital SD project, within which we had to make our services more competitive with the help of modern technologies: Machine Learning, various voice and text bots, omnichannel platforms, auto-classifications and auto-resolving applications. In other words, all those things that take the client's experience to a new level without increasing the labor costs of providing a service.
Honestly, after looking back at the whole hype around AI, at first I was very skeptical about the task. And it seems that the direction is correct, and the goal is good, but something was wrong here.
And the catch was that no one knew what exactly needed to be developed. There are a lot of directions, what to grab onto? Usually at such moments in articles or books they confidently write: "even then we knew how and where to move." So, this is not about us. I stood with the product owner and made a list of questions that we needed to answer when designing the product:
- What useful functionality do we bring to the user and the customer?
- How does this integrate into the current service?
- How do you need to change your current workflows for this to work and be beneficial?
- How and on what technologies should this be done?
- How should an economic solution model be built?
But we did it quite thoughtfully: we gathered a group of experts from SD from among project managers, line managers, technical experts, wrote down user stories, made a value map - here's a small piece of it:
In parallel, together with the developers, we described the top-level architecture, thus obtaining a starting point in the project ...
SCRUM work
So our introductions: Product Owner, PM, Scrum Master, and Development Team. A working sprint lasting 2 weeks. The task is to organize the product release process in such a way that ...
But seriously, it was necessary to take into account the requirements of all stakeholders, deal with the pains and understand what really needs to be done, what can be done later, and what not to do at all.
We had 3 large areas of activity:
- . , , , .
- . , «» .
- . HR, , IT, . .
We managed to establish normal work only by the 3rd sprint: we more or less realized that we need to focus on areas 1 and 2, because project managers talk about pain, and the heads of support services talk about wishes.
I will not stretch the article and tell you how cool Agile is in itself and how it helps a lot. But here's what I can really say after working in this mode for about 30 sprints:
- Scrum is a working approach. You put in resources and methodology, and you get supplies.
- – . , , , . , , , , « », – .
- . , , . , , – . PM – , , , – , – .
I have less experience of this kind, so this large and heavy burden fell to a greater extent on the Product Owner: to compare the expectations of stakeholders with reality, clearly understand the capabilities of my team, have a clean eye, understand the market situation, balance between directing resources to useful functionality, experiments and "necessary evil" - security of solutions, organization of service architecture, environments, documentation.
- PM . , . -. Devops , , : , /. , , Agile - , : , , .
- - – . , -. , . , PM -, . , . , , !
We didn't start development from scratch. We had a lot of developments in the company that could be used in one form or another - there were two auto-classifiers alone.
I felt déjà vu when I collected all this from different projects, divisions, as I once collected developments on my first project.
We started developing the dialogue system with several open source libraries - and one of them was DeepPavlov. We did not have much experience with it, and on our tasks the recognition quality came out mediocre. Soon we switched to Rasa, and there things went much better - and after training the model on our specific data, we got good and confident recognition of dialogues.
The layout of the dialogs themselves was done manually - at that moment there were guys in SD who took on this task. Our lead Python developer quickly wrote a markup program, and we fed the model a couple of tens of thousands of conversations. The fragments were taken rather short, 3 seconds each - this way the result came out better.
Initially, we had 2 virtual machines on Windows and on Linux, some of the services could only work under Windows. But when we started to bring the first developments to the pilot, we quickly realized that 2 virtual machines for one project is too expensive, we need to redo it. Now we use one virtual machine on Ubuntu for production. Of course, they are all isolated and each project has its own area.
We also quickly realized that setting up two virtual machines, raising and debugging all services, opening ports and other settings take absolutely indecent time. Then we made a CI / CD solution based on Docker - both for the main code and for the ML part.
Somewhere in the 9-10th sprint, we were faced with a request from many customers to make their own speech recognition system. Most of the customers were not at all ready to transfer their confidential information to the "clouds" to third parties. So, we wrote such a system and now we can offer it where it is important to have the entire architecture within the security perimeter - for example, for companies closely related to the state. Or place it in our infrastructure, knowing for sure that sensitive data will not get to third parties.
We deployed a component monitoring system, set up health checks, and integrated the system with a chat channel in Telegram.
And finally, I'll tell you about one more subtlety that may be useful to someone when designing their chat bot. Initially, all our code was quite monolithic and it was laborious to make changes to it. We have divided the chatbot into two large parts: the base bot and the customization. We had to rewrite the logic - but thanks to this separation, we could quickly deploy the basic and common components of the bot and edit only what was custom for each specific project.
Project results
We clearly understood the niche nature of our history: we will not be able to compete with boxed products that have been developed for a dozen years, and there is no need for this either. Our niche is to provide automation tools at any stage of a service project, from presale to contract expiration. In other words, initially we did not have a goal to make Google - but the goal was to make a designer that would help sell Service Desk, help reduce the cost of providing a service, and give additional opportunities to customers and their business.
I also noted an interesting point for myself: rarely a boxed solution on the market can completely cover the pain of a customer and at the same time arrange for a price. Either the customer overpays for the functionality, which he will not use later, or some of the improvements will have to be done by specialists or specialists of the selected vendor, if he takes it up.
And here we have an interesting and rather difficult integration task, in addition to offering purely our own solutions for automation, to build a system for the customer from our developments and third-party solutions that are already on the market, suit the customer's functionality and maintain such a system as a service. Perhaps I will talk about the results of this work in the next articles, if there is interest.
Most of our tools and developments have already been implemented in current services, we are piloting some, and we will refine something at BAU. The chatbot still feels the worst of all, today it is least useful to projects - perhaps because expectations are now quite high. Everyone wants a smart bot that can perceive human speech, calmly answer all user questions, be able to handle interruptions, has integration into all systems, learns itself without human intervention, and over time recognizes more and more intents.
But any developer who is well versed in the topic understands how difficult this task is - after all, even by increasing the functionality and increasing the number of intentions that a bot can recognize, we can worsen recognition for existing intentions. But this was a digression - in general, it seems to me that we have nevertheless coped with the main task of the project.
What happened at the exit
1. Intelligent voice assistant and script designer for it. Closes the need to automate the incoming flow of calls, recognizes the user's speech, voice to text and transmits it to mail, chat, ITSM system, depending on the setting. It can integrate with a bunch of different systems, either with ready-made connectors, or with those that we wrote.
2. The dialer with the engine from the voice assistant.Closes the need to call the specified pool of numbers and then collects user responses depending on the scenario. Regular calls are possible, there is a setting for the number of repetitive questions to clarify how many times and after what time to call back. Stores data on calls made together with call results and call recording. Now it is used to remind engineers about patching on a project implemented for a large international food retailer, in the HR department when organizing interviews for mass positions, in plans to collect information on the quality of resolved applications in a large fast food restaurant chain.
3. A chatbot for creating an application, resetting a password and a script designer for it.Knows how to ask for primary information for registering a request, register a request in the ITSM system, and return a request number By the time the article is published, it will learn to show a list of open applications with the ability to add information to them or close them. It can connect to different ITSM systems, with or without api access.
4. Tool for quality control. He knows a little so far: he tracks calls, recognizes whether the operator has greeted or not, identifies conflict-generating words, mate in a dialogue, has a full-fledged interface for the quality controller. They did it for themselves, but it can come in handy at the CC.
5. Auto-classifier.He is able to parse applications in the ITSM system, fill them in and send them to the necessary solution groups. May take into account the availability, workload and specialization of engineers. For example, you can set up the logic that all EDS applications should be sent to the main specialists Vasily or Andrey: if Vasily is not on duty, the application will go to Andrey, and vice versa. If not both, the ticket will be added to the general line of business applications support. If Vasily has 2 applications, and Andrey has 1, a new application will be sent to Andrey. You can confidently retrain the model, increasing the accuracy. The disadvantage of the system is its point.You should not expect 100% accuracy from the model or work on the entire volume of applications. On a test sample, we did 90% accuracy for 50% of applications with very consistent data, where users are used to working with templates. The second disadvantage is the volume of orders. It makes no sense to train the model if you have less than 1000 applications per month.
6. Tools for auto-solving applications.This is a set of tools from the GUI with scripts that automate the standard actions of a technical support agent: collecting logs from systems, taking screenshots, including in the shadow mode, updating policies and other things specific to each project. The second tool is the automation of those applications where approval is required. The tool itself generates a letter to the approver with links to approve / refuse and, based on the results, itself gives a command to provide access, or it generates a letter with a refusal.
Instead of goodbye
Winter is coming, the project closes at the end of this year, I
Thank you for your interest and your time, don't get sick!