My name is Anton Fedoseev, I am a developer of an AR platform for video conferencing, which is used by our production facilities. We launched a service through which our employees can communicate via video with colleagues from other regions, receive online consultations from experts - equipment manufacturers (also from other regions), and the company can save from 500 thousand rubles to several million for 1 video call.
Where did this savings come from, what is the peculiarity of using such a service at industrial facilities, why we didn’t manage with existing analogues on the market and how we made our product - I will tell you under the cut.
A bit of history
The AR project started at SIBUR in January 2018, simultaneously with the start of active work on the integration of the Industry 4.0 concept into various production scenarios at our factories. By that time, SIBUR had already analyzed a number of cases related to the repair of industrial equipment, and the company was thinking about how to reduce the cost of attracting external specialists from suppliers of this equipment and service companies, replacing their in-person visits with consultations on audio and video communications.
In the same year, we tested the technology of remote maintenance using AR-glasses, using a number of solutions available on the market, and also worked out basic scenarios for the application of this technology, and calculated the economic effect of its implementation. We realized that such a project is both expedient and feasible from a technical point of view. And our Tobolsk enterprises became the test site.
We named our product “Remote AR Expert”. In 2019, with the help of our "Remote Expert", who was still working on the temporary software of a partner, more than 300 video calls were made - all these are cases when equipment suppliers, internal experts, representatives of construction companies transferred or received all the necessary information through the AR service, without coming to our production. And in June 2020, SIBUR began active deployment of its own AR platform for remote maintenance and repair of industrial equipment.
What is AR remote maintenance and repair service?
Our AR service is easy to use and, generally speaking, includes 2 key components: augmented reality glasses RealWear and Epson (depending on environmental conditions, more on that later) and SIBUR's own mobile media platform integrated with them based on Android OS. developed in collaboration with G-Core Labs. The platform allows you to fully digitize the entire process of maintenance and repair, organizing HD broadcasting, using WebRTC and a full-fledged communication platform for effective interaction of a specialist on site with a remote expert. All control is carried out by voice, the hands of the person working with the installation remain completely free.
A representative of an industrial equipment supplier or an internal SIBUR expert is in touch with the factory specialist, who simply receives a link to his personal e-mail and starts video conferencing in any browser (Chrome, Edge, Mozilla, Safari, etc.) on any device, in incl. tablet and smartphone.
There is no need to install any special applications or programs.
SIBUR is a geographically distributed company, our enterprises operate in more than 20 regions throughout Russia (and not only), from the European part of the country to Siberia and the Far East. Sometimes there are thousands of kilometers between the plant and a qualified expert who can manage the maintenance or repair of a particular equipment. For example, it takes 7-9 hours to get from a corporate center in Moscow to an object located in Siberia - during this time everything can change both for the better and for the worse. And what if the expert is not in Moscow, but in another country? The factories operate both Russian installations and equipment of American, Japanese, European and Asian manufacturers. We also take into account that you need to arrange a business trip, sometimes - get a visa. It should be noted herethat an in-person consultation with a specialist is not cheap: these are travel expenses, the cost of paying for a full working day of experts, or even several days (hourly billing does not work in a face-to-face format). One such consultation can cost a company from several hundred thousand to a million rubles. And what if quarantine restrictions, border closures and suspension of international flights again? That is why the launch of the AR platform for remote consulting was very relevant for us.closure of borders and suspension of international flights? That is why the launch of the AR platform for remote consulting was very relevant for us.closure of borders and suspension of international flights? That is why the launch of the AR platform for remote consulting was very relevant for us.
AR service: necessary functionality and how we implemented it
The main scenario for using our service is group video calls. To reduce the load on the network and improve the quality of the transmitted video, we use a media server. The media server used in the project allows duplicating the incoming media stream for several clients, adapting it to the communication channel of each client. It also allows you to create live previews of users, eliminating the need to transfer full-size videos of other clients to the client. For group work of clients on the basis of the media server, such tools as zooming and recording are implemented.
When they say “video conferencing,” everyone is representing skype or zoom. But we work with interfaces and documents, and in production we work with equipment, so we need free hands to demonstrate the processes. The environment itself is also different - the communication tool should not cause a spark.
To free the hands of a specialist "in the field", the platform provides full voice control of AR-glasses: on command, they are activated, start remote broadcasting, record video and take high-resolution photos. After the session ends, the consultation records are saved on the enterprise server. At the beginning of the session, the name and regalia of the connected expert, indicators of the quality of the network connection and the battery level are displayed on the microdisplay of the glasses.
In severe frost, the glasses work smoothly for 2 hours. And here we will return to the topic of the spark. At our production sites, we use equipment designed for operation in a gas hazardous environment - this is RealWear. For work in repair shops, it is possible to use dust and moisture resistant devices in industrial design. Here we use Epson equipment.
About the picture
G-Core Labs helped us develop the media platform. A stable and clear picture, the ability to see even small details of equipment - all this was very important for us. In this regard, we have changed the standard behavior of the WebRTC library, in which when the speed of the connection to the network decreases, the picture quality deteriorates, but the frame rate remains. In our case, it is the quality of the picture that is critical, even if there are delays in the delivery of the video stream. Well, if the connection completely fails, our platform gives priority to maintaining the audio connection with the operator.
In addition to a high-quality picture, we also needed augmented reality - tips, notifications. With the help of a virtual pointer displayed on the micro display of the glasses, an employee of the equipment supplier can direct the actions of a specialist from the plant, indicating where he needs to look, what to turn and where to control. Also, chat and sending stickers are built into the functionality of the AR platform. The latter are needed, for example, if something is wrong with the connection. In this case, the remote expert sends signal stickers - for example, a "tick" if everything is done correctly, or a "stop" if you need to stop actions.
What makes our service unique?
The main distinguishing features of SIBUR's solution from other analogues on the market are the depth of its integration into the real production tasks and processes of enterprises, the comprehensive digitalization of the repair process and versatility, which allows the service to be deployed at a variety of production sites.
In our solution, we worked out a full-fledged role model focused on business customers - production managers, site managers, mechanics, technical customer service managers, etc. The AR service provides a ticket system, which implies not only the possibility of video communication, but also a full-fledged digitization of the business process for conducting a remote consultation. With the help of this ticket system, we guide the customer along the entire path of resolving the issue that has arisen, from checking the completeness of the necessary information to assessing the cost of work, the necessary competencies and qualifications of external or internal experts, ensure information security and answer questions about who, at what time and with whom must take glasses,how this system is supported and how exactly all this interaction takes place until the successful completion of service or repair.
Of course, in Russia, Europe, Asia and America there are systems that allow you to organize video communication using augmented reality glasses. However, to date, the success of the implementation of such projects in industrial enterprises largely depends on the quality of integration into internal processes, as well as the used IT platform, customized for industrial processes and tasks.
Limitations and challenges: no - plugins and WiFi, yes - LTE and microservice architecture
One of the requirements when creating the platform was the absence of installing anything (plugins or programs) on users' computers. We met this requirement using a web client and a modern browser. Almost all modern browsers (both desktop and mobile) support WebRTC technology. Our service is built on it. In short, WebRTC allows you to establish streaming audio and video communication between participants using the RTP protocol, which in turn runs over UDP. There is a mechanism for selecting paths for establishing a connection between participants (ICE), a mechanism for negotiating the format of media streams (SDP), algorithms for estimating the bandwidth of a communication channel, algorithms for adapting compression of media data streams and media processing algorithms such as echo cancellation, etc.Recently, WebRTC has received a good spread in browsers and development. Using UDP for data transfer between participants is not always possible due to the presence of firewalls, antiviruses, routers and NAT. To solve these problems, STUN and TURN were invented. For cases where UDP is not available, it is now possible to use TCP for ICE. TCP support has also been added to STUN and TURN.
TCP increases the availability of our service to external clients who may be behind NAT, firewall, corporate firewalls and other various network restrictions. However, using the TCP protocol instead of UDP made a change in the behavior of WebRTC. During the transfer of media data via UDP protocol, packet loss, duplication, reordering, small delays are possible. All these situations are correctly handled by WebRTC: there may be a short-term appearance of image artifacts or sound distortion. When using TCP, there are no losses, duplication and reordering of packets - this is monitored by the algorithms embedded in the protocol. But this all translates into big delays and jitter when receiving packets.Visually, it looked like freezing frames and a gradual accumulation of delay between the video source and the display on another client. After making changes to the media server, we practically managed to get rid of the accumulation of latency, but the frame freeze was not overcome. It was decided to temporarily disable video transmission from the media server in case of long delays in receiving packets. This solution made it possible to preserve voice communication between the participants, even under poor network conditions.This solution made it possible to preserve voice communication between the participants, even under poor network conditions.This solution made it possible to preserve voice communication between the participants, even under poor network conditions.
All industrial smart glasses on the market connect to the network via WiFi. This creates certain difficulties in the operation of glasses on the vast territories of SIBUR factories. Providing a stable WiFi connection to all places in a factory where work is supposed to be carried out using smart glasses, given the scale of factories and installations, is quite a challenge. It is more efficient and cheaper to use LTE wireless network in open spaces. Such a solution, however, requires the employee to have additional equipment - an LTE modem. The LTE network is also well shielded by the reinforced concrete structures of the workshops and the metal cases of the equipment inside them. To provide coverage inside the building, stationary LTE modems and a network of WiFi translators are used. When working with smart glasses, the quality of the connection to the servers is unstable,sometimes breaks off. In order not to lose information about what the employee was doing with glasses and how, the glasses record video on an internal medium. This video is recorded in HD quality and does not depend on network conditions. We have the ability to remotely erase data on glasses, as well as delete data in case of root detection.
All SIBUR services are built using a microservice architecture. Our platform is no exception. In fact, it consists of two microservices: BPM microservice and video calling microservice. We also use microservices deployed or developed by other teams: an authentication microservice for working with users, a notification microservice for mail and SMS mailings, and some others. Microservices communicate with each other using gRPC. Clients communicate with our service in the same way: for smart glasses and a web client. However, gRPC is not directly supported by web clients. They have a library for working with gRPC-web, which is not compatible with gRPC. To access the service, a gRPC gateway is used, which translates gRPC-web to gRPC and vice versa.
At the moment, we have built a modular platform that provides stable and high-quality (as far as the network allows) video and audio communication, and also has the ability to expand its functionality by adding new modules and equipment. There is still a lot of work ahead: plans are for closer integration with other information services of SIBUR, research of a new generation of smart glasses and AR and VR helmets, implementation of augmented reality tools and effective collaboration of our colleagues and third-party partners.