As we have already written many times, our facilities are quite large productions, both in terms of the occupied area and the number of various installations and units. In order for all this to work and there are no situations that can cause a stop in the production process, each node must be monitored. Therefore, we have both special people who are engaged in this, and an application for mobile bypasses, which greatly simplifies life for these people.
Separately, it is worth talking about video analytics. It can solve various problems - to improve product quality through automatic control and rejection, help eliminate sudden stops of production lines, promptly warn the operator about the need to intervene, monitor compliance with industrial safety rules, which is the number one task for an industrial facility. What, in principle, can be analyzed by viewing video from objects (and whether it is necessary to view it), how video analytics helps to save time and money, on which everything works for us - this is under the cut.
Where do we get the data from
At each facility, we have several hundred cameras in the technological perimeter, which we will talk about. There is also a perimeter, which includes security cameras, but we will not talk about it for now :). But the so-called production cameras, which constantly look at a particular site, unit, equipment - this is exactly what the focus will be on.
Our plants are of different ages, and there was a time when not all of them belonged to SIBUR. Over the previous 15 years, their modernization has been actively carried out, and now there is a situation where different plants may have different cameras and systems for working with them. Accordingly, each operator on such objects is accustomed to a particular software and the usual output of video streams for it.
To take and comb everything under one comb at all objects, retrain the operators, collect everything and display on our interface - this is an additional stress for all parties involved, as well as an extraordinary cost in terms of time and money. As well as the option of replacing cameras with new ones.
Therefore, our idea was this: to integrate with any of these surveillance systems so that we use exactly its interface, familiar to operators, and can display the necessary pictures when certain events occur. At the same time, we do not create our own interface on top of it, the user does not have to retrain. By the way, more about users.
Users of our system
All this was conceived for the convenience of the operator, who conducts the production process in his control system. By default, the output of information from cameras is a standard picture, when on the screen in front of you there are many rectangular windows with video from different cameras. Almost like Zoom on a remote during a day.
The main tool of the operator is the production control system. If something critical happens, the operator notices it and takes appropriate measures. The combination of two factors, the operator's experience and his knowledge of the processes, information from the instrumentation, coupled with the picture from the video cameras, make it possible not only to make decisions on the events that have already happened, but also to predict something not very pleasant and avoid it. It is always easier to prevent something by pressing a couple of buttons than to spend a couple of hours (days, millions) to eliminate the consequences.
So it is here - the operator is able to notice something on the video that after 5 minutes of inactivity will result in an emergency stop and losses for the company. Or something else not very useful.
For example, equipment clogging. We wrote hereabout our extruder and how it can clog. The production control system gives a signal when the clogging is already a fait accompli. Everything is already hammered, an appropriate signal is given, the mechanism stops to avoid damage or a serious accident. The operator, watching the installation, can notice the appearance of factors that precede the clogging in advance. In the case of polypropylene, agglomerates will be visible (pieces of granules sticking together and gradually filling the surface of the vibrating screen).
And, of course, video surveillance is labor safety. When they describe such cases, they like to give an example of cases when the system is able to recognize whether a person is wearing a helmet or not. Perhaps it sounds nice for people out of touch with production. The reality is that at serious facilities the level of self-discipline of employees is quite sufficient in order to understand: since a helmet is needed, it must be on the head, just like other elements of PPE.
Therefore, in the case of labor protection, we analyze the correctness of the sequence of actions of an employee during typical operations, for example, analyzing product samples, or simpler cases with a person being in a restricted area.
But the cases described are, rather, optional features of the system. The primary function of video surveillance and video analytics is to unload the operator and help to avoid the occurrence of pre-emergency situations. This helps not to broadcast the online stream from all cameras to the operator’s screen, but to automatically display only the picture on which something is happening. Something different from normal conditions, requiring human attention and decision making.
But it would be too boring if the whole point of the system was only to observe and warn. Therefore, now we are developing this idea in two directions at once.
The first is predictions. Where without it. Since we collect records about some situations, it means that we have an archive of these records, they can be used. For example, the same clogs: we collect statistics about how they happen, where most often, what kind of clogs and so on, what is their intensity. Then we connect the received data with the information collected from the sensors. Then we go to colleagues who are engaged in building predictive models, and we get the opportunity to predict blockages much more accurately and adjust the technical process even before their first signs appear.
The second is tight integration with the existing production line. Previously, if it was clear that a marriage had gone on some line, it only remained to signal the operator about this and ask that they send a living person to rake it all up. After integration, you can automatically send quick commands to open the dampers, after which the marriage will go to the bump stop.
So how does the system work?
Taking into account the geography of our work, we cannot collect all video streams in Moscow and process them here. Therefore, our system is distributed, we must put it as close as possible to the video streams, otherwise the existing network will not be enough for data transfer. Here we are often asked - why do not you want to use cameras that allow you to conduct preliminary analytics directly on the camera? We want! :) But! Here, there are two factors at once, according to which processing on cameras remains a beautiful thing, but difficult to achieve in most cases.
Firstly, explosion protection. Yes, that's why we write about it in almost every post, because it is needed almost everywhere. Of course, there are explosion-proof machine vision cameras, but the price tag for them is by no means small. Plus, any change in such an infrastructure in an explosive production is always expensive and time-consuming (since, in addition to the installation work itself, it is necessary to prepare and coordinate a project with the supervisory authorities, and this is also time and money).
Secondly, production stops. If we have the opportunity to replace the cameras with more modern ones, it is impossible to go and install the new necessary hardware without pauses in production. The process of finding a suitable technology window also delays implementation.
So it turns out that the processing is much easier not on cameras, but on servers. Therefore, the paradigm of our system is that we try to make the most of the existing infrastructure that we have already installed. The servers responsible for analyzing the video stream are located in the explosion-proof zone, as close as possible to the camera. The integration bus, which provides integration with the video surveillance system, data lake, IoT platform and other services, as well as event processing logic, is deployed on several virtual machines in the production cluster to ensure high system availability.
Of particular interest to us is the solution to the problem of directly analyzing the signal. Here it’s worth mentioning right away that not all image analysis models working in SIBUR are developed from scratch inside. In accordance with our paradigm of using to the maximum of what is, we actively use the results of the work of models that are already in video surveillance systems or in video analytics systems from other companies. But in case there are no installed models, we need to find / develop something new. For this, at the system design stage, we have developed and fixed the software interface for the model.
This gave us the opportunity to attract third-party developers and quickly integrate such cool stuff like OpenVINO. The range of tasks in recognition in our country is very wide. Starting with the classical problems of detection and classification in rgb space (analysis of the appearance of agglomerates on vibrating screens), continuing with object tracking (analysis of product re-grading) and ending with working with rgbd when analyzing the flow from cameras to control the manipulators.
The main subtlety in the case of petrochemicals is the imbalance of the sample. There is a lot of data in a situation where everything is fine, even a lot. But there is very little data when something goes wrong. It is impossible to deliberately go and, for example, score a line just to collect a test dataset. Somewhere it is simply impossible without stopping and restarting the line (which will cause loss of money due to downtime), somewhere it is generally unsafe from the point of view of the technological process. Therefore, we are actively exploring the possibilities of additional augmentation and data synthesis. This has its successes, we hope to finish writing and publish an article about it :)
The very same development we have is in Python, both back and models. Deployment primarily to Docker.
results
The project was launched a little over a year ago, during which time we have already managed to implement our software at our facilities in Tomsk, Tobolsk, Voronezh.
Thanks to the system, a lot of events were noticed that were processed by operators in a timely manner.
The main thing is that the system has proved that it is useful and receives a positive assessment from production, as it was planned, situations are leveled in which we would continue to produce a substandard product for some time, wasting production that is impossible to sell and quite difficult to collect before how to dispose. Or when instead of the expected product, due to unnoticed failures, we produce another, cheaper product.
For example, this is possible with polypropylene. The final stage of production with us is polypropylene granules, which you saw in our other posts and videos. It is convenient to put them in bags, transport, fill up in containers, they correspond to the declared performance characteristics. But if you fail to notice and handle the situation with the plug in time, you will get another product. Agglomerate will crumble, granules will begin to stick together. This is still the same polypropylene, but because of such jambs during production, this is a different brand of product. Cheaper, of course.
Therefore, it turns out that the system saved us both from additional emergency stops, when people waste time and cars are sadly idle, and from the production of such a product with a class lower than planned.
Plans for the future
We continue to refine the system, because there are many video surveillance systems at the facilities and they are different, but we need to integrate with each one. Right now we are busy with establishing a full-fledged remote deployment of the system at sites. With remote access in factories, it is also not so simple, so there are some peculiarities - a corporate circuit, different network segments, security restrictions, and so on.
Plus, we are working towards a more complete collection of data from sensors in order to have the most detailed picture of what is happening.
The feedback from the operators comes mostly positive. If we decided to create a system based on the paradigm “And now you will all work in this new thing,” it would most likely be a little different.