When it comes to object recognition, the first clicks will be towards Google or Microsoft. What if they fight each other to recognize cars? We have carried out a study adding the Belarusian SpotVision Car Detection service to the list of players. Who will win?
Let's start with the fact that we needed to solve an applied problem - to identify cars in pictures, select them and report the coordinates of the cars. The idea, first of all, appeared for personal use - to quickly and efficiently find and see free parking spaces in the courtyard even before the driver arrives there. Later, this idea grew into a task for business - to provide additional services for customers of any parking company, as well as customers of video surveillance services.
The goal is to recognize cars on images from CCTV cameras and transmit information in a schematic form to a map in a mobile application or dashboard.
First of all, we tested Google AI, since we trust many Google products unconditionally. And it seemed that for such a large and influential monster, car recognition would be a trifling matter (it will be given easily, it will be easy to do). However, we were disappointed. Google AI recognition for cars is not very accurate. Therefore, it looks more like a PR for other Google products than a well-functioning computer vision system.
For the first test, we took well-lit images at the optimal height and scale of the vehicles. The sighting launch failed. Out of more than 40 cars, Google AI and only circled 10 cars.
Test 1. Google AI
Having discovered that even the great Google fail recognition at such a banal level, we took the Microsoft Computer Vision service for comparison. We got the following result.
Test 1. Microsoft Computer Vision
Only 6 (!) Cars. And it looks very sad.
Realizing that the AI sphere is just beginning to develop, and our sports interest cannot be alleviated, we began to collect a dataset from cars and train our own neural network. We took the YOLOv3 Darknet framework as a basis. Experiments have shown that it is he who creates the least problems in implementation. It processes images quickly and automatically applies augmentation when there are insufficient images. After a period of intensive training, we have a trained neural network based on more than 25 thousand images. At the moment we are adding another 2500 pictures for training.
This is how the Belarusian AI system SpotVision Car Detection coped with the same task.
Test 1. SpotVision Car Detection
As you can see, the vast majority of cars have been identified, and not only in the parking lot. That is, where the human eye can easily see the car, the SpotVision system does an excellent job.
Let's complicate the task and take a night view. From the point of view of the relevance of the application, it is more difficult for drivers to find a free parking space at night due to poor visibility and a large number of already parked cars. Therefore, nighttime discrimination is most valuable.
This is what Google gave as a result.
Test 2. Google AI
And a similar version from Microsoft
Test 2. Microsoft Computer Vision
That is - complete zero and the absence of any signs of cars in the image.
Let's compare the results of the SpotVision machine recognition system.
Test 2. SpotVision Car Detection
Moving on. Most often, CCTV cameras are installed on the upper floors of multi-storey buildings. This provides the greatest visual coverage of the parking area with a single camera. That reduces the costs of video surveillance services for equipment and installation. Since the most frequent and demanded case is the recognition of cars in the yards of residential complexes and business centers, we also undertook to analyze it in three systems.
Test 3. Google AI
Test 3. Microsoft Computer Vision
Test 3. SpotVision Car Detection
Now let's put the question point-blank: how do the systems cope if the image shows one whole car and the edges of the adjacent ones are captured?
Test 4. Google AI
Test 4. Microsoft Computer Vision
Test 4. SpotVision Car Detection
According to the results, Google AI found 4 objects in the picture, and three of them are wheels, not whole cars. Microsoft Computer Vision has identified 2 cars. Spotvision has recognized 4 vehicles as a whole and one wheel separately. The Belarusian car recognition system coped with the task with a small error, but most fully.
It is interesting what the systems will circle where there are no obvious signs of the car, on which recognition systems most often rely - this is the usual contour of the car and the outline of the wheels.
Test 5. Google AI
Test 5. Microsoft Computer Vision
Test 5. SpotVision Car Detection
Surprisingly, all three services identified the car accurately, only Google AI additionally reacted to the wheel or swirls from colored smoke, circling it with a rectangle.
Rain and fog is another natural complication of recognition. They reduce the sharpness of the image and can hide objects from computer vision.
Test 6. Google AI
Test 6. Microsoft Computer Vision
Test 6. SpotVision Car Detection
Google AI and SpotVision Car Detection performed best, although they lost sight of two cars traveling in the distance. Microsoft Computer Vision missed one machine, and out of the obvious four, only circled three.
So, two control tests with the maximum number of vehicles shown. Test 7 shows 46 vehicles.
Test 7. Google AI
Test 7.Microsoft Computer Vision
Test 7. SpotVision Car Detection
Of the 46 cars in the picture, Google AI found 11 cars, Microsoft Computer Vision - 30 cars and mistakenly circled the sign on the pole, and SpotVision found 46 cars.
Let's continue with examples where there are many cars. Change the angle of view.
Test 8. Google AI
Test 8. Microsoft Computer Vision
Test 8. SpotVision Car Detection We add
pictures from real yards, taking into account seasonal changes. In the variant below, it is winter time and snow. As you can see, thawed patches and places where cars have left can be mistakenly recognized as cars. And also cars under the snow become invisible to computer vision systems.
Test 9. Google AI
Test 9. Microsoft Computer Vision
Test 9.SpotVision Car Detection
The same view, only in night mode.
Test 10. Google AI
Test 10. Microsoft Computer Vision
Test 10. SpotVision Car Detection
It turns out that Google AI and Microsoft Computer Vision work with a wider range of tasks, but have suffered an absolute failure in car detection. Most of the cases were completely or partially failed, some were recognized with blots. Therefore, they are not suitable for serving businesses that would like to expand their capabilities through integration with AI services. At the same time, the Belarusian SpotVision Car Detection service has coped 98% with the assigned tasks and is ready for real-time application.
If you want to test each service yourself, here are the links
Google AI
Microsoft Computer Vision
Spotvision Car Detection