Microsoft's new Flight Simulator is a technological marvel that sets new standards in the genre. Microsoft and Asobo Studios used the work of many partners to recreate a world that appears to be real and alive, containing billions of buildings in the right places.
One of them is a small Austrian startup Blackshark.ai from the city of Graz, which with only about 50 employees was able to recreate every city in the world using artificial intelligence and huge cloud computing resources.
Prior to the release of the new Flight Simulator, we met with Blackshark co-founder and CEO Michael Patz to discuss the partnership with Microsoft and the future of his company.
Blackshark is an offshoot of the game studio Bongfish , which developed World of Tanks: Frontline, Motocross Madness, and the Stoked snowboarding series. It was thanks to Stoked that the company created Blackshark, Patz says.
“One of the first games we developed in 2007 was a snowboarding game called Stoked and S Stoked Bigger Edition. It was one of the first games with 360-degree simulated mountains: the player could fly around the mountain in a helicopter, land anywhere and roll down. The mountain itself was created and described procedurally, and obstacles in the form of vegetation, other snowboarders and small animals were placed. Then we moved on to the genres of racing, shooters and car control, but did not forget the idea of procedural placement and description of objects. "
Bongfish came back to this idea while working on World of Tanks, because creating huge maps with each stone placed by hand would be very slow.
Taking advantage of its experience, Bongfish began building its own AI development department. This department used a set of machine learning techniques to build a system that can learn how designers build maps and, over time, can create them on its own. The company has already started using this system in several of its projects, after which Microsoft contacted it.
“Quite by accident I met people at Microsoft looking for a studio to help them with the new Flight Simulator. The main idea behind Flight Simulator was to use Bing Maps as the playing field, map and background , ”says Patz.
But Bing Maps' photogrammetry data was able to create accurate replicas of only 400 cities, and for most of the planet's area, such data did not exist. Microsoft and Asobo Studios needed a system to build everything else.
It was then that Blackshark took over. For Flight Simulator, the studio recreated 1.5 billion buildings from 2D satellite images.
While Patz said he met the Microsoft people by accident, it's worth adding that Graz once had a Bing Maps department that developed the first cameras and 3D versions of Bing Maps. And while Google Maps took over the market , Bing Maps actually surpassed Google in 3D. Then Microsoft opened a research center in Graz, and after it closed, local talent was poached by Amazon and other companies.
“Therefore, it was very easy for us to find people in positions with requirements like 'PhD in roof reconstruction',” says Patz. "I didn't even know that such a thing existed, but it was exactly what we needed, and we found two such people."
“It's easy to see why recreating a 3D building from a 2D map is challenging. It is not even easy to define the exact outline of a building. "
“Basically, we did the following task in Flight Simulator: we studied 2D areas and looked for signs of buildings on them, which is the task of computer vision,” says Patz. “But if a building is obscured by the shadow of a tree, then we need machine learning, because the shadow overlay makes it unclear what is part of the building ... Machine learning can restore the rest of the building. And this is another very simple example. "
While Blackshark may have used some other data, including photographs, sensor data and pre-existing map data, Blackshark had to determine building heights and other characteristics based on very limited information.
The next obvious problem is determining the height of the building. If GIS data is available, this task is easy to solve, but for most areas of the world, such data simply did not exist or it was not easy to get it. In such cases, the team took a 2D image and studied various clues, such as shadows, on it. However, to determine the height of a building from the shadow, you need to know the time of day, and images in Bing Maps were not timestamped. In other cases, Blackshark had them, which greatly simplified the work. This is where machine learning comes to the rescue again.
“Machine learning takes a slightly different path,” explains Patz. “It also takes shadows into account, but since it's just a dark figure, we don't know how it behaves. However, if you look at a flat roof, for example, compare a skyscraper with a shopping mall, the equipment on the roof of a skyscraper is different from the equipment on a shopping mall. So by tagging buildings we help AI learn. ”
If the system knows that the average height of a shopping mall in this area is usually three etaea, it can work with that.
Blackshark does not hide that its system will make mistakes, and if you buy Flight Simulator, you will see that there are errors with the placement of buildings in the game. Patz informed me that one of the biggest challenges for this project was convincing the development partners and Microsoft to allow this approach.
“We're talking about 1.5 billion buildings. At this scale, you can no longer rely on traditional quality control. The traditional approach of Halo games is that you point your finger and say "this pixel is bad, fix it", but it doesn't work when you are developing with AI based on statistics. It may turn out that 20% of the buildings were created in error, and this is probably the case in the case of Flight Simulator; but we could not have solved this problem otherwise, because outsourcing the modeling of 1.5 billion buildings is neither logistically nor financially possible. "
Over time, this system will improve, and with Microsoft streaming large amounts of data from Azure, users will surely see the changes.
However, markup is just the way the studio team trains the model, and Blackshark has made a lot of progress in this area. Patz does not go into details, because it is part of the company's secret know-how, thanks to which it was possible to carry out such a volume of work with the efforts of only 50 people.
“Data labels have not been a priority for our partners,” he says. “Therefore, we used our living marking system, essentially marking the entire planet with the forces of two or three people. This provides data analysts with a very powerful tool and user interface. For example, if an analyst wants to detect a ship, he tells the learning algorithm what a ship is, and then instantly receives the ships found in the sampled image at the output. "
Based on the output, the analyst can train the algorithm to better recognize specific objects, in our example, ships or shopping malls in Flight Simulator. Patz says that other geospatial analysis companies tend to focus on specific niches, and Blackshark's tools are independent of the type of content being analyzed.
This is where Blackshark's broader vision comes into play. As the company is now receiving good reviews for its work with Microsoft, it is also partnering with other companies that are remodeling city models, such as autonomous driving simulations.
“Our larger goal is to create a near real-time digital twin of our planet, and especially the planet's surface. This will allow the data to be used in many different ways in cases where traditional photogrammetry like Google Earth or Apple Maps does not help, because it is simplified to the level of photographs superimposed on simple geometric shapes. To do this, we have a processing loop: we extract exploratory information from aerial imagery, which can be 2D images or even sets of 3D points. And after that we visualize the semantics. "
This semantics, which describe the building in great detail, has one major advantage over photogrammetry: in fact, the information about shadows and lighting is baked into the photograph, which makes it difficult to realistically superimpose different lighting. Since Blackshark knows everything about the building she is creating, she can apply windows and lighting to those buildings, which creates surprisingly realistic night scenes in Flight Simulator.
Point clouds, which are not used in Flight Simulator, are another area that Blackshark is currently actively pursuing. Point clouds are very difficult for people to read, especially if you get close to them. Blackshark uses its AI systems to analyze point clouds to determine the number of floors in a building.
“At the heart of our entire company is the understanding that we need a tremendous technological advantage to accomplish a task. This is especially true in the video game industry, where projects as large as Assassin's Creed and GTA have faced the limits of the possible: thousands of people work on them, this work is very difficult to scale and coordinate between continents, while turning it into a finished product. It was obvious to us that a more automated or partially automated process was required to solve such a task. ”
Although Blackshark was founded as a gaming company and now partnered with Microsoft and Asobo Studios, its primary focus is not on gaming, but in areas such as autonomous driving and geographic data analysis. Patz notes that another good example of this development is the Unreal Engine, which was originally a game engine only and is now used everywhere.
“For a long time in the gaming industry, this situation is very inspiring because when you develop games you realize how revolutionary technology can be compared to other industries,” says Patz. “And if you look at the simulators, from the military to the industrial, they always looked lame compared to the racing games. Now is the time for gaming technology to get out of the gaming industry and start helping all other industries. I think Blackshark has become one example of how to do this. "