Clever Geek Handbook

Wargaming Platform: Distribution

Hello!



About two years ago, my colleague Maxim (max_posedon) published an article Wargaming Platform: Hello World , in which he "tried" (as he himself designated) to explain what the Wargaming Platform is. My colleagues and I want to continue to share information and this time we'll dig a little deeper - as the title of the article suggests, into distribution.



Everyone is familiar with digital distribution, we use it regularly, we have seen everything over the 20 years of its existence, so I will bypass the theory and examples of the market, and I will talk more about the specific implementation and experience.



Generalized distribution scheme for Wargaming



Let's start right away with the picture: it shows both the main components of the Wargaming Distribution System (WDS) and the Data Flow of the content.





Figure 1



The process can be briefly described as follows: the game studio (Game Studio) uploads the next assembly of the game to the WDS server, which, after the necessary processing (more on this in the "Under the hood" section), through the distribution channels (components marked in blue) distributes them to the client WGC applications.



WGC (Wargaming Game Center) is a desktop client application, a means of user interaction with the Wargaming Platform, in particular with the distribution system. The WGC downloads updates to the user's machine from currently available sources.



WDS has three main distribution channels :





Yes, the delivery of updates is based on BitTorrent technology. I must say right away that p2p data exchange between clients is, in absolute terms, the minimum channel in terms of traffic volume. Its main function is to accelerate loading in local networks with a large number of WGC clients (for example, Internet cafes) and reduce the load on the external network interface. Also, the use of BitTorrent makes it easy to organize a point of presence in any desired place.



The distribution channel Static Seeds is interesting from a historical perspective. Take a look at the bar chart below: it shows the dynamics of the minimum cost of CDN traffic for the largest consumers (i.e. the wholesale, lowest price).





Figure 2



In 2010, at the dawn of World of Tanks, a gigabyte of traffic cost about $ 0.2 for orders over 10 PB. With regular updates of AAA games with millions of audiences, the cost of traffic turns out to be quite impressive. The ability to easily add your node to any point (from the point of view of geography / providers) allows you to pull tangible volumes of traffic to it. Depending on the presence of a CDN at one point or another, the load on the system during releases, the volume of releases of a particular game, such a tool has a good economic effect even today.



To make the basic distribution scheme completely understandable, it is worth saying a few words about the component, which in Figure 1 is designated as the Wargaming Distribution System backend. The main tasks of the component:



  • prepare updates, ensuring the minimum amount of content downloaded and installed by the client;
  • distribute updates through distribution channels;
  • provide the client with information for updates.


Now let's take a quick look at the distribution options.



Functionality



BitTorrent technology, and in particular the implementation of libtorrent (thanks to Arvid Norberg), "out of the box" allows you to implement with minimal costs:



  • local retracker and seeds - relevant for both user groups and game studios;
  • connecting multiple CDNs - horizontal scalability;
  • weighting factors for different CDNs - balancing, "spare" CDNs;
  • integrity check and restoration - game files can be damaged by negligence, cleaning applications, antiviruses, problems with drivers or physical media.


Regardless of libtorrent, but also about downloads:



  • Preloading updates. The game studio can publish updates only for download (without installation): at the time of the release of the new version, they will be applied, users will start playing the new version earlier.
  • Preload balancing. With large volumes of updates, you can evenly "smear" the preliminary receipt of updates to reduce the load on the distribution nodes and smooth the peak.


Installing / updating games:



  • Patches. The user downloads only the missing parts of the game, well compressed.
  • DLC. Some of the content can be downloaded optionally at the user's choice or depending on the logic of the game.
  • Client types. The game can give the user a choice of which "type of client" to download (for example, World of Tanks: there is a choice between standard textures (SD) and high-definition textures (HD), which allows users who do not have the system do not support maximum graphics settings. ).
  • Mini client. The ability to play until complete installation, the game studio has the ability to determine the minimum required set of files to start the game.


Other:



  • Installing redistributables.
  • Running arbitrary installers during install / uninstall processes.




Under the hood



Parts & Patches



The game can be divided into parts (parts), which specify a certain sequence of installation. This enables the game studio to:



  • deliver content to the user only for the selected game language;
  • implement the minimum necessary part of the gameplay to start the game and deliver it first;
  • designate some parts as DLC and allow the game / user to request them separately.


To install a part, one or more patches must be delivered and applied. At its core, the patch is a 7z archive with the LZMA2 codec. The compression level can be changed depending on the type of game content. To install a game from scratch, a patch must contain all the content for a specific part of the game. If this is an update, then the patch contains only the required difference. This difference is calculated file by file using the following algorithms:





The first of the algorithms is block-based, and the second provides the ability to copy parts of arbitrary length and repeat. Depending on the nature of the changes (insertion, deletion, replacement), the place in the file where they were made (beginning, end, middle), their size, file redundancy, one of the algorithms can greatly outperform the other, or in terms of the size of the resulting diff file, or by the speed of its application on the user's machine. WDS chooses the optimal binary difference algorithm for a particular case.



Torrent



To download patches, torrent files are prepared by version, the WGC client receives the necessary information from the WDS backend and starts a torrent session with the necessary torrents.



Seeds are, as already indirectly mentioned, CDNs (webseeds), Static Seeds (specially prepared machines with libtorrent-based clients), the WGC clients themselves, and any torrent clients with the necessary torrents. Receives these seeds a torrent session from the following sources:



  • WDS backend;
  • torrent tracker;
  • DHT ;
  • LPD ;
  • PEX .


Some providers block either the BitTorrent protocol itself or the download of a * .torrent file. Since the download from webseeds occurs via HTTP, and the * .torrent file itself is downloaded via HTTPS, such locks are not critical for us, the game will still be delivered. In the worst case, traffic from Static Seeds and p2p will be lost. There have been cases in the United States when a provider demanded the installation of its certificates and filtered, including HTTPS traffic (not very nice), found a torrent file there and blocked it. But suddenly changing the extension of the torrent file allowed it to be downloaded, that is, the filters were only at the content name level.



The WGC is currently establishing up to 50 distribution channel connections. Several connections can be established on the same CDN, which allows for more efficient utilization of the channel between its node and the client. In addition to this, the main CDN uses the BBR congestion control algorithm , which is considered greedy and utilizes the channel quite well. This configuration provides the highest bandwidth utilization for fast game delivery, but of course the comfort of using the Internet connection for other applications may suffer during updates. A series of testing of alternative configurations is currently underway, and in the near future we can expect a decrease in the number of connections on an ongoing basis with an increase in comfort, but without a tangible impact on the speed of delivery.



Just in case, here I would like to emphasize once again that any network activity can affect the quality of traffic for the game client for which it is critical. That is why, while the game is running, the WGC does not perform any network operations without an explicit need from the game client (for example, an authorization operation).



Installing updates



The processes of downloading and unpacking patches are organized in parallel, and it looks like this:



  • Start downloading the patch, changing the download priority to the part with the archive header.
  • After receiving the full header of the archive, the unpacking process starts and the data received from the network is immediately unpacked.
  • The unpacked data is saved to disk.
  • The downloaded patch is also saved to disk, since not all fragments are needed for unpacking at the time of receipt, and plus for distribution to others.
  • After processing, the patch file can either be left on disk or immediately deleted depending on the configuration given by the game studio.


The processes of unpacking and applying binary differences occur in a separate process, which allows you to control its system and I / O priority by means of the OS. On top of this functionality, there is a custom "turbo" setting that will speed up the installation process, but also compete strongly with other applications for resources. By default, the WGC runs in a low resource mode, relying on comfort, but upon first installation, the WGC explains to the user that he has a choice.



Performance



Volume of updates



Minimizing the amount of downloaded data is one of the main tasks of our system. Unlike the Steam approach, in which the discreteness of the processed data of the game client is 1 MB, we often operate with bytes when constructing binary differences. In addition, blocks of these bytes can either be added to any part of the file, or removed from it. As a result, scheduled version updates of World of Tanks and World of Warships usually have a volume of 1–2 GB, while the same versions of games downloaded on Steam require about 5 GB to be downloaded. World of Warships is distributed via Steam, so this data is received regularly, and for World of Tanks the data is taken from special test downloads.



Required user disk space



For the minimum amount of downloaded data and p2p you have to pay with the disk space required for the operations to install and update the client. At the moment, the user needs more space than the game client takes: such a buffer can be up to half the size of the client. We are working on reducing this value, and it is possible in the near future to make it fixed and not exceed 2 GB, which becomes quite acceptable given the size of modern games. But there is another important feature: to update the game (which is a regular activity in the modern world), a relatively small buffer is also sufficient, which is already about 4 GB for most game patches. At the same time, some of these solutions require twice as much disk space as the game client takes.





We carried out a series of measurements to compare the installation speed of the World of Warships client in WGC and Steam at equal network channel speeds. When using the standard WGC installation mode, both systems show almost identical results (as far as possible to record in such experiments, even taking into account the cleanest systems, regular reboots, etc.). When using "turbo", the WGC wage average gain is in the range of 5-7 percent of the full installation time. We must also take into account the fact that the bandwidth of the network interface had to be underestimated for both systems in order to get the same conditions and more or less stable bandwidth. Taking into account the fact that the WGC utilizes the network channel more strongly (see the Torrent section),without additional restrictions on the custom machine, the speed of the initial installation of the game becomes noticeably higher.



Final word, or what was not here



The Wargaming platform, and in particular the Wargaming distribution system, are large software systems with a ten-year history and dozens of different technologies and programming languages. In this article, we got acquainted only with the high-level distribution scheme and its main characteristics. The system architecture, most of the technologies, development processes, releases, operations, geographic nuances of distribution, the WGC application itself with all its non-distribution functionality, and much more remained outside the brackets due to the "non-rubber" article. This only means that I am waiting for questions about the article and especially about what was not included in it, in the comments. Good code, good luck everyone!


More articles:


All Articles