M.Video-Eldorado Group presented its Hacking Retail strategy in early 2021. Over the next 5 years, we plan to double the total turnover to 1 trillion rubles and triple the assortment, including through the development of our own marketplace. “Monolithic” IT systems are not able to provide this growth in a short time, and non-standard developments slow down the whole process. Microservices come to the rescue. We will tell you how we "sawed off" a piece of SAP ERP, and what came of it.
A couple of years ago we completed the process of merging the M.Video and Eldorado back offices. As a result, the united IT infrastructure of the retail network began to represent a three-layer structure: at the first level, a united back office, the main accounting system of which is a “monolithic” SAP ERP.
Above it is a constantly expanding microservice platform that provides key data to all front systems (prices, promotions, products, balances, customer information, etc.) and provides integration of front systems with back systems. Upstairs, this scheme is crowned with two separate front offices for each of the brands.
In general, this bundle meets the current needs of the business. But they are constantly growing, so we are constantly looking for suboptimalities, the elimination of which could improve the performance of business processes and prepare them for increased workload.
Prepare seriously. M.Video-Eldorado has big plans to expand the range of goods and launch new projects. The expected growth in supply volumes should be two to five times. And not someday, but within the next year or two.
Long hours of waiting
One of the bottlenecks that needed to be eliminated as soon as possible was the algorithm for calculating logistic calendars. Let us explain in a little more detail. There is a concept of logistical accessibility. It determines at what moment the goods can be delivered from point A to point B. The logistic accessibility parameter is used by all front offices, business applications refer to it, etc. - this is one of the main parameters of the product as a logistic unit.
The calendar contains complete information about the possibility of moving goods between the supplier, central and regional warehouses and stores in both networks. If the shipment is scheduled for today, today's calendar is used. But tomorrow it will lose its relevance and it will be necessary to apply "tomorrow".
To increase the stability of the work of logistics processes, the calculation was performed daily for three days in advance. This solution has long been delivered as a "crutch" in SAP ERP due to the speed of implementation. The process took 9-11 hours! Testing under a higher load showed that the system could not cope with the calculation even in 24 hours. To achieve a qualitatively higher productivity with a significant increase in volumes, it was important to take this service out of the monolith ...
Current situation
The algorithm for calculating logistics calendars, which is part of SAP ERP, was a package of PL / SQL procedures provided by SAP in the form of a customized box solution. The calculation of each of the three calendars in it was carried out "from scratch", taking into account all the conditions for each supply chain. Accordingly, most of the calculations at each stage simply duplicated each other. This was the logic of the product's operation, and we did not have the resources to fully eliminate the technical debt.
In addition, the algorithm left duplicate "garbage" supply chains in the finished calendars. This led to an increase in the size of the database. Instead of the required 10 million records, the database could contain 20-30 million and required manual cleaning.
The original tool did not come to such an optimal state in one day. It was implemented a very long time ago, and throughout the entire period of operation, numerous improvements were made to it, initiated by business customers. A stream of gradual changes over the years led to the emergence of various inaccuracies, some of which were not fully reflected in the specifications.
At the same time, it was about a critical business tool, on the performance of which the activities of hundreds of stores in the retail network depended. Therefore, the approach to product development had to be very careful, but prompt.
Gnaw off a piece of the "monolith"
Realizing that the potential for SAP ERP productivity growth in this area (by the way, not typical for an ERP service) has been exhausted, we turned to microservices. M.Video-Eldorado has been developing this direction for several years.
We have our own micro-service platform running over a hundred microservices. It was necessary to create a number of services based on the same algorithm as the original module of the ERP system. We wanted to keep the general logic of calculations, but avoid repeating the same processes.
The work was carried out in three main directions. First, it was necessary to optimize the SAP algorithm by eliminating unnecessary calculations from it. Secondly, it was necessary to eliminate the appearance of duplicate supply chains, leading to the accumulation of unnecessary records in the database. Thirdly, it was necessary to eliminate inaccuracies in the results of the original algorithm, which complicate the optimization of business processes.
Recall that SAP ERP is one of the main back-systems of the company, which receives data from all front-systems. Despite the fact that the calculation of calendars had to be transferred to microservices, the individual purchasing steps themselves, the movement of goods were still created in the central accounting system.
To generate logistics calendars, SAP ERP provides more than 20 tables with settings received from business users. It is on the basis of this data that the micro-service platform performs calculations. ERP itself also uses them for internal calculations when creating second and third delivery arms. Therefore, it was impossible to simply rip the entire process out of the monolith and transfer it.
Process diagram:
1 and 3 - The process through SAP PI receives data with settings for calculating calendars from SAP ERP;
2 and 4 - Recording the data received from the ERP into the database;
5 - The process takes one of the versions of the data stored in its database (by default - the latest version). It then calculates the calendars based on this data and stores them in a staging table. In the course of work, the process asks for information on objects.
6 - The data is transferred from the staging table to the productive table in the same format in which the process receives it from SAP ERP.
7 - Deleting old data with calendar settings from the database.
Tools, time, resources
Since we have been creating microservices for several years, we had no particular doubts about the choice of development tools. The case went well mastered a bunch of Java 11 and the Boot Spring
2. No magic in the code, the only standard for Java things -. Collections, interfaces, etc.
The new algorithm we originally built so that all operations are carried out with the tables completely in memory, rather than using SQL queries inside the database.
On the part of the SAP ERP operation team, preparations were made for transferring master data to the microservice platform for calculating calendars. It took about 40 hours.
In the future, they only needed to support the project. The main work, of course, fell on the shoulders of the engineers of the platform itself. The logic of the algorithm was essentially re-created. From the ERP, only master data and basic business principles for calculating calendars were taken. This took another 150 hours.
In general, the implementation of the project took about six months. We spent two months alone on numerous autotests based on different approaches. Piloting of the new instrument continued for about a month. During this period, we simultaneously calculated two types of calendars - old and new, studied and compared the results, it was critically important to make sure that the service did not deteriorate and the calculation was correct. Only after that it was decided to transfer the new tool to the production environment.
results
What did we get in the end? The calendar is now calculated only once. Then its instances for each of the three days are obtained as a result of a little additional calculations associated with changing dates. The volume of the database is automatically maintained in an optimal state due to the absence of a "crutch" add-on, the load on the ERP system has decreased.
But the main result that we have managed to achieve is the reduction of the calendar generation time from 9-11 hours to ... 10 minutes! There is a lot of work ahead, the future multiple increase in the load on the IT infrastructure will still require improvements, but we won this piece of technical debt.