Quite simply, BPA is such a centralized network that combines a delivery service, a huge warehouse and an order management system, with development and integration with partners. All this is tied up with reports and analytics. We design and write all services for the interaction of these systems and the provision of similar services to partners.
Our IT department meets the real world : it is easy to set a 15-minute delivery interval in an IT system, but it is quite difficult to make sure that thousands of sales representatives in Moscow come to the address exactly during this interval. The main task of our department is to link business and technology. The BPA department itself is now in three areas: delivery, warehouse and commercial function, which includes interaction with B2B and market place. And everything that we use ourselves and sell outside is really cool: our features bring money, save time and offer high-quality service to our customers.
5 languages and 2 million lines of code
BPA is a set of systems that are integrated with each other and other Lamoda systems. For their development, we use PHP, Java, Kotlin, there is a little Go and Typescript - there are 5 languages on our technical radar. We write all systems for the tasks of our business. Now it is 2 million lines of code, 25 services and more than a hundred reusable libraries.
The main language is PHP, and several other large services are written in Java and Kotlin. Why PHP? Firstly, it implements an order processing system - a monolith, which we have been sawing into microservices for many years, and secondly, we are great at working with PHP: we write quickly in it, we serve it well, we have settled it with a bunch of libraries... This approach allows us to launch new services with all the infrastructure connections: with all the logs and monitoring, in just a few days. We can use both Go and Kotlin as in a warehouse. We do not use duplicate technologies, and everything else is not prohibited, as long as it works.
In business processes, the logic and thoughtfulness of each step is important. So that it does not happen that your order went from the warehouse to Vladivostok, and there an unexpected plate suddenly popped up in the system, and it was not given to you.
We must cover each step with tests, since the stability and correctness of the execution of the system logic is critical. Moreover, our partners also use our business processes. Therefore, the company's priority is stability, despite the fact that the speed of launching new services is also important for business. To ensure stability, we invest time in designing services and tests around them. This gives us the opportunity to develop the system without risking breaking everything.
There is even buckwheat!
I am Anton Dmitrienko, technical lead of WMS - Lamoda distribution center management system or, more simply, warehouse.
My main tasks are to speak the same language with business, develop a domain model and prepare the system for changes. To do this, you need to understand where our "hot spots" are. There are processes that are lined up once every three years, and there are those that change every three months - these are “hot spots”. They are important to consider when designing a system.
Now I have four teams - 18 people who are engaged in the creation, testing and support of the system. We accelerate all operational and business processes of the warehouse: without us with the current volumes of Lamoda's business, the shipment of one customer order would take a lot of time, our participation reduces the processing time of any order to 4 hours.
The scale is, of course, very large: the Lamoda warehouse is about 40 thousand square meters or several football fields in 5 floors, where 10 million goods are stored. It works around the clock 364 days a year, with a day off once a year - January 1. And every day more than 500 employees leave for a shift, and 200 thousand goods are shipped and received through our system. Plus, a network of mechanized conveyors and equipment is deployed inside the warehouse, which are designed to transport goods, pack parcels, physically consolidate customer orders, sort parcels by destination, we also manage them.
Lamoda sells to customers all kinds of goods, from oversized umbrellas and suitcases to small watches and jewelry. All this is stored and processed in our warehouse. Each category has its own characteristics. For example, oversized goods cannot be processed on automated equipment, jewelry is stored and processed with additional security measures, and cosmetics, perfumes, for example, require certain temperature storage conditions and control of the shelf life of goods.
We recently had a project called "Buckwheat". Since the introduction of the self-isolation regime, Lamoda has added popular long-life foods to its range. By the way, we implemented this feature in just a few days. Of course, the main assortment is clothes and shoes, the work with which also has some nuances: for the sale of goods it is necessary to prepare and check their quality: it will be bad if we, for example, bring a shirt without buttons to the client.
Lamoda is not just an online store, but a full-fledged e-commerce platform that provides its infrastructure and services (including a warehouse) to external partners. For example, large international fashion brands store their goods with us. What difference does it make? Dresses, coats and shoes from partners are the same as those that are already in our warehouse. But we all have individual requirements for the processes that we are obliged to comply with under contracts, for example, some of the goods of large customers are stored in separate zones, and things from the marketplace are located throughout the stock. These features must also be taken into account.
We check and process each product at every stage of the process taking into account the specifics not only in the physical world, but also in the IT system. Moreover, the IT system itself prompts the warehouse employee how to handle a specific product, since the operator cannot know all the features or agreements: the product does not say who it belongs to: a large international company or a small marketplace partner.
The key feature of the system is interaction through scanners and barcodes: each product has a unique barcode, which can be read to obtain all the necessary information about the product. This approach minimizes errors and allows you to carry out all actions with the product faster, reducing the number of employees. If we ship 200 thousand goods per day, then 1 second saved on one product is 55 hours of working time. For business, it is tangible.
Under the hood of the warehouse
Our technologies:
Java 8,
PostgreSQL,
Wildfly,
Spring,
Redis,
ActiveMQ,
Hibernate.
We started with a ready-made solution in Java, but since then the functionality of the system has greatly expanded and the code has already been completely refactored and rewritten several times.
Now we have 350 thousand lines of code - and this is only the backend of the main application, and there are also 5 secondary services and 2 user interfaces: a thin web client and a mobile native application. We are doing full-stack development.
The warehouse employees use the mobile client. They move around the warehouse using our own Android application, this is a more modern and convenient technology that allows you to use the features of the device itself: vibration, sounds for errors, and the battery is spent less if the application is native, and not browser-based. The application itself is written in Kotlin, we have good experience using it. We even started writing a backend on it: so far this is only one service, but we like that the code is simpler and more expressive than in Java, and the performance is the same. We do not have a goal to rewrite everything in Kotlin, but we will continue to use it for suitable tasks.
Interaction with other Lamoda systems is built through a data bus written in Apache Camel using the ActiveMQ message broker. This approach provides guaranteed delivery and relatively simple transformation of messages for various formats, the ability to distribute messages to several consumers. We were the first to use Apache Camel back in 2014, then this method of integration was adopted by other teams in our department.
The principles of building warehouse systems:
· reliability and quality
· extensibility and flexibility
· simplicity
The warehouse is at the heart of Lamoda's logistics processes, and errors and system downtime have a major impact on business. Therefore, we pay close attention to the stability and fault tolerance of the system. Reliability is achieved by duplicating critical system and infrastructure elements. We have two independent servers that are sliced into virtual machines. Each application is deployed on two instances, each instance on its own virtual machine. The client interacts with the server through a balancer in the form of Haproxy, which routes requests to one of the instances. The database is also backed up by using the Master-Slave replication scheme.
All our servers are physically located right at the warehouse, which also increases the reliability and speed of work due to the interaction of all elements of the system over the local network. Even if the external Internet falls off, we will still continue to work, and in case of a power outage, we have uninterruptible power supplies and gasoline generators right in the warehouse.
To minimize the number of errors in the code, we write automated tests: unit, integration, acceptance, load, and the testing team checks everything by hand. All this is a mandatory practice and is built into the lifecycle of the task, sometimes, of course, bugs come out, but we haven't had full rollbacks of releases for several years.
Expandability and flexibility - for us it is primarily a question of how much effort and time we will spend to change the system to meet the new requirements of business processes. Therefore, the development provides for the ability to quickly and easily change the code in the future. For example, any category of goods can be configured for a specific physical storage location, this allows you to quickly connect new categories of goods for sale. For each order, it is configured in what material it will be packed and what accompanying documents for it will be printed. And this is only a small part of the functions.
We have a complex subject area with a large number of business processes and a branched domain model. The simpler the code, the better, because it increases its readability, comprehensibility, and reduces the likelihood of errors. And it allows you to focus more on what function the code performs for the business.
Error insurance
My name is Denis Plisko and I am the Development Leader for Delivery Services at BPA.
We have created, develop and maintain the LM Express IT system. It serves to automate the work of transit warehouses, sales representatives and points of issue of orders. Our system knows everything about the arrival and departure of deliveries, deals with their processing, sells and issues already paid orders, accepts returns and many other related business processes. Also, in it we record and analyze the physical actions of our sales representatives and warehouse workers - we track everything that happens on the “last mile” before the goods are handed over to the buyer. Most of the technical processes that are in the information form in the Express system reflect the physical actions of a delivery service employee.
What does it look like? For example, an employee accepts a delivery, takes out a pallet - a box where packages with goods are, takes out packages, sorts them into shelves: every action he does is accompanied by an operation in our system. And this is where mistakes can arise: you can never be sure that even if the delivery employee “clicked” everything correctly, then it is so. As a consequence, one of the biggest parts of our process logic is warning and correcting errors: sometimes this can be done automatically, sometimes through the person himself, sometimes through the helpdesk or users who have more advanced rights. If not for these features, then our systems would be 5 times smaller.
The principles of building delivery systems:
· stability and availability of systems
· data consistency
· independence from external systems and services
The task is to ensure the most stable operation of the systems, since whether people receive their purchases on time or not directly depends on this. When implementing any functionality, we try to be independent from external services and systems, for this, many interactions are duplicated. Suppose a delivery has arrived, and there is no order data for it, then we will request this information ourselves so as not to stop work. It is also important to make many operations idempotent in order to maintain the necessary data consistency, especially in exchanges with other services.
Moreover, we strive to receive all the necessary data as early as possible, so that it does not turn out that a person has come for a product and cannot receive it due to a system failure. We strive to make the internal parts of the system extremely reliable so that the user's interaction with them depends only on our own services. Thus, even if all Lamoda systems become temporarily unavailable, the work of warehouses and pickup points will not stop and the main operations will continue.
On the streets couriers, couriers
In addition to its own delivery, Lamoda cooperates with third-party courier services, for example, Russian Post, Pony Express, DPD and others. This happens when in a city, region or country (for example, in Ukraine), our delivery is possible only through partners. We provide information exchange through interaction with their IT systems.
Connecting each new courier is an exciting adventure, since the APIs are very different for everyone and are not always adapted for integration. This is because some of the courier services are historically offline companies. They've been shipping since the days of paperwork, and writing a good API is no easy task.
We use a special system based on the ESB framework Apache Camel to implement the integration with the new partner. Using Camel makes it easy to perform a wide variety of data transformations and interactions over different exchange protocols. Typically, setting up integration with a new courier service takes from two weeks to a month.
Delivery is one of the services that we sell to our partners along with the warehouse. When ordering a product from an online store of a popular clothing brand that does not seem to have anything to do with Lamoda, the purchase will be delivered by a sales representative in the form of an LM express. We are working on these projects together with the commercial function team.
To implement projects of our business, I and our technical leads from 2 teams (the third is now hiring developers and testers) are determining how best to fit all changes into our current architecture.
We work on the basis of backlogs from the list of projects. Business evaluates each project in terms of financial and marketing effect. We estimate labor costs and think over a technical approach: is it possible to create a new feature quickly and efficiently, or, on the contrary, it will be so expensive and difficult that it will block the entire business effect. Based on these estimates, we adjust the backlog and make projects at the top of the list. About 30% of our development time is in the technical backlog, so you don't have to find time to refactor and improve systems from a technical point of view, everything goes according to plan.
The inner world of delivery
:
· PHP,
· Java,
· Kotlin,
· TypeScript,
· MySQL,
· PostgreSQL,
· RabbitMQ,
· ActiveMQ,
· Apache Kafka,
· Apache Camel,
· Docker, K8S.
Most of our services are built on PHP. Historically, the systems on which Lamoda was created were written in this language. One of the first versions of the site was created on Magento, then Rocket Internet systems were taken as a basis for delivery and order processing. They have gone through many iterations of development and rework, but PHP still underlies almost all services, including new ones. We have already written about 200 thousand lines of code, and many experienced PHP developers (the delivery direction together with QA engineers numbers 20 people and is actively expanding) - but they are not limited to one technology, they are excellent engineers who choose the right tool for the task.
We are not going to abandon PHP in the backend, as we have a well-developed development culture on it: we use fresh versions of the language and frameworks, cover almost all the code with tests, conduct a thorough code review, have a well-built CI and many common libraries that speed up development process.
Unlike online-shop, we do not have large loads associated with RPS. First of all, we need a complex and flexible application architecture, with a large number of abstraction levels (we try to adhere to DDD) for complex and sometimes confusing business logic. We also have a huge number of both synchronous exchanges (JSON-RPC, SOAP) and asynchronous exchanges (RabbitMQ, Apache Kafka) with other systems.
In total, about 80-85% of systems are written in PHP, although we sometimes use Java. It contains a service for interacting with external delivery services, which we are constantly changing.
The frontend is represented by a bundle of VueJS + TypeScript. Our admin panels for internal users do not require super-layout or complex design.
We also have a mobile native application for sales representatives, which is written in Kotlin for Android. It was once developed by dedicated mobile developers, but now we are very successfully developing it on our own.
For employees who work in transit warehouses, from where orders are sent to all warehouses and pickup points in Russia, we have a web application. With this approach, there is a risk of unavailability of our application due to the disconnection of the Internet at a specific pickup point or in a warehouse, but we minimize it by using the reservation of communication channels.
In addition to these delivery systems ("Express" and a mobile application, a bus for integration with external courier services), we have two more services that our team currently supports: a service for printing electronic receipts and Datamatrix... The latter is intended to ensure the marking of goods with unique codes in accordance with the new requirements of the laws. These two projects are not directly related to delivery, but it is through our services that the data necessary for them is exchanged.
The delivery service prints receipts: each sales representative has a cash desk, plus there are cash desks at each of our pickup points. Therefore, the fiscalization service constantly interacts with our system. Datamatrix affects almost all BPA systems (warehouse, shipping, partnerships, accounting systems). Our division is also very actively interacting with him, so we took an active part in its development. Right now we are developing a new system for incentive policies for sales representatives.
The commercial function of BPA: what we sell
The company has an ambitious goal - to be the best fashion platform in the CIS. For this, Lamoda invested a lot in the quality of service and raised operational processes from scratch: delivery, warehouse, photo studio, work of contact center operators. All operational processes have been tested and tested on our clients. We experimented, monitored customer feedback, made mistakes, fixed bugs, continually improved everything that could be improved in the online shopping experience. In the fifth year of the company's life, we made the decision that we are ready to share our resources and expertise with other market players.
So we have the first B2B direction, which provides large partners with photography studio services, storage and delivery of goods, and customer calls. Soon, a Marketplace appeared within the company, for medium and small companies, which provides a Lamoda showcase as an additional service to the previous list. Both directions worked and developed practically in parallel, catching up and surpassing each other in the range of services.
After another 3 years and after evaluating the results, it was decided to combine B2B and Marketplace into a commercial function, which is engaged in the creation, development and support of services for external partners, based on the existing internal Lamoda services.
Now we are cooperating with about 20 large brands - these are all global companies with a worldwide reputation that have their own complex business processes. Another 1000 of our clients are medium and small companies, which often do not have a developed IT infrastructure. For all partners, we on our side maintain a separate integration layer, which is designed for quick and flexible development of functionality that converts external processes into ours.
I am Alexey Felde, a solution architect, and I am responsible for the technical and product development of the commercial function at Lamoda.
I am engaged in the development and support of services in the integration layer, and for this I must thoroughly understand the business processes of the company and our partners in order to design any changes in Lamoda systems.
As I mentioned above, we are developing two main areas of interaction with brands: Marketplace for small brands, to which we provide SAAS solutions in the form of a separate WEB application, and B2B - separate large projects for large global brands, to which we additionally either fully provide our operating capacities, or we help to integrate with ours.
B2B direction is one of the most interesting in the company, but at the same time one of the most difficult. It happens that world fashion giants come to us with completely new requirements or restrictions of integration. In such cases, my team interacts with other Lamoda teams and we design changes to any required company service. To connect one of the leaders of the global fashion industry, we needed to immerse ourselves in all of its business processes, including accounting and operational. Within the framework of the project, we touched upon 15 of our internal processes: from the description of goods on the site to accounting and delivery. My team has the broadest expertise in operating Lamoda IT systems and business processes. This requires careful systematization and documentation of our knowledge.
For small manufacturers of clothing, footwear and accessories, which often do not even have their own IT department, we provide a boxed Marketplace solution that allows you to track sales, movement of goods, returns. It fully works at the facilities of our systems: photos of goods are taken by our studio and are stored with us, goods are physically not at the client's place, but in our warehouse, and are delivered by the Lamoda delivery service.
Another our boxed product with source code is a photo studio. It was bought by a large luxury retailer a few years ago. They had very different photos on their website, and this greatly influenced the sales conversion. Our studio conducted a large number of tests with photographs: which of them sells better (photos of men without a head or with a head, correct leg angles, etc.). The client really liked the results of these tests, and he wanted to apply our experience in his online store. To do this, he needed to build a studio similar to ours, where people would work, organize the photography process, and install our IT system.
Lamoda in miniature
In contrast to service teams, we have a different task - the development of an integration layer behind which hides an enterprise of a large company. The main product is a B2B platform, which at the code level is internally divided into modules: a module for working with a warehouse, a module for working with delivery, and so on. It is one big system that at the level of these small modules can manage certain services. It turns out that our platform is a miniature Lamoda. At the same time, partners see only one API - universal and very simple, behind which all the complexity of Lamoda is hidden. At the same time, we must be as simple as possible for integration - our first principle in development, then comes stability and reliability. We support various SLAs, for example, one of them is a 4 hour maximum system downtime per year. Like colleagues,we pay a lot of attention to the tasks of fault tolerance, release processes, code quality, coverage by functional tests. Our B2B platform is almost the first system on the list in terms of the number of tests, we have over 2,000 functional tests and 3,000 unit tests.
B2B :
·
·
·
·
:
· PHP,
· Java,
· JavaScript,
· GO,
· Symfony,
· Camel,
· PostgreSQL,
· AngularJS,
· Apache Kafka,
· RabbitMQ.
Before connecting the brand, we agree with the partner on all the details of the relationship: the range and expected quantity of goods, the geography of delivery, the set of connected services. Next, the IT component comes into play: the integration details are discussed with the brand and we help to connect to our API. If something differs from the standard integration scheme, then we additionally raise services that are middleware between partner systems and our B2B platform.
90% of the time we upgrade our platform and implement new business processes. The last of the implemented business processes is working with Datamatrix codes, which was already discussed in another article , and colleagues mentioned above.
In our work, the main thing is to maintain a balance of technologies and choose tools for the tasks being solved. We honestly don't chase fashion in the industry. Our stack is PHP, Java, Javascript and some Go.
As mentioned in the article, the company has historically written many systems in PHP. It is convenient and fast to program business processes on it. The basis of our B2B platform is written on it - authorization, role management, all the main business processes. It is based on hexagonal architecture, active use of DDD and API first design. We support Single Page Application written in Angular 1.5. It communicates with our B2B platform via API and is primarily responsible for rendering the UI for our employees and partner employees.
We also use Java, in particular the Camel integration framework. Brands are not always ready to use our API, and for very large players we write middleware that converts the domain of the brand into the domain of Lamoda. Camel saves time developing systems integration solutions, changes the mindset of people towards good engineering practices using enterprise integration patterns (EIP). We plan to develop Camel in places where it is more convenient to use integration patterns than to write a separate service.
We actively use Kafka to exchange data between internal systems. This significantly reduces the connectivity between systems, because in the specifics of our work we need to store snapshots of data from all systems sharpened for partners. For everything that needs to be done quickly and for office tasks, we write in Go. One of our services calculates metrics every minute based on the stored data. Previously, this solution was built into a B2B platform and was written in PHP, but we rewrote it to nimble Go, since it has higher performance, easier support, it scales better and can be deployed outside the release cycle of the B2B platform.
I am sincerely happy when I see the result of the work of our platform on 30+ third-party sites of large and famous fashion brands. I see how an order, which is placed in our systems from someone else's site, is processed by our contact center operator, and then it is transported by our delivery. Nobody knows that all these processes are created and automated in Lamoda, but we do it so cool that our processes are bought by global brands.