We all love stories. We like to sit by the fire and talk about our past victories, battles, or just about our work experience.
Today is such a day. And even if you are not at the fire now, but we have a story for you. The story of how we started working with storage in Tarantool.
Once upon a time in our company there was a couple of "monoliths" and one "ceiling" for all, to which these monoliths were slowly but surely approaching, limiting the flight of our company, our development. And there was an unambiguous understanding: one day we will be hard against this ceiling.
It is now that we are dominated by the ideology of dividing everything and everyone, from equipment to business logic. As a result, we, for example, have two DCs that are practically independent at the network level. And then everything was completely different.
Today, there are a bunch of tools and tools for making changes in the form of CI / CD, K8S, etc. In the "monolithic" time, we did not need so many foreign words. It was enough just to fix the "storage" in the database.
But time went forward, and the number of requests went forward with it, sometimes firing RPS beyond our capabilities. With the entry into the market of the CIS countries, the load on the database processor of the first monolith did not fall below 90%, and the RPS remained at the level of 2400. And these were not just small selectors, but hefty queries with a bunch of checks and JOINs that could be run through almost half of the data on the background of a large IO.
When full-fledged sales on Black Friday began to appear on the stage - and Wildberries began to hold them one of the first in Russia - the situation became completely sad. After all, the load on such days increases threefold.
Oh, these "monolithic times"! I am sure that you have also encountered something similar, and still cannot understand how this could happen to you.
What can you do - fashion is inherent in technology. Even 5 years ago, we had to rethink one of these mods in the form of an existing site on .NET and MS SQL-server, which carefully kept all the logic of the site itself. He kept it so carefully that cutting such a monolith turned out to be a long and quite difficult pleasure.
A small digression.
At various events, I say: "If you did not saw the monolith, then you did not grow!" I am interested in your opinion on this matter, write it, please, in the comments.
A Sound of Thunder
Let's go back to our "bonfire". To distribute the load of "monolithic" functionality, we decided to divide the system into microservices based on opensource technologies. Because, at the very least, they are cheaper to scale. And the understanding that we would have to scale (and a lot) we had 100%. Indeed, already at that time it turned out to enter the markets of neighboring countries, and the number of registrations, as well as the number of orders, began to grow even more.
After analyzing the first applicants for leaving the monolith in microservices, we realized that in 80% of them, 99% of them write from back office systems, and read from the front-end systems. First of all, this concerned a couple of important subsystems for us - user data and the system for calculating the final cost of goods based on information about additional client discounts and coupons.
Indent. Now it's scary to imagine, but in addition to the aforementioned subsystems, product catalogs, a user basket, a product search system, a filtering system for product catalogs and various recommendation systems were also taken out of our monolith. For the operation of each of them, there are separate classes of narrowly sharpened systems, but at one time they all lived in one "little house".
We planned to transfer data about our clients to a sharded system. The removal of the functionality for calculating the final cost of goods required good read scalability, because it created the greatest RPS load and was the most difficult to implement for the database (a lot of data are involved in the computation process).
As a result, we have a scheme that works well with Tarantool.
At that time, schemes for working with multiple data centers on virtual and hardware machines were chosen for the operation of microservices. As shown in the figures, Tarantool replication options were applied in both master-master and master-slave modes.
Architecture. Option 1. User service
At the current time there are 24 shards, each of which has 2 instances (one for each DC), all in master-master mode.
On top of the database are applications that access database replicas. Applications work with Tarantool through our custom library, which implements the Tarantool Go driver interface. She sees all replicas and can work with the master for reading and writing. In fact, it implements the replica set model, which adds the logic for selecting replicas, performing retries, a circuit breaker and rate limit.
In this case, it is possible to configure the policy of choosing a replica in the context of a shard. For example, roundrobin.
Architecture. Option 2. Service for calculating the final cost of goods
Several months ago, most of the requests for calculating the final cost of goods went to a new service, which, in principle, works without databases, but some time ago it was 100% processed by the service with Tarantool under the hood.
The service database is 4 masters into which the synchronizer collects data, and each of these replication masters distributes data to readonly replicas. Each master has about 15 such lines.
Both in the first and in the second scheme, if one DC is unavailable, the application can receive data in the second.
It should be noted that replication in Tarantool is quite flexible and configurable at runtime. In other systems, there were difficulties. For example, changing the max_wal_senders and max_replication_slots parameters in PostgreSQL requires the wizard to be restarted, which in some cases may lead to a disconnection between the application and the DBMS.
Seek and you will find!
Why didn't we do it "like normal people", but chose an atypical way? It depends on what is considered normal. Many people generally make a cluster from Mongo and spread it over three geo-distributed DCs.
At that time, we already had two projects on Redis. The first is a cache, and the second is a persistent storage for not too critical data. It was quite difficult with him, partly through our fault. Sometimes quite large volumes were in the key, and from time to time the site felt bad. We used this system in the master-slave version. And there were many cases when something happened to the master and replication broke.
That is, Redis is good for stateless tasks, not stateful ones. In principle, it allowed solving most of the problems, but only if these were key-value solutions with a pair of indices. But at the time, Redis was pretty sad with persistence and replication. In addition, there were complaints about the performance.
Thinking about MySQL and PostgreSQL. But the first one somehow did not take root with us, and the second one is a rather sophisticated product in itself, and it would be inappropriate to build simple services on it.
We tried RIAK, Cassandra, even a graph database. All of these are quite niche solutions that did not fit the role of a general universal tool for creating services.
Ultimately, we settled on Tarantool.
We reached out to him when he was in version 1.6. We were interested in it in the symbiosis of key-value and the functionality of a relational database. There are secondary indexes, transactions and spaces, they are like tables, but not simple, you can store a different number of columns in them. But Tarantool's killer features were secondary indexes combined with key-value and transactional.
The responsive Russian-speaking community also played a role, ready to help in the chat. We actively used this and directly lived in the chat. And do not forget about a decent persistent without obvious blunders and jambs. If you look at our history with Tarantool, we had a lot of pains and fakups with replication, but we never lost data due to his fault!
Implementation started hard
At that time, our main development stack was .NET, to which there was no connector for Tarantool. We immediately started doing something in Go. Lua worked pretty well too. The main problem at that time was with debugging: in .NET everything is gorgeous with this, and after that it was difficult to plunge into the world of embedded Lua, when you, besides logs, have no debugging, was difficult. In addition, replication for some reason periodically fell apart, so I had to delve into the structure of the Tarantool engine. The chat helped with this, to a lesser extent - the documentation, sometimes looked at the code. At that time, the documentation was so-so.
So, within a few months, I managed to fill the shots and get decent results when working with Tarantool. We formalized the reference developments in git, which helped with the formation of new microservices. For example, when the task arose: to create another microservice, the developer looked at the source code of the reference solution in the repository, and it took no more than a week to create a new one.
These were special times. Conventionally, then it was possible to go to the administrator at the next table and ask: "Give me a virtual machine." Thirty minutes later you already had the car. You yourself connected, installed everything, and you got traffic to it.
Today it will not work like that: you need to wind up monitoring, logging on the service, cover functionality with tests, order a virtual machine or delivery to Kuber, etc. In general, it will be better, although longer and more troublesome.
Divide and rule. What about Lua?
There was a serious dilemma: some teams could not reliably roll out changes in a service with a lot of Lua logic. This was often accompanied by the inoperability of the service.
That is, the developers are preparing some kind of change. Tarantool starts the migration, and the replica still has the old code; some DDL, something else, arrives there by replication, and the code just falls apart, because it is not taken into account. As a result, the update procedure for the admins was scheduled on sheet A4: stop replication, update it, enable replication, turn it off here, update there. Nightmare!
As a result, now we most often try to do nothing in Lua. Just using iproto (a binary protocol for communicating with the server), and that's it. Perhaps this is a lack of knowledge among the developers, but from this point of view, the system is complex.
We don't always blindly follow this scenario. Today we don't have black and white: either everything is in Lua, or everything is in Go. We already understand how you can combine them so that you don't get problems with migration later.
Where is Tarantool now?
Tarantool is used in the service for calculating the final cost of goods, taking into account discount coupons, aka "Promoter". As I said earlier, now it is retiring: it is being replaced by a new catalog service with pre-calculated prices, but six months ago, all calculations were done in Promoter. Previously, half of its logic was written in Lua. Two years ago, a storage was made from the service, and the logic was rewritten to Go, because the mechanics of the discounts changed slightly and the service lacked performance.
One of the most critical services is the user profile. That is, all Wildberries users are stored in Tarantool, and there are about 50 million of them. A system sharded by user ID, distributed over several DCs with a connection to Go services.
According to RPS, "Promoter" was once the leader, reaching 6 thousand requests. At one point, we had 50-60 copies. Now the leader in RPS is user profiles, about 12 thousand. This service uses custom sharding with a division by user ID ranges. The service serves more than 20 machines, but this is too many, we plan to reduce the allocated resources, because the capacity of 4-5 machines is enough for it.
Session service is our first service on vshard and Cartridge. Setting up vshard and updating Cartridge required some work from us, but in the end everything worked out.
The service for displaying different banners on the website and in the mobile application was one of the first to be released directly on Tarantool. This service is notable for the fact that it is 6-7 years old, it is still in service and has never rebooted. The replication was master-master. Never broke anything.
There is an example of using Tarantool for quick reference functionality in a warehouse system to quickly double-check information in some cases. We tried to use Redis for this, but the data in memory took up more space than Tarantool.
Waiting list services, client subscriptions, trendy stories and shelved goods also work with Tarantool. The last service in memory is about 120 GB. This is the most extensive service of the above.
Conclusion
Secondary indexes combined with key-value and transactional properties make Tarantool great for microservice architectures. However, we ran into difficulties when rolling out changes to services with a lot of logic in Lua - services often stopped working. We did not manage to defeat this, and over time we came to different combinations of Lua and Go: we know where to use one language, and where - another.
What else to read on the topic
- We create from scratch a highly loaded application on Tarantool habr.com/ru/company/mailru/blog/510440
- Tarantool Cartridge's trusted leader choice habr.com/ru/company/mailru/blog/513912
- Tarantool Telegram channel with product news t.me/tarantool_news
- Discuss Tarantool in the community chat t.me/tarantoolru