Multichannel bulk mailing on Redis

Introductory

Hello, Habr! My name is Boris and in this work I will share with you my experience in designing and implementing a mass mailing service as part of a comprehensive system for alerting students to teachers (hereinafter also referred to as Ada), which I also implement.

Hell

It is then needed to negate the number of interruptions in the educational process for the following reasons:

1. Teachers don't want to share personal contact details;
2. The students really are too - they just don't have much choice;
3. Due to the specifics of my alma mater, many teachers are forced or prefer to use mobile devices without access to the Internet;
4. If you send messages through the leaders of the groups, then the effect of a "damaged phone" comes into play, as well as the factor "oh, I forgot :(".

It is running about so :

1. The teacher through one of the communication channels available to him: SMS, Telegram, SPA-application - sends Ada the text of the message and the list of recipients
2. Ada broadcasts the received message to all interested * students through various communication channels.

* Access to the service is provided on a voluntary application basis.

It is assumed that

1. The total number of users will not exceed ten thousand;
2. The ratio of student - teacher / member of the Office of Internal Affairs (dean's office, health center, military registration desk, etc.) will be kept at the level of 10: 1;
3. : « », « ))0» ..

1. ;
2. , , ;
3. ;
4. : - - , , .

This work consists of five parts: introductory, preparatory, conceptual, subject and final.



You can safely skip the preparatory part if you are familiar with the Redis interpretation of the Pub / Sub pattern, as well as the mechanisms of events, LUA scripting and handling of obsolete keys, in addition, it is highly desirable to have at least some idea of ​​the microservice architecture of the software.



In the subject part, the code in Python is reviewed, but I believe that there is enough information for you to write something like that on anything.

Preparatory

Conceptual

Naive approach

The most obvious solution you can think of: multiple delivery methods ( send_vk, send_telegrametc.) and one handler that will call them with the required arguments.

Extensibility problem

If we want to add a new delivery method, we will be forced to modify the existing code, and this is the limitations of the software platform.

Stability problem

One of the methods has broken = the whole service has broken.

Applied problem

The APIs of different communication channels differ significantly from each other in terms of interaction. For example, VKontakte supports mass mailings, but no more than hundreds of users per call. Telegram does not exist, but it allows more calls per second.



The VK API works only via HTTP; Telegram has an HTTP gateway, but it is less stable than MTProto and is less well documented.



There are a lot of such differences: maximum message length random_id, interpretation and error handling, etc. etc.

How to deal with this?

It was decided to separate the process of enqueuing messages and sending processes (hereinafter referred to as couriers) at the organizational level, and so that the former would not even suspect the existence of the latter, and vice versa, and Redis would act as a connecting link between them.

Unclear? Order a meal!

In the meantime, you are waiting - let me introduce you to my interpretation of this noble action, starting with the design and ending with the door closed behind the courier.

1. You click on the big yellow "Order" button;
2. Yandex.Food finds a courier, informs the restaurant about the selected items and returns the order number to you in order to dilute the uncertainty of expectations;
3. Upon completion of cooking, the restaurant updates the order status and gives the food to the courier;
4. The courier, in turn, gives the food to you, and then marks the order as completed.

Bon Appetit!

Back to design

It is possible that the model given in the paragraph earlier does not fully correspond to reality, but it was she who formed the basis of the developed solution.



The data associated with the order number will be called history , it allows you to answer the following questions at any time :

1. Who sent;
2. What he sent;
3. Where from;
4. To whom;
5. Who got it and how.

The history is created along with the order as two separate Redis keys, linked via a suffix:

suffix={ }:{UNIX-  }
=history:{suffix}
=delivery:{suffix}

The order determines when the couriers will see the history once, so that, upon completion of the dispatch, the answer to the question “Who received it and how” will be changed accordingly.



The "vision" of the couriers works through a subscription to the event DELkeys in the form delivery:*.



When the moment of delivery comes, Redis deletes the order key, after which the couriers begin to process it.



Since there are several couriers, there is a high probability of competition at the stage of history change.







You can avoid it by defining the corresponding operation atomically - in Redis, this is done through LUA scripting.



Implementation details will be discussed in detail in the next chapter. Now it is important to get a clear idea of ​​the solution as a whole, which can be helped by the figure below.

Tracking status

The client can track the delivery status through the history key that is generated by a separate API method of the service being developed before the message is queued (just like the order number is generated by Yandex.Food at the very beginning).



After the key is generated, a tracker with a timeout is hung on it (optionally and also by a separate method), which will monitor the number of history changes by couriers ( SETevents). Only now the message is being queued.







If the courier does not find recipient contacts in his domain - the communication channel, then he triggers an artificial event SETthrough the command PUBLISH, thereby showing that he is “okay” and there is no need to wait any longer.

Why mess with events in Redis when you have RabbitMQ and Celery

There are at least five objective reasons for this:

1. Redis , RabbitMQ/Celery — ;
2. Redis , , , IPC;
3. Redis’a SQL- ;
4. . , API-, ;
5. Celery asyncio, asyncio .

The notification system (encompassing) is implemented in the form of a set of microservices. For convenience's sake, interfaces, methods for initializing data layers, error text, as well as some blocks of repetitive logic have been moved to the library core, which, in turn, relies on: gino(asyncio wrapper SQLAlchemy), aioredisand aiohttp.



You can see different entities in the code, for example User, Contactor Allegiance. The connections between them are presented in the diagram below, a short description is under the spoiler.

Generating a history key

delivery / handlers / history_key / get - GitHub

Queue

delivery / handlers / queue / put - GitHub



Note:

1. Comment 171: 174;
2. That all manipulations with Redis [164: 179] are wrapped in a transaction.

Sight of the couriers [94: 117]

core / delivery - GitHub

Updating history by couriers

core / redis_lua - GitHub The



instructions [48:60] do not convert empty lists to dictionaries ( [] -> {}), since most programming languages, including CPython, interpret them differently than LUA.



ISS: Allow differentiation of arrays and objects for proper empty-object serialization - GitHub

Tracker

delivery / handlers / track / post - GitHub - implementation.

connect / telegram / handlers / select - GitHub [101: 134] - example usage in user interface.

Couriers

Any delivery from task_stream(@Sight Couriers) is handled in a separate asyncio coroutine.



The general strategy for dealing with the timing constraints of APIs is as follows: we do not count RPS (requests per second), but we correctly / react / respond to responses by type http.TooManyRequests.



If the interface implements, in addition to global (for the application), custom time limits, then they are processed in the order of the queue, i.e. first we send to everyone we can and only then we start to wait, if not very long.

Telegram

courier / telegram - GitHub

As noted earlier, Telegram's MTProto interface outperforms its HTTP counterpart in terms of stability and documentation size. To interact with it, we will use a ready-made solution, namely LonamiWebs / Telethon .

In contact with

courier / vk - GitHub

VKontakte API supports mass mailings by passing a list of identifiers to the messages.send method (no more than a hundred), and also allows you to "glue" up to twenty-five messages.sendin one execute , which gives us 2500 messages per call.

The final

In the present work, a method for organizing a multichannel mass warning system is proposed. The resulting solution satisfies the request (@ Key requirements for the mailing service) of most interested parties, and also assumes the possibility of expansion.



The main disadvantage is the fire & forget Pub / Sub effect, i.e. if the deletion of the order key is necessary at the time of one of the couriers' illness, then no one will receive anything in the corresponding domain, which, however, will be reflected in the history.