Queues are a great tool that scales almost perfectly. Iron does not cope? We just added nodes to the cluster. When a queue is present in a project, it is tempting to implement more and more functionality with its help.
We will talk about the pitfalls of this path in this article.
Sooner or later, when using queues, the user is faced with the question of using them in conjunction with some service, database, etc.
The order is completed, you need to send an SMS notification to the user.
A new order has arrived, you need to send a push notification to the executors.
The work is done, you need to write off the money from the client's account.
In all of the above examples, changes in some business entity are recorded in the database (or a service with a database), and there is a great temptation to send notifications using queues.
What do we have in this situation? Initial, simplest code structure:
The service (our program) records changes in data in the database.
The service then puts the job in the queue.
In fact, in this case, you need to implement an event trigger to change the data record.
And in the general case, it turns out that here we have two records in two different databases: services and queues.
Now let's jump into the real world and consider what situations may arise:
Everything is fine. DB is available, queue DB is available;
, ;
, ;
, .
: , , .
, , ... . .
, .
, ( , , ), , :
.
.
.
.
, , .
:
, . 3 4 ( ).
. .
.
, . : , . , , .
, (, , http- /).
, .
/, ( queue
) .
, , ( ):
/* */
UPDATE
"orders"
SET
"status" = 'complete'
WHERE
"order_id" = $1
RETURNING
*
/* */
WITH "o" AS (
UPDATE
"orders"
SET
"status" = 'complete'
WHERE
"order_id" = $1
RETURNING
*
),
"q" AS (
INSERT INTO
"queue"
(
"key",
"data"
)
SELECT
"o.order_id",
"o.status"
FROM
"o"
)
SELECT
*
FROM
"o"
: queue
, , orders
.
, queue
, , . , .
:
queue
.
.
.
/
, , . , - ( , ..), :
.
, , , , O_APPEND
- .
( ) ,
.
( ) .
, , , , .
As you can see, there are few options for solving the problem. If we want to keep the system simple (KISS-principle), then the introduction of an additional daemon and cache / log messages in the database or local file / database will give a slight increase in complexity. At the same time, it is very important to keep the handler idempotent, since in case of failures at the time of transferring tasks from the local cache to the general queue, duplicates may appear.
A generalized solution is to use a two-phase commit.