Faust Background Tasks Part I: An Introduction

1. Part I: Introduction
2. Part II: Agents and Teams

How did I come to this life?

Not so long ago, I had to work on the backend of a highly loaded project, in which I needed to organize the regular execution of a large number of background tasks with complex calculations and requests for third-party services. The project is asynchronous and before I came to it, it had a simple mechanism for launching tasks in a crown: a loop with checking the current time and launching groups of coroutines via gather - this approach turned out to be acceptable until there were dozens and hundreds of such coroutines, however, when their number exceeded two thousand, I had to think about organizing a normal task queue with a broker, several workers, and so on.

Celery, . , , , .

, , , , . , , , (. group). issue , , . , , … , . , 2-3 http- , 4 tcp , 2 — … . - aiohttp .

, ! celery, , Ask Solem, Faust, robinhood. Faust Kafka Streams Kafka , rocksdb, — , .

, celery faust : , , . , , faust — .

?

, , Faust. - , , alphavantage.co. , (sink, , ), (cron) , cli- faust ( click), , datadog ( ) , - . mongodb motor .

P.S. , , , - , - :

, , , :

1. overview ( .. , , cash flow — ) —
2. ( ) —
3. —
4. —

, : horton

, , — docker-compose kafka ( zookeeper — ), kafdrop ( ), mongodb. [docker-compose.yml](https://github.com/Egnod/horton/blob/562fa5ec14df952cd74760acf76e141707d2ef58/docker-compose.yml) :

version: '3'

services:
  db:
    container_name: horton-mongodb-local
    image: mongo:4.2-bionic
    command: mongod --port 20017
    restart: always
    ports:
      - 20017:20017
    environment:
      - MONGO_INITDB_DATABASE=horton
      - MONGO_INITDB_ROOT_USERNAME=admin
      - MONGO_INITDB_ROOT_PASSWORD=admin_password

  kafka-service:
    container_name: horton-kafka-local
    image: obsidiandynamics/kafka
    restart: always
    ports:
      - "2181:2181"
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: "INTERNAL://:29092,EXTERNAL://:9092"
      KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka-service:29092,EXTERNAL://localhost:9092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT"
      KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
      KAFKA_ZOOKEEPER_SESSION_TIMEOUT: "6000"
      KAFKA_RESTART_ATTEMPTS: "10"
      KAFKA_RESTART_DELAY: "5"
      ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL: "0"

  kafdrop:
    container_name: horton-kafdrop-local
    image: 'obsidiandynamics/kafdrop:latest'
    restart: always
    ports:
      - '9000:9000'
    environment:
      KAFKA_BROKERCONNECT: kafka-service:29092
    depends_on:
      - kafka-service

. kafka listener': (internal) , (external) , . 2181 — zookeeper'. , , .

:

horton
├── docker-compose.yml
└── horton
    ├── agents.py *
    ├── alphavantage.py *
    ├── app.py *
    ├── config.py
    ├── database
    │   ├── connect.py
    │   ├── cruds
    │   │   ├── base.py
    │   │   ├── __init__.py
    │   │   └── security.py *
    │   └── __init__.py
    ├── __init__.py
    ├── records.py *
    └── tasks.py *

, , .

. , mongodb. , , .

— pyproject.toml

, virtualenv (, venv ):

pip3 install poetry (   )
poetry install

config.yml — . alphavantage. config.py — . , , — sitri.

— . , .

?

, , — , .

, :

1. alphavantage aiohttp .
2. , .