Hello!
We have huge nightly reports on product metrics that go to their owners in the morning. A couple of years ago, they were collected with a regular Cron, but this turned out to be a very unstable story. Cron requires taking into account certain nuances and discipline: if the report is not collected, then you need to restart the script. And it can fail for many reasons: one of the MySQL databases did not respond, or the polled service did not respond, or the server on which Cron is running rebooted or failed at night. Most of these reports were created by analysts who usually did not take all this into account. Because they are analysts and this is not their task. So we started looking for options that would make life easier for everyone.
And they found it.
Airflow Kubernetes. . , , Data Team, – , - .
– , , . – , , . , , , , . , , , , , – , .
. – . : , . , production-, , . , . , . .
, - : . python, Cron' . , . , , – . : , , , .
DataTeam . , .
: "", , .
- – . , , , – . , , , . , , . Cron . , .
Cron' . , , - . Cron' . , , . .
– , . , , . : , , - . - ( , ). , .
, ( ), . Kubernetes'. , CronJob. , Cron'. : , CronJob , . , , . , Cron , . CronJob "" ( : A cron job creates a job object about once per execution time of its schedule. ). "" , . . CronJob . .
, Kubernetes' . , CronJob , , . – , . , , , . , , , , - , , .
, . Airflow.
Airflow
(, ). Arflow , . – ( directed acyclic graph).
– , . , , . Airflow . , .
– . Cron CronJob . Cron' , , CronJob – Kubernetes'. , Airflow , Kubernetes', - . , . Airflow , , . .
, , Kubernetes'. Airflow , , : KubernetesPodOperator KubernetesExecutor. KubernetesPodOperator: docker- Kubernetes'.
kubernetes-client API Kubernetes', Airflow. configmap' , . KubernetesExecutor' : Airflow , , .
Airflow + KubernetesPodOperator + Kubernetes , , , Airflow Kubernetes, , - .
KubernetesPodOperator'
Kubernetes' Docker' Airflow. - SimpleHttpOperator' PythonOperator', http- , . , , - . – , -. , : , , , . Airflow. – - , , , , .
, .
: . . BI- Metabase, . ().
Airflow + KubernetesPodOperator + Kubernetes .
, " !"?
:
;
, ;
, ;
, .
, Airflow , / ;, , ;
5.
, "how to" Airflow , .