How to be confident in your workflows, conveyor by conveyor
On the eve of the start of the "Data Engineer" online course , we prepared a translation of the material.
DevOps, « - ». .
, Big Data Institute « : Big Data », , Monte Carlo, , , .
(Data Engineering) « data science» - , , , , . -, , : , - .
- - , Looker .
- - , , .
Slack : . , - .
, : . !
, , : - , , .
- (Data Observability).
#1. —
- ! - . , , , , . , .
, . , - Blinkist , 20 . , .
#2. DevOps
, , , , , . , : ​​DevOps, , 20 .
, DevOps, , , . , - , , .
, : , , , .
(Freshness) , .
(Distribution) , .
(Volume ) .
(Schema) , .
(Lineage) , , .
#3. - ... 3 ,
. , , , .
-, , - , . , , - ( ).
-, . , , , , , - , , , , - .
, : . , -, , , , .
#4. - ,
, - . , , , .
(lineage), . Lineage , , , , .
, , . , : ? ? ? ?
, . , , - , , , .
#5. , ,
. - - , , , , - . .
-. , API . , .
, , DevOps - . , , . .