If your position is data engineer, then you may have encountered an offensive information bias. The topic of data science is actively covered, there are many useful materials on it. And you work in a related field, where many important questions also arise, but much less is said about these questions.
, , 9-12 SmartData. - : data scientists, - , .
- , , . , , , .
, 2017- « SmartData», 2018- SmartData. : « , , ». : !
, , . , , . , data science, :
, , , . , : « , , , - …»
, SmartData 2020? , , . , :
Streaming
- Flink
- Spark
- Kafka
, , noSQL, SMP/MPP- DWH:
- Hive, Impala, Presto, Vertica, ClickHouse, Cassandra
- Teradata, Redshift, GreenPlum, exadata
- MSSQL, PostgreSQL
- MongoDB, DynamoDB
- S3, ADLS, GCS, HDFS
DWH
- Ad-hoc reporting
- Hadoop
Data governance
- Data security
- Data quality
- Metadata catalog management
- Master data management
ETL
- Spark
- Hadoop MapReduce
- Sqoop
- Performance analysis and optimization
MLOps
- Airflow, NiFi, Luigi, Azkaban, Oozie etc
- MLflow
-
- - , data engineer
- CI/CD
SmartData
Call to action
? :
SmartData!
