Big Data Engineering has emerged as a separate profession quite recently. And even large companies very often confuse what this specialist does, what his competencies are and why he is in the organization at all.
, Big Data Engineer, Data Analyst Data Scientist. , . , , .
Big data engineer
, , . data science.
Data engineer — , , . , .
: e-commerce, , , — , .
, «» . IoT- . , . , . , , .
, :
. BDE . . ETL (extract, transform, load) — , . . Kafka, Apache Spark, Storm, Flink, Google Cloud Azure.
. — -. — , .
. , , . , .
. - , , .
: , .
Data Engineer
;
SQL NoSQL . : MySQL, PostgreSQL, MongoDB, Oracle, HP Vertica, Amazon Redshift;
ETL- (BM WebSphere DataStage; Informatica PowerCenter; Oracle Data Integrator; SAP Data Services; SAS Data Integration Server);
Amazon Web Services, Google Cloud Platform, Microsoft Azure;
Apache SQL- ;
(Python, Scala, Java).
-, , , .
Data Engineer , -. , , . - , — , .
Business Intelligence, .
, Data Engineer Data Scientist , . — , . — .
:
. . BDE , , . hh.ru, , 768 .
Big Data Engineer , Data Scientist. , . — ( ). , .
— , . . .
. — . . -, , .
, .
Spark — , , .
Flink — Spark / , .
Kafka — , . ElasticSearch — , Apache Lucene.
PostgreSQL — .
Redshift — / AWS.
BD Engineering . , 1–2 . Python Scala — .
. , . — Data Scientist Data Analyst, - , . . .
Data Engineer
- . Python Scala, Big Data. , , 50/50. Java — .
SQL . Data Engineer , : Data Analyst, Business Analyst, Data Scientist. - 1–2 .
, .
Big Data Engineers . Data Science Data Analytics, , Devops-. Python Scala, .
. Dice Tech Job Report 2020, Data Engineering — 2019 50 %. : 3–5 %.
2020 , . 24,8 %. .
, Data Engineering Data Engineering Data Science . , , .
