Hello, Habr! In anticipation of the start of the course "Ecosystem Hadoop, Spark, Hive" we have prepared for you a translation of a useful article. We also offer to watch a free recording of a demo lesson on the topic: "Spark 3.0: What's New?" ...
(executor) - , (.. ) (node) . , EC2 . r5.4xlarge, , AWS EC2, 16 .
(job), (Cluster Manager). 16 . , Spark , 15 .
, , , , (core) Spark . (X * Y = 15), , , 15 Spark :
.
, , - 15 . , , , , , (64 +), . .
, — 15 , . , , , . , . .
10% 384 MB ( , ). big data , Qubole, , . , «Environments» Spark spark.executor.memoryOverhead.
Spark , . , , , . , , .
, . Spark , 5 — . , , , . , . , .
--executor-cores 5— , . , , . , — . , EC2 , , , . r5.4xlarge AWS , 128 .
, 112 .
, 112 , . , 3. .
( Qubole), . (112/3) = 37–2,3 = 34,7 = 34.
Spark , . (112/3) = 37 / 1,1 = 33,6 = 33.
Qubole.
--executor-memory 34G, Spark . . , Spark Spark. , , , . , , 2, . , , , — , .
, , EC2, . , , .
. , , 5.
, , , . , - , 15 , , , . 3, .
. . , 3, 16- , , 14 . !
, , , 3 , .
--num-executors (3x - 1)4 , .
data- . AWS , . , .
--driver-memory 34G, , , . 2 , (3x - 2), .
. , , 500 Spark, , . . , , .
--driver-cores 5?
, , 16 128 , .
--driver-memory 34G --executor-memory 34G --num-executors (3x - 1) --executor-cores 5:
— , .
, . . , , , .
, , Expedia Group ™, . , , , .
, , . ? ? , , ? - ? 4: « Apache Spark ».