In virtually every sector that deals with complex data, Spark has quickly become a “de facto” distributed computing environment for teams at all stages of the data and analytics lifecycle. One of the most anticipated features of Spark 3.0 is the new Adaptive Query Execution (AQE) framework, which addresses issues that arise with many Spark SQL workloads. They were documented in early 2018 by a team of Intel and Baidu experts. For a deeper look at the framework, you can take our updated Apache Spark Performance Tuning course .
Our experience with Workload XM certainly confirms the reality and severity of these issues.
AQE Spark 2.4, Spark 3.0 3.1 . , , AQE.
Catalyst
, , group-by-count DataFrames.
Spark , " " - 200.
:
1. 200 , , ;
2. , 200 ;
3. : , .
? :
spark.conf.set(“spark.sql.shuffle.partitions”,”2″)
:
.
.
.
, Spark . , , , - : .
Adaptive Query Execution
AQE , . , « », .
Catalyst , , .
AQE Spark Catalyst, Spark " ".
? , :
, Spark , .
Spark , Spark , , .
AQE , Spark 2.4.
, spark.sql.adaptive.enabled true, - false. AQE , 200 .
TPC-DS AQE:
Sort Merge Joins Broadcast Joins
AQE sort-merge broadcast -, broadcast -.
TPC-DS AQE:
shuffle
, , - .
:
· spark.sql.adaptive.enabled
· spark.sql.adaptive.coalescePartitions.enabled
true, Spark , spark.sql.adaptive.advisoryPartitionSizeInBytes. , .
Skew () - . :
, , .
skew join, , A0 , , park.sql.adaptive.advisoryPartitionSizeInBytes, B0 B.
, AQE .
:
1. spark.sql.adaptive.skewJoin.skewedPartitionFactor : , , , , , ,
2. spark.sql.adaptive.skewedPartitionThresholdInBytes, : , .
(dynamic partition pruning, DPP) - : , . DPP, AQE . DPP Spark 2.4 CDP.
, .
1. .
2. , , .
DPP (, SortMergeJoin), spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly.
Spark , DPP .
DPP , , 5 .
, 72 99 TPC-DS DPP.
Spark : .
- : Spark RDD, DataFrames.
SQL, - , : , .
AQE, DPP, Kubernetes , Spark 3.1