Hello, Habr. For future students of the course "Ecosystem Hadoop, Spark, Hive" prepared a translation of the material.
We also invite everyone to the "Testing Spark Applications" webinar . In this open lesson, we will consider the problems in testing Spark applications: stat data, partial verification and start / stop of heavy systems. Let's study the libraries for the solution and write tests.
This article focuses exclusively on the Join operation in Apache Spark and provides an overview of the foundation on which Spark Join technology is built.
Joins are often used in typical data mining streams to correlate two sets of data. Apache Spark, being a unified analytics engine, has also provided a solid foundation for executing a wide variety of Join scenarios.
Join , , , , . ( ) Join , Joined . .
Join:
, Join Apache Spark. :
1) : Join. , Join, Join.
2) Join: , , (Join Condition). () , . , : Join Joins.
, , . . , (A.x == B.x) ((A.x == B.x) (A.y == B.y)) - x, y A B, Join.
. , . , (A.x < B.x) ((A.x == B.x) (A.y == B.y)) - x, y A B, Join.
3) Join type: Join Join Join . Join:
(Inner Join): Inner Join Joined ( Join) .
(Outer Join): Outer Join , . , () .
(Semi Join): Semi Join , , , . , , , (Semi Join) (Anti Join).
: Cross Join , .
Join, Apache Spark Join.
Join
, Join, .
Apache Spark Join. :
(Shuffle Hash Join)
(Broadcast Hash Join)
(Sort Merge Join)
(Cartesian Join)
(Broadcast Nested Loop Join)
Broadcast Hash Join: «Broadcast Hash Join» ( Join) . - , , -.
“Broadcast Hash Join" . , . Spark , .
Shuffle Hash Join: 'Shuffle Hash Join' () ( , ”Guide to Spark Partitioning ( Spark)”. , , (shuffle) Join.
, , Shuffle Hash Join, , Hash Join. , - , -.
"Shuffle Hash Join" "Broadcast Hash Join". , - . , , Join 'Shuffle Hash Join'. , 'Broadcast Hash Join', Spark .
Sort Merge Join: 'Sort Merge Join' 'Shuffle Hash Join'. () . , , (shuffle) Join.
, , Sort Merge Join , Sort Merge Join.
'Sort Merge Join' 'Shuffle Hash Join' 'Broadcast Hash Join', , 'Sort Merge Join' , 'Shuffle Hash' 'Broadcast Hash'. , 'Shuffle Hash Join', , (shuffle) , , 'Sort Merge Join'.
Cartesian Join: Cartesian Join . . , . , .
Cartesian Join . Join, Cartesian - .
Broadcast Nested Loop Join: 'Broadcast Nested Loop Join' . Nested Loop Join .
«Broadcast Nested Loop Join» , . , , .
Spark Join?
Join Join, , Spark :
Spark Join, :
Join
Join
(Equi or Non-Equi Join)
Spark API Join Join Join. Join, 'broadcast', 'merge', 'shuffle_hash' 'shuffle_replicate_nl', , Join.
, Spark Join :
'Broadcast Hash Join'
Equi Join
'Full Outer' Join
, :
'Broadcast', Join - 'Right Outer', 'Right Semi' 'Inner'.
, '
spark.sql.autoBroadcastJoinThreshold
( 10 )' Join - 'Right Outer', 'Right Semi', or 'Inner'.
'Broadcast' , Join - 'Left Outer', 'Left Semi' 'Inner'.
, '
spark.sql.autoBroadcastJoinThreshold
( 10 )' Join - 'Left Outer', 'Left Semi', or 'Inner'.
'Broadcast' , Join - 'Left Outer', 'Left Semi', 'Right Outer', 'Right Semi' 'Inner'.
, '
spark.sql.autoBroadcastJoinThreshold
( 10 )' Join - 'Left Outer', 'Left Semi', 'Right Outer', 'Right Semi' 'Inner'.
'Shuffle Hash Join'
Equi Join
'Full Outer' Join
'
spark.sql.join.prefersortmergeJoin
( true)' false
, :
'shuffle_hash' , Join - 'Right Outer', 'Right Semi', 'Inner'.
, , Join - 'Right Outer', 'Right Semi' 'Inner'.
'shuffle_hash' , Join - 'Left Outer', 'Left Semi', 'Inner'.
, , Join - 'Left Outer', 'Left Semi', 'Inner'.
'shuffle_hash' , Join - 'Left Outer', 'Left Semi', 'Right Outer', 'Right Semi', 'Inner'.
, , Join - 'Left Outer', 'Left Semi', 'Right Outer', 'Right Semi', 'Inner'.
'Sort Merge Join'
Equi Join
Join Keys, Equi Join,
'spark.sql.join.prefersortmergeJoin ( true)' true.
, :
'merge' , Join .
, Join .
'Cartesian Join'
'Inner'
, :
'shuffle_replicate_nl' , Join Equi Non-Equi.
, Join Equi Non-Equi.
'Broadcast Nested Loop Join'
'Broadcast Nested Loop Join' - Join ; , 'Broadcast Nested Loop Join' Join Join.
, Join , 'Broadcast Hash Join', 'Sort Merge Join', 'Shuffle Hash Join', 'Cartesian Join'.
Cartesian Broadcast Nested Loop Join, Broadcast Nested Loop Inner, Non-Equi Joins, Cartesian Join, , .
, : Join. , .
, Join Apache Spark. - , , .