Join Logical Operator¶

Join is a binary logical operator that represents the following high-level operators in a logical plan:

• JOIN SQL statement
• Dataset.crossJoin, Dataset.join and Dataset.joinWith operators

Creating Instance¶

Join takes the following to be created:

• Left logical operator
• Right logical operator
• JoinType
• Optional Join Expression
• JoinHint

Join is created when:

• AstBuilder is requested to withJoinRelations (and visitFromClause)
• Dataset.crossJoin, Dataset.join and Dataset.joinWith operators are used

Maximum Number of Records¶

maxRows: Option[Long]

maxRows...FIXME

Catalyst DSL¶

DslLogicalPlan defines join operator to create a Join.

import org.apache.spark.sql.catalyst.dsl.plans._
val t1 = table("t1")
val t2 = table("t2")
val j = t1.join(t2)

import org.apache.spark.sql.catalyst.plans.logical.Join
assert(j.isInstanceOf[Join])