Union Logical Operator¶

Union is a logical operator that represents the following high-level operators in a logical plan:

• UNION SQL statement
• Dataset.union, Dataset.unionAll and Dataset.unionByName operators

Union is resolved using ResolveUnion logical analysis rule.

Union is resolved into UnionExec physical operator by BasicOperators execution planning strategy.

Creating Instance¶

Union takes the following to be created:

• Child LogicalPlans
• byName flag (default: false)
• allowMissingCol flag (default: false)

Union is created (possibly using apply utility) when:

• AstBuilder is requested to visitFromStatement and visitMultiInsertQuery
• Dataset is requested to flattenUnion (for Dataset.union and Dataset.unionByName operators)
• Dataset.unionByName operator is used

Creating Union¶

apply(
  left: LogicalPlan,
  right: LogicalPlan): Union

apply creates a Union logical operator (with the left and right plans as the children operators).

apply is used when:

• ResolveUnion logical resolution rule is executed
• RewriteUpdateTable is requested to buildReplaceDataWithUnionPlan
• RewriteExceptAll logical optimization is executed
• RewriteIntersectAll logical optimization is executed
• AstBuilder is requested to parse UNION SQL statement
• Dataset.union operator is used

Maximum Number of Records¶

maxRows: Option[Long]

maxRows is the total of the maxRows of all the children.

Node Patterns¶

nodePatterns: Seq[TreePattern]

nodePatterns is UNION.

Metadata Output Attributes¶

metadataOutput: Seq[Attribute]

metadataOutput is empty.

Catalyst DSL¶

Catalyst DSL comes with union operator to create an Union operator.

union(
  otherPlan: LogicalPlan): LogicalPlan

Logical Optimizations¶

• EliminateUnions
• CombineUnions