RepartitionOperation Unary Logical Operators¶

RepartitionOperation is an extension of the UnaryNode abstraction for repartition operations.

Contract¶

shuffle¶

shuffle: Boolean

Number of Partitions¶

numPartitions: Int

Partitioning¶

partitioning: Partitioning

Partitioning

Implementations¶

Logical Optimizations¶

CollapseRepartition logical optimization collapses adjacent repartition operations
Repartition operations allow FoldablePropagation and PushDownPredicate logical optimizations to "push through"
PropagateEmptyRelation logical optimization may result in an empty LocalRelation for repartition operations

Output Attributes¶

output: Seq[Attribute]

output simply requests the child logical operator for the output attributes.

output is part of the QueryPlan abstraction.

Demo: Coalesce Operator¶

val numsCoalesced = nums.coalesce(numPartitions = 4)
assert(numsCoalesced.rdd.getNumPartitions == 4, "Number of partitions should be 4")

scala> numsCoalesced.explain(extended = true)
== Parsed Logical Plan ==
Repartition 4, false
+- Range (0, 5, step=1, splits=Some(16))

== Analyzed Logical Plan ==
id: bigint
Repartition 4, false
+- Range (0, 5, step=1, splits=Some(16))

== Optimized Logical Plan ==
Repartition 4, false
+- Range (0, 5, step=1, splits=Some(16))

== Physical Plan ==
Coalesce 4
+- *(1) Range (0, 5, step=1, splits=16)