Repartition Logical Operator¶
Repartition is a concrete RepartitionOperation.
Repartition takes the following to be created:
- Number of partitions
- shuffle flag
- Child LogicalPlan
Repartition is created for the following:
- Dataset.coalesce and Dataset.repartition operators (with shuffle disabled and enabled, respectively)
REPARTITIONhints (via ResolveCoalesceHints logical analysis rule, with shuffle disabled and enabled, respectively)
Repartition is planned to the following physical operators based on shuffle flag:
- ShuffleExchangeExec with
- CoalesceExec with
Catalyst DSL defines the following operators to create a
Repartition logical operator:
- coalesce (with shuffle disabled)
- repartition (with shuffle enabled)
partitioning uses the numPartitions to determine the Partitioning:
- SinglePartition for
- RoundRobinPartitioning otherwise
partitioning requires that the shuffle flag is enabled or throws an exception:
Partitioning can only be used in shuffle.
partitioning is part of the RepartitionOperation abstraction.