ResolveCoalesceHints Logical Resolution Rule¶

ResolveCoalesceHints is a logical resolution rule to resolve UnresolvedHint logical operators with the following hint names:

Hint Name	Arguments	Logical Operator
`COALESCE`	Number of partitions	Repartition (with `shuffle` off / `false`)
`REBALANCE`		RebalancePartitions
`REPARTITION`	Number of partitions alone or like `REPARTITION_BY_RANGE`	Repartition (with `shuffle` on / `true`)
`REPARTITION_BY_RANGE`	Column names with an optional number of partitions (default: spark.sql.shuffle.partitions configuration property)	RepartitionByExpression

ResolveCoalesceHints is a Catalyst rule for transforming logical plans (Rule[LogicalPlan]).

ResolveCoalesceHints is part of Hints batch of rules of Logical Analyzer.

Creating Instance¶

ResolveCoalesceHints takes the following to be created:

SQLConf

ResolveCoalesceHints is created when Logical Analyzer is requested for the batches of rules.

Executing Rule¶

Rule

apply(
  plan: LogicalPlan): LogicalPlan

apply is part of the Rule abstraction.

apply resolves UnresolvedHint logical operators with the following hint names (case-insensitive).

Hint Name	Trigger
`COALESCE`	createRepartition (with `shuffle` off)
`REBALANCE`	createRebalance
`REPARTITION`	createRepartition (with `shuffle` on)
`REPARTITION_BY_RANGE`	createRepartitionByRange

createRebalance¶

createRebalance(
  hint: UnresolvedHint): LogicalPlan

createRebalance handles a REBALANCE hint and creates a Repartition logical operator.

createRepartition¶

createRepartition(
  shuffle: Boolean,
  hint: UnresolvedHint): LogicalPlan

createRepartition handles COALESCE and REPARTITION hints (and creates Repartition or RepartitionByExpression logical operators).

createRepartitionByRange¶

createRepartitionByRange(
  hint: UnresolvedHint): RepartitionByExpression

createRepartitionByRange creates a RepartitionByExpression logical operator.

Examples¶

Using COALESCE Hint¶

// Use Catalyst DSL to create a logical plan
import org.apache.spark.sql.catalyst.dsl.plans._
val plan = table("t1").hint(name = "COALESCE", 3)
scala> println(plan.numberedTreeString)
00 'UnresolvedHint COALESCE, [3]
01 +- 'UnresolvedRelation `t1`

import org.apache.spark.sql.catalyst.analysis.ResolveHints.ResolveCoalesceHints
val analyzedPlan = ResolveCoalesceHints(plan)
scala> println(analyzedPlan.numberedTreeString)
00 'Repartition 3, false
01 +- 'UnresolvedRelation `t1`

Using REPARTITION Hint¶

// Use Catalyst DSL to create a logical plan
import org.apache.spark.sql.catalyst.dsl.plans._
val plan = table("t1").hint(name = "REPARTITION", 3)
scala> println(plan.numberedTreeString)
00 'UnresolvedHint REPARTITION, [3]
01 +- 'UnresolvedRelation `t1`

import org.apache.spark.sql.catalyst.analysis.ResolveHints.ResolveCoalesceHints
val analyzedPlan = ResolveCoalesceHints(plan)
scala> println(analyzedPlan.numberedTreeString)
00 'Repartition 3, true
01 +- 'UnresolvedRelation `t1`

Using COALESCE Hint in SQL¶

val q = sql("SELECT /*+ COALESCE(10) */ * FROM VALUES 1 t(id)")
val plan = q.queryExecution.logical
scala> println(plan.numberedTreeString)
00 'UnresolvedHint COALESCE, [10]
01 +- 'Project [*]
02    +- 'SubqueryAlias `t`
03       +- 'UnresolvedInlineTable [id], [List(1)]

import org.apache.spark.sql.catalyst.analysis.ResolveHints.ResolveCoalesceHints
val analyzedPlan = ResolveCoalesceHints(plan)
scala> println(analyzedPlan.numberedTreeString)
00 'Repartition 10, false
01 +- 'Project [*]
02    +- 'SubqueryAlias `t`
03       +- 'UnresolvedInlineTable [id], [List(1)]