ShuffleExchangeLike Physical Operators¶

ShuffleExchangeLike is an extension of the Exchange abstraction for physical operators.

Contract¶

getShuffleRDD¶

getShuffleRDD(
  partitionSpecs: Array[ShufflePartitionSpec]): RDD[_]

RDD (Spark Core)

Used when:

• CustomShuffleReaderExec physical operator is requested for the shuffleRDD

mapOutputStatisticsFuture¶

mapOutputStatisticsFuture: Future[MapOutputStatistics]

MapOutputStatistics (Apache Spark)

Used when:

• ShuffleQueryStageExec physical operator is requested to doMaterialize and cancel

Number of Mappers¶

numMappers: Int

Used when:

• OptimizeShuffleWithLocalRead physical optimization is requested for the shuffle partition specification

Number of Partitions¶

numPartitions: Int

Used when:

• OptimizeShuffleWithLocalRead physical optimization is requested for the shuffle partition specification

Runtime Statistics¶

runtimeStatistics: Statistics

Statistics with data size and row count

See:

• ShuffleExchangeExec

Used when:

• ShuffleQueryStageExec physical operator is requested for the runtime statistics

ShuffleOrigin¶

shuffleOrigin: ShuffleOrigin

ShuffleOrigin

Used when:

• AdaptiveSparkPlanExec physical operator is requested for the finalStageOptimizerRules
• CoalesceShufflePartitions physical optimization is requested to supportCoalesce
• OptimizeShuffleWithLocalRead physical optimization is requested to supportLocalReader
• ShuffleStage utility is used to destructure a SparkPlan to a ShuffleStageInfo

Implementations¶

• ShuffleExchangeExec

Submitting Shuffle Job¶

submitShuffleJob: Future[MapOutputStatistics]

submitShuffleJob executes a query to materialize the mapOutputStatisticsFuture.

submitShuffleJob is used when:

• ShuffleQueryStageExec adaptive leaf physical operator is requested for the shuffleFuture