Skip to content

ShuffleExchangeLike Physical Operators

ShuffleExchangeLike is an extension of the Exchange abstraction for physical operators.

Contract

getShuffleRDD

getShuffleRDD(
  partitionSpecs: Array[ShufflePartitionSpec]): RDD[_]

RDD (Spark Core)

Used when:

  • CustomShuffleReaderExec physical operator is requested for the shuffleRDD

mapOutputStatisticsFuture

mapOutputStatisticsFuture: Future[MapOutputStatistics]

MapOutputStatistics (Apache Spark)

Used when:

Number of Mappers

numMappers: Int

Used when:

Number of Partitions

numPartitions: Int

Used when:

Runtime Statistics

runtimeStatistics: Statistics

Used when:

ShuffleOrigin

shuffleOrigin: ShuffleOrigin

ShuffleOrigin

Used when:

  • AdaptiveSparkPlanExec physical operator is requested for the finalStageOptimizerRules
  • CoalesceShufflePartitions physical optimization is requested to supportCoalesce
  • OptimizeShuffleWithLocalRead physical optimization is requested to supportLocalReader
  • ShuffleStage utility is used to destructure a SparkPlan to a ShuffleStageInfo

Implementations

Submitting Shuffle Job

submitShuffleJob: Future[MapOutputStatistics]

submitShuffleJob executes a query with the mapOutputStatisticsFuture.

Final Method

submitShuffleJob is a Scala final method and may not be overridden in subclasses.

Learn more in the Scala Language Specification.

submitShuffleJob is used when:

  • ShuffleQueryStageExec adaptive leaf physical operator is requested for the shuffleFuture