StatefulOperatorPartitioning¶

getCompatibleDistribution¶

getCompatibleDistribution(
  expressions: Seq[Expression],
  numPartitions: Int,
  conf: SQLConf): Distribution
getCompatibleDistribution(
  expressions: Seq[Expression],
  stateInfo: StatefulOperatorStateInfo,
  conf: SQLConf): Distribution

getCompatibleDistribution returns the following Distributions (Spark SQL) based on spark.sql.streaming.statefulOperator.useStrictDistribution configuration property:

• StatefulOpClusteredDistribution when enabled
• ClusteredDistribution (Spark SQL) otherwise

getCompatibleDistribution is used when:

• BaseAggregateExec (Spark SQL) is requested for the required child output distribution (of a streaming query)
• UpdatingSessionsExec is requested for the required child output distribution (of a streaming query)
• FlatMapGroupsWithStateExec is requested for the required child output distribution
• StateStoreRestoreExec is requested for the required child output distribution
• StateStoreSaveExec is requested for the required child output distribution
• SessionWindowStateStoreRestoreExec is requested for the required child output distribution
• SessionWindowStateStoreSaveExec is requested for the required child output distribution
• StreamingDeduplicateExec is requested for the required child output distribution