SparkSessionExtensions¶

SparkSessionExtensions is an "injection" API for Spark SQL developers to extend the features of a SparkSession.

Spark SQL developers use Builder.withExtensions method or register custom extensions using spark.sql.extensions configuration property.

SparkSessionExtensions is an integral part of SparkSession.

injectedFunctions¶

SparkSessionExtensions uses a collection of 3-element tuples with the following:

1. FunctionIdentifier
2. ExpressionInfo
3. Seq[Expression] => Expression

"Injection" API¶

injectCheckRule¶

type CheckRuleBuilder = SparkSession => LogicalPlan => Unit
injectCheckRule(
  builder: CheckRuleBuilder): Unit

injectCheckRule injects an check analysis Rule builder into a SparkSession.

The injected rules will be executed after the analysis phase. A check analysis rule is used to detect problems with a LogicalPlan and should throw an exception when a problem is found.

injectColumnar¶

type ColumnarRuleBuilder = SparkSession => ColumnarRule
injectColumnar(
  builder: ColumnarRuleBuilder): Unit

Injects a ColumnarRule to a SparkSession

injectFunction¶

type FunctionDescription = (FunctionIdentifier, ExpressionInfo, FunctionBuilder)
injectFunction(
  functionDescription: FunctionDescription): Unit

injectFunction...FIXME

injectOptimizerRule¶

type RuleBuilder = SparkSession => Rule[LogicalPlan]
injectOptimizerRule(
  builder: RuleBuilder): Unit

injectOptimizerRule registers a custom logical optimization rules builder.

injectParser¶

type ParserBuilder = (SparkSession, ParserInterface) => ParserInterface
injectParser(
  builder: ParserBuilder): Unit

injectParser...FIXME

injectPlannerStrategy¶

type StrategyBuilder = SparkSession => Strategy
injectPlannerStrategy(
  builder: StrategyBuilder): Unit

injectPlannerStrategy...FIXME

injectPostHocResolutionRule¶

type RuleBuilder = SparkSession => Rule[LogicalPlan]
injectPostHocResolutionRule(
  builder: RuleBuilder): Unit

injectPostHocResolutionRule...FIXME

injectQueryStagePrepRule¶

type QueryStagePrepRuleBuilder = SparkSession => Rule[SparkPlan]
injectQueryStagePrepRule(
  builder: QueryStagePrepRuleBuilder): Unit

injectQueryStagePrepRule...FIXME

injectResolutionRule¶

type RuleBuilder = SparkSession => Rule[LogicalPlan]
injectResolutionRule(
  builder: RuleBuilder): Unit

injectResolutionRule...FIXME

Registering Custom Logical Optimization Rules¶

buildOptimizerRules(
  session: SparkSession): Seq[Rule[LogicalPlan]]

buildOptimizerRules gives the optimizerRules logical rules given the input SparkSession.

buildOptimizerRules is used when BaseSessionStateBuilder is requested for the custom operator optimization rules (to add to the base Operator Optimization batch).

Logical Optimizer Rules (Builder)¶

optimizerRules: Buffer[SparkSession => Rule[LogicalPlan]]

optimizerRules are functions (builders) that take a SparkSession and return logical optimizer rules (Rule[LogicalPlan]).

optimizerRules is added a new rule when SparkSessionExtensions is requested to injectOptimizerRule.

buildColumnarRules Internal Method¶

buildColumnarRules(
  session: SparkSession): Seq[ColumnarRule]

buildColumnarRules...FIXME

buildColumnarRules is used when BaseSessionStateBuilder is requested for columnarRules.

buildCheckRules¶

buildCheckRules(
  session: SparkSession): Seq[LogicalPlan => Unit]

buildCheckRules...FIXME

buildCheckRules is used when BaseSessionStateBuilder is requested to customCheckRules.