SparkSessionExtensions¶
SparkSessionExtensions
is an "injection" API for Spark SQL developers to extend the features of a SparkSession.
Spark SQL developers use Builder.withExtensions method or register custom extensions using spark.sql.extensions configuration property.
SparkSessionExtensions
is an integral part of SparkSession.
injectedFunctions¶
SparkSessionExtensions
uses a collection of 3-element tuples with the following:
FunctionIdentifier
ExpressionInfo
Seq[Expression] => Expression
"Injection" API¶
injectCheckRule¶
type CheckRuleBuilder = SparkSession => LogicalPlan => Unit
injectCheckRule(
builder: CheckRuleBuilder): Unit
injectCheckRule
injects an check analysis Rule
builder into a SparkSession.
The injected rules will be executed after the analysis phase. A check analysis rule is used to detect problems with a LogicalPlan and should throw an exception when a problem is found.
injectColumnar¶
type ColumnarRuleBuilder = SparkSession => ColumnarRule
injectColumnar(
builder: ColumnarRuleBuilder): Unit
Injects a ColumnarRule to a SparkSession
injectFunction¶
type FunctionDescription = (FunctionIdentifier, ExpressionInfo, FunctionBuilder)
injectFunction(
functionDescription: FunctionDescription): Unit
injectFunction
...FIXME
injectOptimizerRule¶
type RuleBuilder = SparkSession => Rule[LogicalPlan]
injectOptimizerRule(
builder: RuleBuilder): Unit
injectOptimizerRule
registers a custom logical optimization rules builder.
injectParser¶
type ParserBuilder = (SparkSession, ParserInterface) => ParserInterface
injectParser(
builder: ParserBuilder): Unit
injectParser
...FIXME
injectPlannerStrategy¶
type StrategyBuilder = SparkSession => Strategy
injectPlannerStrategy(
builder: StrategyBuilder): Unit
injectPlannerStrategy
...FIXME
injectPostHocResolutionRule¶
type RuleBuilder = SparkSession => Rule[LogicalPlan]
injectPostHocResolutionRule(
builder: RuleBuilder): Unit
injectPostHocResolutionRule
...FIXME
injectQueryStagePrepRule¶
type QueryStagePrepRuleBuilder = SparkSession => Rule[SparkPlan]
injectQueryStagePrepRule(
builder: QueryStagePrepRuleBuilder): Unit
injectQueryStagePrepRule
...FIXME
injectResolutionRule¶
type RuleBuilder = SparkSession => Rule[LogicalPlan]
injectResolutionRule(
builder: RuleBuilder): Unit
injectResolutionRule
...FIXME
Registering Custom Logical Optimization Rules¶
buildOptimizerRules(
session: SparkSession): Seq[Rule[LogicalPlan]]
buildOptimizerRules
gives the optimizerRules logical rules given the input SparkSession.
buildOptimizerRules
is used when BaseSessionStateBuilder
is requested for the custom operator optimization rules (to add to the base Operator Optimization batch).
Logical Optimizer Rules (Builder)¶
optimizerRules: Buffer[SparkSession => Rule[LogicalPlan]]
optimizerRules
are functions (builders) that take a SparkSession and return logical optimizer rules (Rule[LogicalPlan]
).
optimizerRules
is added a new rule when SparkSessionExtensions
is requested to injectOptimizerRule.
buildColumnarRules Internal Method¶
buildColumnarRules(
session: SparkSession): Seq[ColumnarRule]
buildColumnarRules
...FIXME
buildColumnarRules
is used when BaseSessionStateBuilder
is requested for columnarRules.
buildCheckRules¶
buildCheckRules(
session: SparkSession): Seq[LogicalPlan => Unit]
buildCheckRules
...FIXME
buildCheckRules
is used when BaseSessionStateBuilder
is requested to customCheckRules.