ExperimentalMethods¶

ExperimentalMethods holds extra <> and <> that are used in SparkOptimizer and SparkPlanner, respectively.

[[attributes]] .ExperimentalMethods' Attributes [width="100%",cols="1m,2",options="header"] |=== | Name | Description

| extraOptimizations a| [[extraOptimizations]] Collection of catalyst/Rule.md[rules] to optimize spark-sql-LogicalPlan.md[LogicalPlans] (i.e. Rule[LogicalPlan] objects)

[source, scala]¶

extraOptimizations: Seq[Rule[LogicalPlan]]¶

Used when SparkOptimizer is requested for the User Provided Optimizers

| extraStrategies a| [[extraStrategies]] Collection of SparkStrategies

[source, scala]¶

extraStrategies: Seq[Strategy]¶

Used when SessionState is requested for the SessionState.md#planner[SparkPlanner] |===

ExperimentalMethods is available as the <> property of a SparkSession.

[source, scala]¶

scala> :type spark org.apache.spark.sql.SparkSession

scala> :type spark.experimental org.apache.spark.sql.ExperimentalMethods

=== Example

[source, scala]¶

import org.apache.spark.sql.catalyst.rules.Rule import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan

object SampleRule extends Rule[LogicalPlan] { def apply(p: LogicalPlan): LogicalPlan = p }

scala> :type spark org.apache.spark.sql.SparkSession

spark.experimental.extraOptimizations = Seq(SampleRule)

// extraOptimizations is used in Spark Optimizer val rule = spark.sessionState.optimizer.batches.flatMap(.rules).filter( == SampleRule).head scala> rule.ruleName res0: String = SampleRule