Skip to content


SQLExecutionRDD is an RDD[InternalRow] to wrap the parent RDD and make sure that the SQL configuration properties are always propagated to executors (even when rdd or QueryExecution.toRdd are used).


Review SPARK-28939 to learn when and why SQLExecutionRDD would be used outside a tracked SQL operation (and with no defined).

Creating Instance

SQLExecutionRDD takes the following to be created:

While being created, SQLExecutionRDD initializes a sqlConfigs internal registry.

SQLExecutionRDD is created when:

  • QueryExecution is requested to toRdd


SQLExecutionRDD is given an RDD[InternalRow] when created.

The RDD[InternalRow] is the executedPlan requested to execute.


SQLExecutionRDD requests the given SQLConf for all the configuration properties that have been set when created.

Lazy Value

sqlConfigs is a Scala lazy value to guarantee that the code to initialize it is executed once only (when accessed for the first time) and the computed value never changes afterwards.

Learn more in the Scala Language Specification.

Computing Partition

  split: Partition,
  context: TaskContext): Iterator[InternalRow]

compute looks up the local property in the given TaskContext (Apache Spark).

If not defined (null), compute sets the sqlConfigs as thread-local properties to requests the sqlRDD for iterator (execute the sqlRDD). Otherwise, if in the context of a tracked SQL operation (and the is defined), compute simply requests the parent sqlRDD for iterator.

compute is part of the RDD (Apache Spark) abstraction.