Skip to content

VariableSubstitution

VariableSubstitution allows for variable substitution in SQL commands (in SparkSqlParser and Spark Thrift Server).

Note

VariableSubstitution is meant for SQL commands since in programming languages there are other means like String Interpolation in Scala.

Demo

SET values = VALUES 1,2,3;

SELECT * FROM ${values};

import org.apache.spark.sql.internal.VariableSubstitution
val substitutor = new VariableSubstitution()
substitutor.substitute(...)

Creating Instance

VariableSubstitution takes no arguments to be created.

VariableSubstitution is created when:

  • SparkSqlParser is created
  • SparkExecuteStatementOperation (Spark Thrift Server) is created
  • SparkSQLDriver (Spark Thrift Server) is executed

ConfigReader

VariableSubstitution creates a ConfigReader (Spark Core) when created (for Variable Substitution).

This ConfigReader uses the active SQLConf to look up keys (variables) first. It then binds the same variable provider to handle the following prefixes:

  • spark
  • sparkconf
  • hivevar
  • hiveconf

Note

By default, the ConfigReader handles the other two prefixes:

  • env (for environment variables)
  • system (for Java system properties)

If a reference cannot be resolved, the original string will be retained.

Variable Substitution

substitute(
  input: String): String

With spark.sql.variable.substitute enabled, substitute requests the ConfigReader to substitute variables. Otherwise, substitute does nothing and simply returns the given input.

substitute is used when:

  • SparkSqlParser is requested to parse a command
  • SparkExecuteStatementOperation (Spark Thrift Server) is created
  • SparkSQLDriver (Spark Thrift Server) is executed