VariableSubstitution¶

VariableSubstitution allows for variable substitution in SQL commands (in SparkSqlParser and Spark Thrift Server).

Note

VariableSubstitution is meant for SQL commands since in programming languages there are other means like String Interpolation in Scala.

Demo¶

SQLScala


SET values = VALUES 1,2,3;


SELECT * FROM ${values};


import org.apache.spark.sql.internal.VariableSubstitution
val substitutor = new VariableSubstitution()
substitutor.substitute(...)

Creating Instance¶

VariableSubstitution takes no arguments to be created.

VariableSubstitution is created when:

SparkSqlParser is created
SparkExecuteStatementOperation (Spark Thrift Server) is created
SparkSQLDriver (Spark Thrift Server) is executed

ConfigReader¶

VariableSubstitution creates a ConfigReader (Spark Core) when created (for Variable Substitution).

This ConfigReader uses the active SQLConf to look up keys (variables) first. It then binds the same variable provider to handle the following prefixes:

spark
sparkconf
hivevar
hiveconf

Note

By default, the ConfigReader handles the other two prefixes:

env (for environment variables)
system (for Java system properties)

If a reference cannot be resolved, the original string will be retained.

Variable Substitution¶


substitute(
  input: String): String

With spark.sql.variable.substitute enabled, substitute requests the ConfigReader to substitute variables. Otherwise, substitute does nothing and simply returns the given input.

substitute is used when:

SparkSqlParser is requested to parse a command
SparkExecuteStatementOperation (Spark Thrift Server) is created
SparkSQLDriver (Spark Thrift Server) is executed