ParserInterface¶

ParserInterface is the abstraction of SQL parsers that can convert (parse) textual representation of SQL statements (SQL text) into Spark SQL's relational entities (e.g. Catalyst expressions, logical operators, table and function identifiers, table schema, and data types).

Accessing ParserInterface¶

ParserInterface is available as SessionState.sqlParser.

scala> :type spark
org.apache.spark.sql.SparkSession

scala> :type spark.sessionState.sqlParser
org.apache.spark.sql.catalyst.parser.ParserInterface

Contract¶

parseDataType¶

parseDataType(
  sqlText: String): DataType

Creates a DataType from the given SQL text

parseExpression¶

parseExpression(
  sqlText: String): Expression

Creates an Expression from the given SQL text

Used when:

• Dataset is requested to selectExpr, filter, where
• expr standard function is used

parseFunctionIdentifier¶

parseFunctionIdentifier(
  sqlText: String): FunctionIdentifier

Creates a FunctionIdentifier from the given SQL text

Used when:

• SessionCatalog is requested to listFunctions

• CatalogImpl is requested to getFunction and functionExists

parseMultipartIdentifier¶

parseMultipartIdentifier(
  sqlText: String): Seq[String]

Creates a multi-part identifier from the given SQL text

Used when:

• CatalogV2Implicits utility is requested to parseColumnPath
• LogicalExpressions utility is requested to parseReference

• DataFrameWriter is requested to insertInto and saveAsTable

• DataFrameWriterV2 is created (and requested for tableName)

• SparkSession is requested to table

parsePlan¶

parsePlan(
  sqlText: String): LogicalPlan

Creates a LogicalPlan from the given SQL text

Used when:

• SessionCatalog is requested to fromCatalogTable
• SparkSession is requested to execute a SQL query

parseTableIdentifier¶

parseTableIdentifier(
  sqlText: String): TableIdentifier

Creates a TableIdentifier from the given SQL text

Used when:

• DataFrameWriter is requested to insertInto and saveAsTable

• Dataset is requested to createTempViewCommand

• SparkSession is requested to table

• CatalogImpl is requested to listColumns, getTable, tableExists, createTable, recoverPartitions, uncacheTable, and refreshTable

• SessionState is requested to <>

parseTableSchema¶

parseTableSchema(
  sqlText: String): StructType

Creates a StructType from the given SQL text

Implementations¶

• AbstractSqlParser