AstBuilder — ANTLR-based SQL Parser¶

AstBuilder converts ANTLR ParseTrees into Catalyst entities using visit callbacks.

AstBuilder is the only requirement of the AbstractSqlParser abstraction (and used by CatalystSqlParser directly while SparkSqlParser uses SparkSqlAstBuilder instead).

SqlBaseParser.g4 — ANTLR Grammar¶

AstBuilder is an ANTLR AbstractParseTreeVisitor (as SqlBaseParserBaseVisitor) that is generated from the ANTLR grammar of Spark SQL.

SqlBaseParserBaseVisitor is a ANTLR-specific base class that is generated at build time from the ANTLR grammar of Spark SQL. The Spark SQL grammar is available in the Apache Spark repository at SqlBaseParser.g4.

SqlBaseParserBaseVisitor is an AbstractParseTreeVisitor in ANTLR.

Visit Callbacks¶

visitAddTableColumns¶

Creates an AddColumns logical operator

ALTER TABLE multipartIdentifier
  ADD (COLUMN | COLUMNS)
  '(' columns=qualifiedColTypeWithPositionList ')'

qualifiedColTypeWithPositionList
    : qualifiedColTypeWithPosition (',' qualifiedColTypeWithPosition)*
    ;

qualifiedColTypeWithPosition
    : name=multipartIdentifier dataType (NOT NULL)? commentSpec? colPosition?
    ;

commentSpec
    : COMMENT text
    ;

colPosition
    : position=FIRST | position=AFTER afterCol=errorCapturingIdentifier
    ;

ANTLR labeled alternative: #addTableColumns

visitAnalyze¶

Creates an AnalyzeColumn or AnalyzeTable logical operators

ANALYZE TABLE multipartIdentifier partitionSpec? COMPUTE STATISTICS
  (NOSCAN | FOR COLUMNS identifierSeq | FOR ALL COLUMNS)?

partitionSpec
    : PARTITION '(' partitionVal (',' partitionVal)* ')'
    ;

partitionVal
    : identifier (EQ constant)?
    ;

EQ  : '=' | '==';

visitAnalyze creates an AnalyzeColumn when one of the following is used:

• FOR COLUMNS identifierSeq
• FOR ALL COLUMNS

ANTLR labeled alternative: #analyze

visitBucketSpec¶

Creates a BucketSpec

bucketSpec
  : CLUSTERED BY '(' identifierList ')'
    (SORTED BY '(' orderedIdentifierList ')' )?
    INTO digit BUCKETS
  ;

Column ordering must be ASC

visitCommentTable¶

Creates a CommentOnTable logical command

COMMENT ON TABLE tableIdentifier IS ('text' | NULL)

ANTLR labeled alternative: #commentTable

visitCommonSelectQueryClausePlan¶

Used when:

• withSelectQuerySpecification
• withTransformQuerySpecification

visitCreateTable¶

Creates a CreateTableAsSelect (for CTAS queries with AS clause) or CreateTable logical operator

CREATE TEMPORARY? EXTERNAL? TABLE (IF NOT EXISTS)? [multipartIdentifier]
  ('(' [colType] (',' [colType])* ')')?
  (USING [provider])?
  [createTableClauses]
  (AS? query)?

colType
  : colName dataType (NOT NULL)? (COMMENT [comment])?
  ;

createTableClauses:
  ((OPTIONS options=propertyList) |
  (PARTITIONED BY partitioning=partitionFieldList) |
  skewSpec |
  bucketSpec |
  rowFormat |
  createFileFormat |
  locationSpec |
  commentSpec |
  (TBLPROPERTIES tableProps=propertyList))*

ANTLR labeled alternative: #createTable

visitDeleteFromTable¶

Creates a DeleteFromTable logical command

DELETE FROM multipartIdentifier tableAlias whereClause?

ANTLR labeled alternative: #deleteFromTable

visitDescribeRelation¶

Creates a DescribeColumnStatement or DescribeRelation

(DESC | DESCRIBE) [TABLE] [EXTENDED | FORMATTED] multipartIdentifier
[partitionSpec]
[describeColName]

ANTLR labeled alternative: #describeRelation

visitDropNamespace¶

Creates a DropNamespace logical operator

DROP (NAMESPACE | DATABASE | SCHEMA) [IF EXISTS] identifierReference
[RESTRICT | CASCADE]  #dropNamespace

ANTLR labeled alternative: #dropNamespace

visitExists¶

Creates an Exists expression

ANTLR labeled alternative: #exists

visitExplain¶

Creates a ExplainCommand

ANTLR rule: explain

visitFirst¶

Creates a First aggregate function expression

FIRST '(' expression (IGNORE NULLS)? ')'

ANTLR labeled alternative: #first

visitFromClause¶

Creates a LogicalPlan

FROM relation (',' relation)* lateralView* pivotClause?

Supports multiple comma-separated relations (that all together build a condition-less INNER JOIN) with optional LATERAL VIEW.

A relation can be one of the following or a combination thereof:

• Table identifier
• Inline table using VALUES exprs AS tableIdent
• Table-valued function (currently only range is supported)

ANTLR rule: fromClause

visitFromStatement¶

fromStatement
    : fromClause fromStatementBody+
    ;

fromClause
    : FROM relation (COMMA relation)* lateralView* pivotClause? unpivotClause?
    ;

fromStatementBody
    : transformClause
      whereClause?
      queryOrganization
    | selectClause
      lateralView*
      whereClause?
      aggregationClause?
      havingClause?
      windowClause?
      queryOrganization
    ;

visitFunctionCall¶

primaryExpression:
    : functionName '(' (setQuantifier? [argument] (',' [argument])*)? ')'
      (FILTER '(' WHERE [whereExpression] ')')?
      ((IGNORE | RESPECT) NULLS)? (OVER windowSpec)?
    ...
    ;

setQuantifier
    : DISTINCT
    | ALL
    ;

Creates one of the following:

• UnresolvedFunction for a bare function (with no window specification)
• UnresolvedWindowExpression for a function evaluated in a windowed context with a WindowSpecReference
• WindowExpression for a function over a window

ANTLR rule: functionCall

import spark.sessionState.sqlParser

scala> sqlParser.parseExpression("foo()")
res0: org.apache.spark.sql.catalyst.expressions.Expression = 'foo()

scala> sqlParser.parseExpression("foo() OVER windowSpecRef")
res1: org.apache.spark.sql.catalyst.expressions.Expression = unresolvedwindowexpression('foo(), WindowSpecReference(windowSpecRef))

scala> sqlParser.parseExpression("foo() OVER (CLUSTER BY field)")
res2: org.apache.spark.sql.catalyst.expressions.Expression = 'foo() windowspecdefinition('field, UnspecifiedFrame)

visitInlineTable¶

Creates a UnresolvedInlineTable unary logical operator (as the child of SubqueryAlias for tableAlias)

VALUES expression (',' expression)* tableAlias

expression can be as follows:

• CreateNamedStruct expression for multiple-column tables

• Any Catalyst expression for one-column tables

tableAlias can be specified explicitly or defaults to colN for every column (starting from 1 for N).

ANTLR rule: inlineTable

visitInsertIntoTable¶

Creates a InsertIntoTable (indirectly)

A 3-element tuple with a TableIdentifier, optional partition keys and the exists flag disabled

INSERT INTO TABLE? tableIdentifier partitionSpec?

ANTLR labeled alternative: #insertIntoTable

visitInsertOverwriteTable¶

Creates a InsertIntoTable (indirectly)

A 3-element tuple with a TableIdentifier, optional partition keys and the exists flag

INSERT OVERWRITE TABLE tableIdentifier (partitionSpec (IF NOT EXISTS)?)?

In a way, visitInsertOverwriteTable is simply a more general version of the visitInsertIntoTable with the exists flag on or off based on existence of IF NOT EXISTS. The main difference is that dynamic partitions are used with no IF NOT EXISTS.

ANTLR labeled alternative: #insertOverwriteTable

visitInterval¶

Creates a Literal expression to represent an interval type

INTERVAL (MultiUnitsInterval | UnitToUnitInterval)?

ANTLR rule: interval

visitInterval creates a CalendarInterval.

visitInterval parses UnitToUnitInterval if specified first (and spark.sql.legacy.interval.enabled configuration property turned off).

unitToUnitInterval
    : value from TO to
    ;

month in to leads to a YearMonthIntervalType while other identifiers lead to a DayTimeIntervalType.

INTERVAL '0-0' YEAR TO MONTH        // YearMonthIntervalType
INTERVAL '0 00:00:00' DAY TO SECOND // DayTimeIntervalType

visitInterval parses MultiUnitsInterval if specified (and spark.sql.legacy.interval.enabled configuration property turned off).

multiUnitsInterval
    : (intervalValue unit)+
    ;

visitMergeIntoTable¶

visitMergeIntoTable(
  ctx: MergeIntoTableContext): LogicalPlan

Creates a MergeIntoTable logical command for a MERGE INTO DML statement

MERGE INTO target targetAlias
USING (source | (sourceQuery)) sourceAlias
ON mergeCondition
matchedClause*
notMatchedClause*
notMatchedBySourceClause*

matchedClause
  : WHEN MATCHED (AND matchedCond)? THEN matchedAction
  ;

notMatchedClause
  : WHEN NOT MATCHED (BY TARGET)? (AND notMatchedCond)? THEN notMatchedAction
  ;

notMatchedBySourceClause
  : WHEN NOT MATCHED BY SOURCE (AND notMatchedBySourceCond)? THEN notMatchedBySourceAction
  ;

matchedAction
  : DELETE
  | UPDATE SET *
  | UPDATE SET assignment (',' assignment)*

notMatchedAction
  : INSERT *
  | INSERT '(' columns ')'
    VALUES '(' expression (',' expression)* ')'

notMatchedBySourceAction
  : DELETE
  | UPDATE SET assignment (',' assignment)*
  ;

Requirements:

1. There must be at least one WHEN clause
2. When there are more than one MATCHED clauses, only the last MATCHED clause can omit the condition
3. When there are more than one NOT MATCHED clauses, only the last NOT MATCHED clause can omit the condition

ANTLR labeled alternative: #mergeIntoTable

visitMultiInsertQuery¶

Creates a logical operator with a InsertIntoTable (and UnresolvedRelation leaf operator)

FROM relation (',' relation)* lateralView*
INSERT OVERWRITE TABLE ...

FROM relation (',' relation)* lateralView*
INSERT INTO TABLE? ...

ANTLR rule: multiInsertQueryBody

visitNamedExpression¶

Creates one of the following Catalyst expressions:

• Alias (for a single alias)
• MultiAlias (for a parenthesis enclosed alias list)
• a bare Expression

ANTLR rule: namedExpression

visitNamedQuery¶

Creates a SubqueryAlias

visitPrimitiveDataType¶

Creates a primitive DataType from SQL type representation.

visitPredicated¶

Creates an Expression

ANTLR rule: predicated

visitQuerySpecification¶

Creates OneRowRelation or LogicalPlan

val q = sql("select 1")
scala> println(q.queryExecution.logical.numberedTreeString)
00 'Project [unresolvedalias(1, None)]
01 +- OneRowRelation$

ANTLR rule: querySpecification

visitRegularQuerySpecification¶

withSelectQuerySpecification with a from relation

querySpecification
    : ...                        #transformQuerySpecification
    | selectClause
      fromClause?
      lateralView*
      whereClause?
      aggregationClause?
      havingClause?
      windowClause?              #regularQuerySpecification
    ;

havingClause
    : HAVING booleanExpression
    ;

windowClause
    : WINDOW namedWindow (COMMA namedWindow)*
    ;

ANTLR rule: regularQuerySpecification

visitRelation¶

Creates a LogicalPlan for a FROM clause.

ANTLR rule: relation

visitRenameTableColumn¶

Creates a RenameColumn for the following SQL statement:

ALTER TABLE table
RENAME COLUMN from TO to

ANTLR labeled alternative: #renameTableColumn

visitRepairTable¶

Creates a RepairTable unary logical command for the following SQL statement:

MSCK REPAIR TABLE multipartIdentifier

ANTLR labeled alternative: #repairTable

visitShowColumns¶

Creates a ShowColumns logical command for the following SQL statement:

SHOW COLUMNS
  (FROM | IN) [table]
  ((FROM | IN) [ns])?

ANTLR labeled alternative: #showColumns

visitShowCreateTable¶

Creates a ShowCreateTable logical command for the following SQL statement:

SHOW CREATE TABLE multipartIdentifier (AS SERDE)?

ANTLR labeled alternative: #showCreateTable

visitShowCurrentNamespace¶

Creates a ShowCurrentNamespaceCommand logical command for the following SQL statement:

SHOW CURRENT NAMESPACE

ANTLR labeled alternative: #showCurrentNamespace

visitShowTables¶

Creates a ShowTables for the following SQL statement:

SHOW TABLES ((FROM | IN) multipartIdentifier)?
  (LIKE? pattern=STRING)?

ANTLR labeled alternative: #showTables

visitShowTblProperties¶

Creates a ShowTableProperties logical command

SHOW TBLPROPERTIES [multi-part table identifier]
  ('(' [dot-separated table property key] ')')?

ANTLR labeled alternative: #showTblProperties

visitSingleDataType¶

Creates a DataType

ANTLR rule: singleDataType

visitSingleExpression¶

Creates an Expression

Takes the named expression and relays to visitNamedExpression

ANTLR rule: singleExpression

visitSingleInsertQuery¶

Calls withInsertInto (with an InsertIntoContext) for the following SQLs:

INSERT OVERWRITE TABLE? multipartIdentifier
(partitionSpec (IF NOT EXISTS)?)?
identifierList?

INSERT INTO TABLE? multipartIdentifier
partitionSpec?
(IF NOT EXISTS)?
identifierList?

INSERT OVERWRITE LOCAL? DIRECTORY path=STRING rowFormat? createFileFormat?

INSERT OVERWRITE LOCAL? DIRECTORY (path=STRING)?
tableProvider
(OPTIONS options=tablePropertyList)?

ANTLR labeled alternative: #singleInsertQuery

visitSingleStatement¶

Creates a LogicalPlan from a single SQL statement

ANTLR rule: singleStatement

visitSortItem¶

Creates a SortOrder unary expression

sortItem
    : expression ordering=(ASC | DESC)? (NULLS nullOrder=(LAST | FIRST))?
    ;

// queryOrganization
ORDER BY order+=sortItem (',' order+=sortItem)*
SORT BY sort+=sortItem (',' sort+=sortItem)*

// windowSpec
(ORDER | SORT) BY sortItem (',' sortItem)*)?

ANTLR rule: sortItem

visitStar¶

Creates a UnresolvedStar

ANTLR labeled alternative: #star

visitSubqueryExpression¶

Creates a ScalarSubquery

ANTLR labeled alternative: #subqueryExpression

visitTableValuedFunction¶

Creates an UnresolvedTableValuedFunction

relationPrimary
  : functionTable
  ...
  ;

functionTable
    : functionName '('
      (functionTableArgument (',' functionTableArgument)*)?
      ')' tableAlias
    ;

functionName
    : 'IDENTIFIER' '(' expression ')'
    | 'IDENTIFIER'
    | qualifiedName
    | 'FILTER'
    | 'LEFT'
    | 'RIGHT'
    ;

functionTableArgument
    : functionTableReferenceArgument
    | functionArgument
    ;

functionTableReferenceArgument
    : functionTableSubqueryArgument
    | functionTableNamedArgumentExpression
    ;

functionTableSubqueryArgument
    : TABLE identifierReference
    | TABLE '(' identifierReference ')'
    | TABLE '(' query ')'
    ;

functionTableNamedArgumentExpression
    : key '=>' table=functionTableSubqueryArgument
    ;

identifierReference
    : 'IDENTIFIER' '(' expression ')'
    | multipartIdentifier
    ;

functionArgument
    : expression
    | namedArgumentExpression
    ;

namedArgumentExpression
    : key '=>' value
    ;

ANTLR labeled alternative: #tableValuedFunction

visitTruncateTable¶

Creates either a TruncatePartition or a TruncateTable logical operator based on PARTITION clause (defined or not, respectively)

TRUNCATE TABLE identifierReference
  [PARTITION (partitionVal (, partitionVal)*)]

ANTLR labeled alternative: #truncateTable

visitUpdateTable¶

Creates an UpdateTable logical operator

UPDATE multipartIdentifier tableAlias setClause whereClause?

ANTLR labeled alternative: #updateTable

visitUse¶

Creates a SetCatalogAndNamespace

USE NAMESPACE? multipartIdentifier

ANTLR labeled alternative: #use

visitWindowDef¶

Creates a WindowSpecDefinition

windowSpec
    : '('
      ( CLUSTER BY partition (',' partition)*
      | ((PARTITION | DISTRIBUTE) BY partition (',' partition)*)?
        ((ORDER | SORT) BY sortItem (',' sortItem)*)?)
      windowFrame?
      ')'
    ;

windowFrame
    : RANGE start
    | ROWS start
    | RANGE BETWEEN start AND end
    | ROWS BETWEEN start AND end
    ;

// start and end bounds of windowFrames
frameBound
    : UNBOUNDED (PRECEDING | FOLLOWING)
    | CURRENT ROW
    | literal (PRECEDING | FOLLOWING)
    ;

ANTLR rule: windowDef

Parsing Handlers¶

withAggregationClause¶

Creates an Aggregate logical operator with one of the following grouping expressions (to indicate the aggregation kind):

• GroupingSets for GROUP BY ... GROUPING SETS
• Cube for GROUP BY ... WITH CUBE
• Rollup for GROUP BY ... WITH ROLLUP
• GROUP BY ...

aggregationClause
    : GROUP BY groupingExpressionsWithGroupingAnalytics+=groupByClause
        (COMMA groupingExpressionsWithGroupingAnalytics+=groupByClause)*
    | GROUP BY groupingExpressions+=expression (COMMA groupingExpressions+=expression)* (
      WITH kind=ROLLUP
    | WITH kind=CUBE
    | kind=GROUPING SETS LEFT_PAREN groupingSet (COMMA groupingSet)* RIGHT_PAREN)?
    ;

groupByClause
    : groupingAnalytics
    | expression
    ;

groupingAnalytics
    : (ROLLUP | CUBE) LEFT_PAREN groupingSet (COMMA groupingSet)* RIGHT_PAREN
    | GROUPING SETS LEFT_PAREN groupingElement (COMMA groupingElement)* RIGHT_PAREN
    ;

groupingElement
    : groupingAnalytics
    | groupingSet
    ;

groupingSet
    : LEFT_PAREN (expression (COMMA expression)*)? RIGHT_PAREN
    | expression
    ;

Used in visitCommonSelectQueryClausePlan

withCTE¶

Creates an UnresolvedWith logical operator for Common Table Expressions (in visitQuery and visitDmlStatement)

WITH namedQuery (',' namedQuery)*

namedQuery
    : name (columnAliases)? AS? '(' query ')'
    ;

withFromStatementBody¶

Used in visitFromStatement and visitMultiInsertQuery

withGenerate¶

withGenerate(
  query: LogicalPlan,
  ctx: LateralViewContext): LogicalPlan

Creates a Generate logical operator (with an UnresolvedGenerator) to represent LATERAL VIEWs in SELECT and FROM clauses.

lateralView
  : LATERAL VIEW (OUTER)? qualifiedName '(' (expression (',' expression)*)? ')' tblName (AS? colName (',' colName)*)?
  ;

withHavingClause¶

Creates an UnresolvedHaving for the following:

HAVING booleanExpression

withHints¶

Adds an UnresolvedHint for /*+ hint */ in SELECT queries.

hint is of the format name or name (param1, param2, ...).

/*+ BROADCAST (table) */

withInsertInto¶

Creates one of the following logical operators:

• InsertIntoStatement
• InsertIntoDir

Used in visitMultiInsertQuery and visitSingleInsertQuery

withJoinRelations¶

Creates one or more Join logical operators for a FROM clause and relation.

The following join types are supported:

• INNER (default)
• CROSS
• LEFT (with optional OUTER)
• LEFT SEMI
• RIGHT (with optional OUTER)
• FULL (with optional OUTER)
• ANTI (optionally prefixed with LEFT)

The following join criteria are supported:

• ON booleanExpression
• USING '(' identifier (',' identifier)* ')'

Joins can be NATURAL (with no join criteria)

withQuerySpecification¶

Adds a query specification to a logical operator

For transform SELECT (with TRANSFORM, MAP or REDUCE qualifiers), withQuerySpecification does...FIXME

For regular SELECT (no TRANSFORM, MAP or REDUCE qualifiers), withQuerySpecification adds (in that order):

1. Generate unary logical operators (if used in the parsed SQL text)

2. Filter unary logical plan (if used in the parsed SQL text)

3. GroupingSets or Aggregate unary logical operators (if used in the parsed SQL text)

4. Project and/or Filter unary logical operators

5. WithWindowDefinition unary logical operator (if used in the parsed SQL text)

6. UnresolvedHint unary logical operator (if used in the parsed SQL text)

withPredicate¶

Creates a InSubquery over a ListQuery (possibly "inverted" using Not unary expression)

NOT? IN '(' query ')'

withPivot¶

Creates a Pivot unary logical operator for the following SQL clause:

PIVOT '(' aggregates FOR pivotColumn IN '(' pivotValue (',' pivotValue)* ')' ')'

Used in visitFromClause

withRepartitionByExpression¶

withRepartitionByExpression throws a ParseException:

DISTRIBUTE BY is not supported

Used in withQueryResultClauses

withSelectQuerySpecification¶

visitCommonSelectQueryClausePlan followed by withHints if there are hints

Used when:

• visitRegularQuerySpecification
• withFromStatementBody

withTimeTravel¶

withTimeTravel creates a RelationTimeTravel for the following in a SQL query:

temporalClause
    : FOR? (SYSTEM_VERSION | VERSION) AS OF version
    | FOR? (SYSTEM_TIME | TIMESTAMP) AS OF timestamp
    ;

relationPrimary
    : identifierReference temporalClause?
      sample? tableAlias                                    #tableName
    ...

Used in visitTableName (to handle the optional temporalClause).

withTransformQuerySpecification¶

Creates ScriptTransformation unary logical operator for a Hive-style transform (SELECT TRANSFORM/MAP/REDUCE) query specification

querySpecification
    : transformClause
      fromClause?
      lateralView*
      whereClause?
      aggregationClause?
      havingClause?
      windowClause?                                 #transformQuerySpecification
    | ...                                           #regularQuerySpecification
    ;

Used when:

• visitTransformQuerySpecification
• withFromStatementBody

withWindows¶

Adds a WithWindowDefinition for window aggregates (given WINDOW definitions).

WINDOW identifier AS windowSpec
  (',' identifier AS windowSpec)*

Used in withQueryResultClauses and withQuerySpecification

parseIntervalLiteral¶

parseIntervalLiteral(
  ctx: IntervalContext): CalendarInterval

parseIntervalLiteral creates a CalendarInterval (using visitMultiUnitsInterval and visitUnitToUnitInterval).

parseIntervalLiteral is used when:

• AstBuilder is requested to visitInterval
• SparkSqlAstBuilder is requested to visitSetTimeZone

extractFunctionTableNamedArgument¶

extractFunctionTableNamedArgument(
  expr: FunctionTableReferenceArgumentContext,
  funcName: String) : Expression

If functionTableNamedArgumentExpression is used with spark.sql.allowNamedFunctionArguments enabled, extractFunctionTableNamedArgument creates a NamedArgumentExpression for the key with visitFunctionTableSubqueryArgument as the value. With spark.sql.allowNamedFunctionArguments disabled, extractFunctionTableNamedArgument reports an AnalysisException.

functionTableNamedArgumentExpression
    : key=identifier '=>' table=functionTableSubqueryArgument
    ;

With no functionTableNamedArgumentExpression used, extractFunctionTableNamedArgument visitFunctionTableSubqueryArgument (with functionTableSubqueryArgument).

extractFunctionTableNamedArgument is used when:

• AstBuilder is requested to visitTableValuedFunction

extractNamedArgument¶

extractNamedArgument(
  expr: FunctionArgumentContext,
  funcName: String) : Expression

If namedArgumentExpression is used with spark.sql.allowNamedFunctionArguments enabled, extractNamedArgument creates a NamedArgumentExpression for the key (text) and the value (as an Expression). With spark.sql.allowNamedFunctionArguments disabled, extractNamedArgument reports an AnalysisException.

namedArgumentExpression
    : key=identifier '=>' value=expression
    ;

With no namedArgumentExpression used, extractNamedArgument creates an Expression.

extractNamedArgument is used when:

• AstBuilder is requested to visitFunctionCall and visitTableValuedFunction