LogicalPlanner¶

LogicalPlanner is used (directly or indirectly through QueryEngine) by EngineExecutor for building a logical plan of the following:

• Non-persistent queries
• Persistent queries

Creating Instance¶

LogicalPlanner takes the following to be created:

• KsqlConfig
• ImmutableAnalysis
• MetaStore

LogicalPlanner is created when:

• EngineExecutor is requested to build a query logical plan
• QueryEngine is requested to buil a query logical plan

ImmutableAnalysis and RewrittenAnalysis¶

LogicalPlanner creates a RewrittenAnalysis with the following when created:

• ImmutableAnalysis
• ColumnReferenceRewriter.process as the rewriter

The RewrittenAnalysis is used when LogicalPlanner is requested for the following:

• buildSourceNode
• buildPersistentLogicalPlan
• buildQueryLogicalPlan
• buildOutputNode
• getWindowInfo
• getSinkTopic
• getSinkKeyFormat
• getTargetSchema
• buildAggregateNode
• buildUserProjectNode
• buildFlatMapNode
• buildJoinKey
• buildAggregateSchema

Building Logical Plan of Persistent Query¶

OutputNode buildPersistentLogicalPlan()

In summary, buildPersistentLogicalPlan creates an OutputNode for the RewrittenAnalysis (of a Query statement).

buildPersistentLogicalPlan is used when:

• QueryEngine is requested to build a logical plan of a query

Step 1. Source Node (FROM)¶

buildPersistentLogicalPlan builds a source node (with isWindowed flag based on the RewrittenAnalysis of the query).

Step 2. FilterNode (WHERE)¶

For a query with WHERE clause (per the RewrittenAnalysis), buildPersistentLogicalPlan creates a new QueryFilterNode to be the current PlanNode.

Step 3. UserRepartitionNode (PARTITION BY)¶

For a query with PartitionBy clause (per the RewrittenAnalysis), buildPersistentLogicalPlan buildUserRepartitionNode.

Step 4. FlatMapNode¶

For a query with TableFunctions (per the RewrittenAnalysis), buildPersistentLogicalPlan buildFlatMapNode.

Step 5. AggregateNode (GROUP BY)¶

For a query with GroupBy clause (per the RewrittenAnalysis), buildPersistentLogicalPlan buildAggregateNode. Otherwise, buildPersistentLogicalPlan...FIXME

Step 6. SuppressNode (EMIT FINAL)¶

For a query with a EMIT FINAL (per the RewrittenAnalysis), buildPersistentLogicalPlan...FIXME

OutputNode¶

In the end, buildPersistentLogicalPlan builds an output node.

Building Query Logical Plan¶

OutputNode buildQueryLogicalPlan(
  QueryPlannerOptions queryPlannerOptions,
  boolean isScalablePush)

buildQueryLogicalPlan is used when:

• EngineExecutor is requested to build a query logical plan

Step 1. Source Node¶

buildQueryLogicalPlan builds a source node (with isWindowed flag based on the RewrittenAnalysis of the query).

Step 2. Filter Node¶

For a query with WHERE clause (per the RewrittenAnalysis), buildQueryLogicalPlan creates a new QueryFilterNode to be the current PlanNode. Otherwise, buildQueryLogicalPlan throws a KsqlException for a missing WHERE clause unless getTableScansEnabled is enabled.

Step 3. Limit Node¶

For a non-isScalablePush query with LIMIT clause, buildQueryLogicalPlan builds a limit PlanNode to be the current PlanNode.

Step 4. Project Node¶

In the end, buildQueryLogicalPlan builds an output node with a QueryProjectNode.

Building Source PlanNode¶

PlanNode buildSourceNode(
  boolean isWindowed)

buildSourceNode builds a non-join node when the RewrittenAnalysis is not of a join query.

Otherwise, buildSourceNode...FIXME

buildSourceNode is used when:

• LogicalPlanner is requested for a persistent query logical plan or query logical plan

Creating DataSourceNode¶

DataSourceNode buildNonJoinNode(
  AliasedDataSource dataSource,
  boolean isWindowed,
  KsqlConfig ksqlConfig)

buildNonJoinNode creates a DataSourceNode (with a new PlanNodeId with KsqlTopic ID).

Creating JoinNode¶

JoinNode buildJoin(
  Join root,
  String prefix,
  boolean isWindowed)

buildJoin creates a JoinNode.

Building Output Node¶

OutputNode buildOutputNode(
  PlanNode sourcePlanNode)

buildOutputNode creates one of the two possible OutputNodes:

• KsqlBareOutputNode when the RewrittenAnalysis has no Into sink
• KsqlStructuredDataOutputNode, otherwise

buildOutputNode is used when:

• LogicalPlanner is requested to buildPersistentLogicalPlan and buildQueryLogicalPlan

Building FlatMapNode¶

FlatMapNode buildFlatMapNode(
  PlanNode sourcePlanNode)

buildFlatMapNode creates a FlatMapNode (with FlatMap ID).

Building SuppressNode¶

SuppressNode buildSuppressNode(
  PlanNode sourcePlanNode,
  RefinementInfo refinementInfo)

buildSuppressNode creates a SuppressNode (with Suppress ID).