FlowAnalysis Utility¶

Singleton Object

FlowAnalysis is a Scala object which is a class that has exactly one instance. It is created lazily when it is referenced, like a lazy val.

Learn more in Tour of Scala.

createFlowFunctionFromLogicalPlan¶

createFlowFunctionFromLogicalPlan(
  plan: LogicalPlan): FlowFunction

createFlowFunctionFromLogicalPlan takes a LogicalPlan (that represents one of the supported logical commands) and creates a FlowFunction.

When executed, this FlowFunction creates a FlowAnalysisContext.

FlowFunction uses this FlowAnalysisContext to set the SQL configs (given to the FlowFunction being defined).

FlowFunction analyze this LogicalPlan (with the FlowAnalysisContext). This gives the result data (as a DataFrame).

In the end, FlowFunction creates a FlowFunctionResult with the result data (as a DataFrame) and the others (from the FlowAnalysisContext).

createFlowFunctionFromLogicalPlan is used when:

PipelinesHandler is requested to define a flow
SqlGraphRegistrationContext is requested to handle the following queries:

Analyze Logical Command¶

analyze(
  context: FlowAnalysisContext,
  plan: LogicalPlan): DataFrame

CTEs

analyze resolves pipeline-specific TVFs and CTEs.

SELECT ... FROM STREAM(t1)
SELECT ... FROM STREAM t1

Developers can define CTEs within their CREATE statements:

CREATE STREAMING TABLE a
WITH b AS (
   SELECT * FROM STREAM upstream
)
SELECT * FROM b

analyze...FIXME

Read Batch Input¶

readBatchInput(
  context: FlowAnalysisContext,
  name: String,
  batchReadOptions: BatchReadOptions): DataFrame

readBatchInput...FIXME

readBatchInput is used when:

FlowAnalysis is requested to analyze

Read External Batch Input¶

readExternalBatchInput(
  context: FlowAnalysisContext,
  inputIdentifier: ExternalDatasetIdentifier,
  name: String): DataFrame

readExternalBatchInput...FIXME

Read Stream Input¶

readStreamInput(
  context: FlowAnalysisContext,
  name: String,
  streamReader: DataStreamReader,
  streamingReadOptions: StreamingReadOptions): DataFrame

readStreamInput...FIXME

readStreamInput is used when:

FlowAnalysis is requested to analyze

Read External Stream Input¶

readExternalStreamInput(
  context: FlowAnalysisContext,
  inputIdentifier: ExternalDatasetIdentifier,
  streamReader: DataStreamReader,
  name: String): DataFrame

readExternalStreamInput...FIXME

Read Graph Input¶

readGraphInput(
  ctx: FlowAnalysisContext,
  inputIdentifier: InternalDatasetIdentifier,
  readOptions: InputReadOptions): DataFrame

readGraphInput...FIXME

readGraphInput is used when:

FlowAnalysis is requested to read batch and stream input