Skip to content

FlowAnalysis Utility

Singleton Object

FlowAnalysis is a Scala object which is a class that has exactly one instance. It is created lazily when it is referenced, like a lazy val.

Learn more in Tour of Scala.

createFlowFunctionFromLogicalPlan

createFlowFunctionFromLogicalPlan(
  plan: LogicalPlan): FlowFunction

createFlowFunctionFromLogicalPlan takes a LogicalPlan (that represents one of the supported logical commands) and creates a FlowFunction.

When executed, this FlowFunction creates a FlowAnalysisContext.

FlowFunction uses this FlowAnalysisContext to set the SQL configs (given to the FlowFunction being defined).

FlowFunction analyze this LogicalPlan (with the FlowAnalysisContext). This gives the result data (as a DataFrame).

In the end, FlowFunction creates a FlowFunctionResult with the result data (as a DataFrame) and the others (from the FlowAnalysisContext).


createFlowFunctionFromLogicalPlan is used when:

Analyze Logical Command

analyze(
  context: FlowAnalysisContext,
  plan: LogicalPlan): DataFrame

CTEs

analyze resolves pipeline-specific TVFs and CTEs.

SELECT ... FROM STREAM(t1)
SELECT ... FROM STREAM t1

Developers can define CTEs within their CREATE statements:

CREATE STREAMING TABLE a
WITH b AS (
   SELECT * FROM STREAM upstream
)
SELECT * FROM b

analyze...FIXME

Read Batch Input

readBatchInput(
  context: FlowAnalysisContext,
  name: String,
  batchReadOptions: BatchReadOptions): DataFrame

readBatchInput...FIXME


readBatchInput is used when:

  • FlowAnalysis is requested to analyze

Read External Batch Input

readExternalBatchInput(
  context: FlowAnalysisContext,
  inputIdentifier: ExternalDatasetIdentifier,
  name: String): DataFrame

readExternalBatchInput...FIXME

Read Stream Input

readStreamInput(
  context: FlowAnalysisContext,
  name: String,
  streamReader: DataStreamReader,
  streamingReadOptions: StreamingReadOptions): DataFrame

readStreamInput...FIXME


readStreamInput is used when:

  • FlowAnalysis is requested to analyze

Read External Stream Input

readExternalStreamInput(
  context: FlowAnalysisContext,
  inputIdentifier: ExternalDatasetIdentifier,
  streamReader: DataStreamReader,
  name: String): DataFrame

readExternalStreamInput...FIXME

Read Graph Input

readGraphInput(
  ctx: FlowAnalysisContext,
  inputIdentifier: InternalDatasetIdentifier,
  readOptions: InputReadOptions): DataFrame

readGraphInput...FIXME


readGraphInput is used when:

  • FlowAnalysis is requested to read batch and stream input