Flow¶

Flow is an extension of the GraphElement abstraction for flows in dataflow graphs.

Flows must be successfully analyzed, thus resolved, in order to determine whether they are streaming or not.

Contract (Subset)¶

FlowFunction¶

func: FlowFunction

FlowFunction of this Flow

Used to create an UnresolvedFlow

See:

• ResolutionCompletedFlow
• UnresolvedFlow

once¶

once: Boolean

Indicates whether this is a ONCE flow or not. ONCE flows can only be run once per full refresh.

• ONCE flows are planned for execution as AppendOnceFlows (FlowResolver)
• ONCE flows are marked as IDLE when TriggeredGraphExecution is requested to start flows in topologicalExecution.
• ONCE flows must be batch (not streaming).
• For ONCE flows or when the logical plan for the flow is streaming, GraphElementTypeUtils considers a ResolvedFlow as a STREAMING_TABLE (in getDatasetTypeForMaterializedViewOrStreamingTable).

Default: false

See:

• AppendOnceFlow
• UnresolvedFlow

Used when:

• TriggeredGraphExecution is requested to topologicalExecution
• PipelinesErrors is requested to checkStreamingErrorsAndRetry (to skip ONCE flows with no exception)
• GraphValidations is requested to validateFlowStreamingness
• GraphElementTypeUtils is requested to getDatasetTypeForMaterializedViewOrStreamingTable
• FlowResolver is requested to convertResolvedToTypedFlow

Implementations¶

• ResolutionCompletedFlow
• UnresolvedFlow