PipelineEventSender¶

PipelineEventSender is used by PipelinesHandler to send pipelines execution progress events back to a Spark Connect client asynchronously.

PipelineEventSender uses spark.sql.pipelines.event.queue.capacity configuration property to control the depth of the event queue.

Creating Instance¶

PipelineEventSender takes the following to be created:

• StreamObserver[ExecutePlanResponse]
• SessionHolder

PipelineEventSender is created when:

• PipelinesHandler is requested to start a pipeline run

queueCapacity¶

queueCapacity is the value of spark.sql.pipelines.event.queue.capacity configuration property.

Used when:

• PipelineEventSender is requested to shouldEnqueueEvent

Send Pipeline Execution Progress Event¶

sendEvent(
  event: PipelineEvent): Unit

sendEvent...FIXME

sendEvent is used when:

• PipelinesHandler is requested to start a pipeline run

shouldEnqueueEvent¶

shouldEnqueueEvent(
  event: PipelineEvent): Boolean

shouldEnqueueEvent...FIXME