Skip to content

DAGSchedulerEventProcessLoop

DAGSchedulerEventProcessLoop is an event processing daemon thread to handle DAGSchedulerEvents (on a separate thread from the parent DAGScheduler's).

DAGSchedulerEventProcessLoop is registered under the name of dag-scheduler-event-loop.

DAGSchedulerEventProcessLoop uses java.util.concurrent.LinkedBlockingDeque blocking deque that can grow indefinitely.

Creating Instance

DAGSchedulerEventProcessLoop takes the following to be created:

DAGSchedulerEventProcessLoop is created when:

Processing Event

DAGSchedulerEvent Event Handler
AllJobsCancelled doCancelAllJobs
BeginEvent handleBeginEvent
CompletionEvent handleTaskCompletion
ExecutorAdded handleExecutorAdded
ExecutorLost handleExecutorLost
GettingResultEvent handleGetTaskResult
JobCancelled handleJobCancellation
JobGroupCancelled handleJobGroupCancelled
JobSubmitted handleJobSubmitted
MapStageSubmitted handleMapStageSubmitted
ResubmitFailedStages resubmitFailedStages
SpeculativeTaskSubmitted handleSpeculativeTaskSubmitted
StageCancelled handleStageCancellation
TaskSetFailed handleTaskSetFailed
WorkerRemoved handleWorkerRemoved

ShuffleMergeFinalized

messageProcessingTime Timer

DAGSchedulerEventProcessLoop uses messageProcessingTime timer to measure time of processing events.