DAGSchedulerEvent¶

DAGSchedulerEvent is an abstraction of events that are handled by the DAGScheduler (on dag-scheduler-event-loop daemon thread).

AllJobsCancelled¶

Carries no extra information

Posted when DAGScheduler is requested to cancelAllJobs

Event handler: doCancelAllJobs

BeginEvent¶

Carries the following:

• Task
• TaskInfo

Posted when DAGScheduler is requested to taskStarted

Event handler: handleBeginEvent

CompletionEvent¶

Carries the following:

• Completed Task
• TaskEndReason
• Result (value computed)
• AccumulatorV2 Updates
• Metric Peaks
• TaskInfo

Posted when DAGScheduler is requested to taskEnded

Event handler: handleTaskCompletion

ExecutorAdded¶

Carries the following:

• Executor ID
• Host name

Posted when DAGScheduler is requested to executorAdded

Event handler: handleExecutorAdded

ExecutorLost¶

Carries the following:

• Executor ID
• Reason

Posted when DAGScheduler is requested to executorLost

Event handler: handleExecutorLost

GettingResultEvent¶

Carries the following:

• TaskInfo

Posted when DAGScheduler is requested to taskGettingResult

Event handler: handleGetTaskResult

JobCancelled¶

JobCancelled event carries the following:

• Job ID
• Reason (optional)

Posted when DAGScheduler is requested to cancelJob

Event handler: handleJobCancellation

JobGroupCancelled¶

Carries the following:

• Group ID

Posted when DAGScheduler is requested to cancelJobGroup

Event handler: handleJobGroupCancelled

JobSubmitted¶

Carries the following:

• Job ID
• RDD
• Partition processing function (with a TaskContext and the partition data, i.e. (TaskContext, Iterator[_]) => _)
• Partition IDs to compute
• CallSite
• JobListener to keep updated about the status of the stage execution
• Execution properties

Posted when:

• DAGScheduler is requested to submit a job, run an approximate job and handleJobSubmitted

Event handler: handleJobSubmitted

MapStageSubmitted¶

Carries the following:

• Job ID
• ShuffleDependency
• CallSite
• JobListener
• Execution properties

Posted when:

• DAGScheduler is requested to submit a MapStage for execution

Event handler: handleMapStageSubmitted

ResubmitFailedStages¶

Carries no extra information.

Posted when DAGScheduler is requested to handleTaskCompletion

Event handler: resubmitFailedStages

ShuffleMergeFinalized¶

Carries the following:

• ShuffleMapStage

Posted when:

• DAGScheduler is requested to finalizeShuffleMerge

Event handler: handleShuffleMergeFinalized

SpeculativeTaskSubmitted¶

Carries the following:

• Task

Posted when DAGScheduler is requested to speculativeTaskSubmitted

Event handler: handleSpeculativeTaskSubmitted

StageCancelled¶

Carries the following:

• Stage ID
• Reason (optional)

Posted when DAGScheduler is requested to cancelStage

Event handler: handleStageCancellation

TaskSetFailed¶

Carries the following:

• TaskSet
• Reason
• Exception (optional)

Posted when DAGScheduler is requested to taskSetFailed

Event handler: handleTaskSetFailed

WorkerRemoved¶

Carries the following:

• Worked ID
• Host name
• Reason

Posted when DAGScheduler is requested to workerRemoved

Event handler: handleWorkerRemoved