DAGScheduler Events

AllJobsCancelled

AllJobsCancelled event carries no extra information.

Posted when DAGScheduler is requested to cancelAllJobs

Event handler: doCancelAllJobs

BeginEvent

BeginEvent event carries the following:

Posted when DAGScheduler is requested to taskStarted

Event handler: handleBeginEvent

CompletionEvent

CompletionEvent event carries the following:

  • Task

  • Reason

  • Result (value computed)

  • Accumulator updates

  • TaskInfo

Posted when DAGScheduler is requested to taskEnded

Event handler: handleTaskCompletion

ExecutorAdded

ExecutorAdded event carries the following:

  • Executor ID

  • Host name

Posted when DAGScheduler is requested to executorAdded

Event handler: handleExecutorAdded

ExecutorLost

ExecutorLost event carries the following:

  • Executor ID

  • Reason

Posted when DAGScheduler is requested to executorLost

Event handler: handleExecutorLost

GettingResultEvent

GettingResultEvent event carries the following:

Posted when DAGScheduler is requested to taskGettingResult

Event handler: handleGetTaskResult

JobCancelled

JobCancelled event carries the following:

  • Job ID

  • Reason (optional)

Posted when DAGScheduler is requested to cancelJob

Event handler: handleJobCancellation

JobGroupCancelled

JobGroupCancelled event carries the following:

  • Group ID

Posted when DAGScheduler is requested to cancelJobGroup

Event handler: handleJobGroupCancelled

JobSubmitted

JobSubmitted event carries the following:

  • Job ID

  • RDD

  • Partition function ((TaskContext, Iterator[_]) ⇒ _)

  • Partitions to compute

  • CallSite

  • JobListener to keep updated about the status of the stage execution

  • Execution properties

Posted when DAGScheduler is requested to submit a job, run an approximate job and handleJobSubmitted

Event handler: handleJobSubmitted

MapStageSubmitted

MapStageSubmitted event carries the following:

Posted when DAGScheduler is requested to submitMapStage

Event handler: handleMapStageSubmitted

ResubmitFailedStages

ResubmitFailedStages event carries no extra information.

Posted when DAGScheduler is requested to handleTaskCompletion

Event handler: resubmitFailedStages

SpeculativeTaskSubmitted

SpeculativeTaskSubmitted event carries the following:

Posted when DAGScheduler is requested to speculativeTaskSubmitted

StageCancelled

StageCancelled event carries the following:

  • Stage ID

  • Reason (optional)

Posted when DAGScheduler is requested to cancelStage

Event handler: handleStageCancellation

TaskSetFailed

TaskSetFailed event carries the following:

  • TaskSet

  • Reason

  • Exception (optional)

Posted when DAGScheduler is requested to taskSetFailed

Event handler: handleTaskSetFailed

WorkerRemoved

WorkerRemoved event carries the following:

  • Worked ID

  • Host name

  • Reason

Posted when DAGScheduler is requested to workerRemoved

Event handler: handleWorkerRemoved