Skip to content

DAGSchedulerEvent

DAGSchedulerEvent is an abstraction of events that are handled by the DAGScheduler (on dag-scheduler-event-loop daemon thread).

AllJobsCancelled

Carries no extra information

Posted when DAGScheduler is requested to cancelAllJobs

Event handler: doCancelAllJobs

BeginEvent

Carries the following:

Posted when DAGScheduler is requested to taskStarted

Event handler: handleBeginEvent

CompletionEvent

Carries the following:

Posted when DAGScheduler is requested to taskEnded

Event handler: handleTaskCompletion

ExecutorAdded

Carries the following:

  • Executor ID
  • Host name

Posted when DAGScheduler is requested to executorAdded

Event handler: handleExecutorAdded

ExecutorLost

Carries the following:

  • Executor ID
  • Reason

Posted when DAGScheduler is requested to executorLost

Event handler: handleExecutorLost

GettingResultEvent

Carries the following:

Posted when DAGScheduler is requested to taskGettingResult

Event handler: handleGetTaskResult

JobCancelled

JobCancelled event carries the following:

  • Job ID
  • Reason (optional)

Posted when DAGScheduler is requested to cancelJob

Event handler: handleJobCancellation

JobGroupCancelled

Carries the following:

  • Group ID

Posted when DAGScheduler is requested to cancelJobGroup

Event handler: handleJobGroupCancelled

JobSubmitted

Carries the following:

  • Job ID
  • RDD
  • Partition function ((TaskContext, Iterator[_]) => _)
  • Partitions to compute
  • CallSite
  • JobListener to keep updated about the status of the stage execution
  • Execution properties

Posted when DAGScheduler is requested to submit a job, run an approximate job and handleJobSubmitted

Event handler: handleJobSubmitted

MapStageSubmitted

Carries the following:

Posted when DAGScheduler is requested to submitMapStage

Event handler: handleMapStageSubmitted

ResubmitFailedStages

Carries no extra information.

Posted when DAGScheduler is requested to handleTaskCompletion

Event handler: resubmitFailedStages

SpeculativeTaskSubmitted

Carries the following:

Posted when DAGScheduler is requested to speculativeTaskSubmitted

Event handler: handleSpeculativeTaskSubmitted

StageCancelled

Carries the following:

  • Stage ID
  • Reason (optional)

Posted when DAGScheduler is requested to cancelStage

Event handler: handleStageCancellation

TaskSetFailed

Carries the following:

  • TaskSet
  • Reason
  • Exception (optional)

Posted when DAGScheduler is requested to taskSetFailed

Event handler: handleTaskSetFailed

WorkerRemoved

Carries the following:

  • Worked ID
  • Host name
  • Reason

Posted when DAGScheduler is requested to workerRemoved

Event handler: handleWorkerRemoved


Last update: 2020-10-12