Skip to content

The Internals of Spark Core

Developer API

Initializing search

spark-internals

Spark Core
Features
Internals
Shared Variables
Monitoring
Tools
RDD
Demos
Web UIs

The Internals of Spark Core

spark-internals

Spark Core
Features
Features
- Barrier Execution Mode
  Barrier Scheduling
  Barrier Execution Mode
- Configuration Properties
- Developer API Developer API
  Table of contents
  - DeveloperApi
- Logging
- Plugin Framework
  Plugin Framework
- Push-Based Shuffle
- Stage-Level Scheduling
  Stage-Level Scheduling
Internals
Internals
- Overview
- SparkEnv
- SparkConf
- SparkContext
- SparkCoreErrors
- Local Properties
- Inside Creating SparkContext
- SparkFiles
- Spark Properties
- Executor
  Executor
- External Shuffle Service
  External Shuffle Service
- MapOutputTracker
  MapOutputTracker
- Shuffle System
  Shuffle System
  - BaseShuffleHandle
  - BlockStoreShuffleReader
  - BypassMergeSortShuffleHandle
  - BypassMergeSortShuffleWriter
  - DownloadFileManager
  - FetchFailedException
  - IndexShuffleBlockResolver
  - MigratableResolver
  - SerializedShuffleHandle
  - ShuffleBlockPusher
  - ShuffleBlockResolver
  - ShuffleExternalSorter
  - ShuffleHandle
  - ShuffleInMemorySorter
  - ShuffleManager
  - ShuffleReader
  - ShuffleWriteMetricsReporter
  - ShuffleWriteProcessor
  - ShuffleWriter
  - SortShuffleManager
  - SortShuffleWriter
  - UnsafeShuffleWriter
  - Spillable Collections
    Spillable Collections
    
    Spillable
    
    ExternalAppendOnlyMap
    
    ExternalSorter
  - ShuffleDataIOs
    ShuffleDataIOs
    
    ShuffleDataIO
    
    LocalDiskShuffleDataIO
    
    ShuffleDriverComponents
    
    ShuffleExecutorComponents
    
    ShuffleMapOutputWriter
    
    LocalDiskShuffleMapOutputWriter
    
    ExecutorDiskUtils
    
    SingleSpillShuffleMapOutputWriter
    
    LocalDiskShuffleExecutorComponents
    
    LocalDiskSingleSpillMapOutputWriter
    
    ShuffleDataIOUtils
- Dynamic Resource Allocation
  Dynamic Resource Allocation
- Core
  Core
- OutputCommitCoordinator
- Scheduler
  Scheduler
  - ActiveJob
  - BlacklistTracker
  - CoarseGrainedSchedulerBackend
  - DAGScheduler
  - DAGSchedulerEvent
  - DAGSchedulerEventProcessLoop
  - DAGSchedulerSource
  - DriverEndpoint
  - ExecutorData
  - ExternalClusterManager
  - FairSchedulableBuilder
  - FIFOSchedulableBuilder
  - JobListener
  - JobWaiter
  - LiveListenerBus
  - MapStatuses
    MapStatuses
    
    MapStatus
    
    CompressedMapStatus
    
    HighlyCompressedMapStatus
  - Pool
  - ResultStage
  - ResultTask
  - Schedulable
  - SchedulableBuilder
  - SchedulerBackend
  - SchedulerBackendUtils
  - SchedulingMode
  - ShuffleMapStage
  - ShuffleMapTask
  - Stage
  - StageInfo
  - TaskScheduler
  - TaskSchedulerImpl
  - Task
  - TaskContext
  - TaskContextImpl
  - TaskDescription
  - TaskInfo
  - TaskLocation
  - TaskResult
  - TaskResultGetter
  - TaskSet
  - TaskSetBlacklist
  - TaskSetManager
- RPC
  RPC
- Memory
  Memory
- Storage
  Storage
- Serialization
  Serialization
- Speculative Execution of Tasks
- Deployment Architecture
  Deployment Architecture
  - Architecture
  - Driver
  - Master
  - Workers
- Writers and Committers
  Writers and Committers
- Status
  Status
- Network
  Network
- Misc
  Misc
- Spark's Tips and Tricks
  Spark's Tips and Tricks
- Spark Local
  Spark Local
Shared Variables
Shared Variables
- Accumulators
  Accumulators
- Broadcast Variables
  Broadcast Variables
Monitoring
Monitoring
- ConsoleProgressBar
- Debugging Spark
- DriverLogger
- ListenerBus
- SparkListener
- SparkListenerBus
- SparkListenerEvents
  SparkListenerEvents
  - SparkListenerEvent
  - SparkListenerTaskEnd
- SparkListenerInterface
- SparkStatusTracker
- SpillListener
- StatsReportListener
- TaskCompletionListener
- TaskFailureListener
- Spark History Server
  Spark History Server
- Status REST API
  Status REST API
- Metrics
  Metrics
  - Configuration Properties
  - MetricsSystem
  - MetricsConfig
  - Source
  - Sink
  - Sources
    Sources
    
    JvmSource
  - Sinks
    Sinks
    
    MetricsServlet
    
    PrometheusServlet
Tools
Tools
RDD
RDD
- CoalescedRDD
- CheckpointRDD
- CoGroupedRDD
- Dependency
- HadoopRDD
- LocalCheckpointRDD
- MapPartitionsRDD
- NarrowDependency
- NewHadoopRDD
- ParallelCollectionRDD
- RDD
  Resilient Distributed Dataset
- RDDOperationScope
- ReliableCheckpointRDD
- ShuffleDependency
- ShuffledRDD
- Operators
  Operators
- Partitioners
  Partitioners
- RDD Lineage
- Caching and Persistence
- Partitions and Partitioning
- Partition
- RDD Checkpointing
- RDDCheckpointData
- LocalRDDCheckpointData
- MapPartitionsWithEvaluatorRDD
- ReliableRDDCheckpointData
- ZippedPartitionsWithEvaluatorRDD
- Aggregator
Demos
Demos
- DiskBlockManager and Block Data
- Exercises
  Exercises
Web UIs
Web UIs
- Configuration Properties
- WebUI
- WebUIPage
- WebUITab
- Spark UI
  Spark UI
  - SparkUI
  - SparkUITab
  - Environment
    Environment
    
    EnvironmentTab
    
    EnvironmentPage
  - Executors
    Executors
    
    ExecutorsTab
    
    ExecutorsPage
    
    ExecutorThreadDumpPage
  - Jobs
    Jobs
    
    JobsTab
    
    AllJobsPage
    
    JobPage
  - Stages
    Stages
    
    StagesTab
    
    AllStagesPage
    
    StagePage
    
    PoolPage
  - Storage
    Storage
    
    StorageTab
    
    StoragePage
    
    RDDPage
- PrometheusResource
- UIUtils
- JettyUtils

Table of contents

DeveloperApi

Spark Core
Features

Developer API¶

DeveloperApi¶

Dependency
ExecutorMetrics
NarrowDependency
PartitionEvaluator
PartitionEvaluatorFactory
ResourceID
ShuffleDependency
ShuffleReadMetrics
ShuffleWriteMetrics
ShuffledRDD
SparkEnv
SparkListener
SparkListenerResourceProfileAdded
SparkPlugin
StatsReportListener
StorageLevel
TaskCompletionListener
TaskFailureListener
TaskMetrics

Copyright © 2023-2024 Jacek Laskowski

Made with Material for MkDocs Insiders