Skip to content

The Internals of Apache Spark

RPC System

Initializing search

spark-internals

Apache Spark
Features
Internals
Shared Variables
Monitoring
Tools
RDD
Demos
Web UIs

The Internals of Apache Spark

spark-internals

Apache Spark
Features
Features
- Barrier Execution Mode
  Barrier Scheduling
  Barrier Execution Mode
- Configuration Properties
- Developer API
- Logging
- Plugin Framework
  Plugin Framework
- Push-Based Shuffle
- Stage-Level Scheduling
  Stage-Level Scheduling
Internals
Internals
- Overview
- SparkEnv
- SparkConf
- SparkContext
- SparkCoreErrors
- Local Properties
- Inside Creating SparkContext
- SparkFiles
- Spark Properties
- Executor
  Executor
- External Shuffle Service
  External Shuffle Service
- MapOutputTracker
  MapOutputTracker
- Shuffle System
  Shuffle System
  - BaseShuffleHandle
  - BlockStoreShuffleReader
  - BypassMergeSortShuffleHandle
  - BypassMergeSortShuffleWriter
  - DownloadFileManager
  - FetchFailedException
  - IndexShuffleBlockResolver
  - MigratableResolver
  - SerializedShuffleHandle
  - ShuffleBlockPusher
  - ShuffleBlockResolver
  - ShuffleExternalSorter
  - ShuffleHandle
  - ShuffleInMemorySorter
  - ShuffleManager
  - ShuffleReader
  - ShuffleWriteMetricsReporter
  - ShuffleWriteProcessor
  - ShuffleWriter
  - SortShuffleManager
  - SortShuffleWriter
  - UnsafeShuffleWriter
  - Spillable Collections
    Spillable Collections
    
    Spillable
    
    ExternalAppendOnlyMap
    
    ExternalSorter
  - ShuffleDataIOs
    ShuffleDataIOs
    
    ShuffleDataIO
    
    LocalDiskShuffleDataIO
    
    ShuffleDriverComponents
    
    ShuffleExecutorComponents
    
    ShuffleMapOutputWriter
    
    LocalDiskShuffleMapOutputWriter
    
    ExecutorDiskUtils
    
    SingleSpillShuffleMapOutputWriter
    
    LocalDiskShuffleExecutorComponents
    
    LocalDiskSingleSpillMapOutputWriter
    
    ShuffleDataIOUtils
- Dynamic Resource Allocation
  Dynamic Resource Allocation
- Core
  Core
- OutputCommitCoordinator
- Scheduler
  Scheduler
  - ActiveJob
  - BlacklistTracker
  - CoarseGrainedSchedulerBackend
  - DAGScheduler
  - DAGSchedulerEvent
  - DAGSchedulerEventProcessLoop
  - DAGSchedulerSource
  - DriverEndpoint
  - ExecutorData
  - ExternalClusterManager
  - FairSchedulableBuilder
  - FIFOSchedulableBuilder
  - JobListener
  - JobWaiter
  - LiveListenerBus
  - MapStatuses
    MapStatuses
    
    MapStatus
    
    CompressedMapStatus
    
    HighlyCompressedMapStatus
  - Pool
  - ResultStage
  - ResultTask
  - Schedulable
  - SchedulableBuilder
  - SchedulerBackend
  - SchedulerBackendUtils
  - SchedulingMode
  - ShuffleMapStage
  - ShuffleMapTask
  - Stage
  - StageInfo
  - TaskScheduler
  - TaskSchedulerImpl
  - Task
  - TaskContext
  - TaskContextImpl
  - TaskDescription
  - TaskInfo
  - TaskLocation
  - TaskResult
  - TaskResultGetter
  - TaskSet
  - TaskSetBlacklist
  - TaskSetManager
- RPC
  RPC
- Memory
  Memory
- Storage
  Storage
- Serialization
  Serialization
- Speculative Execution of Tasks
- Deployment Architecture
  Deployment Architecture
  - Architecture
  - Driver
  - Master
  - Workers
- Internal IO
  Internal IO
- Status
  Status
- Network
  Network
- Misc
  Misc
- Spark Tips and Tricks
  Spark Tips and Tricks
- Spark Local
  Spark Local
Shared Variables
Shared Variables
- Accumulators
  Accumulators
- Broadcast Variables
  Broadcast Variables
Monitoring
Monitoring
- ConsoleProgressBar
- Debugging Spark
- DriverLogger
- ListenerBus
- SparkListener
- SparkListenerBus
- SparkListenerEvents
  SparkListenerEvents
  - SparkListenerEvent
  - SparkListenerTaskEnd
- SparkListenerInterface
- SparkStatusTracker
- SpillListener
- StatsReportListener
- TaskCompletionListener
- TaskFailureListener
- Spark History Server
  Spark History Server
- Status REST API
  Status REST API
- Metrics
  Metrics
  - Configuration Properties
  - MetricsSystem
  - MetricsConfig
  - Source
  - Sink
  - Sources
    Sources
    
    JvmSource
  - Sinks
    Sinks
    
    MetricsServlet
    
    PrometheusServlet
Tools
Tools
RDD
RDD
- CoalescedRDD
- CheckpointRDD
- CoGroupedRDD
- Dependency
- HadoopRDD
- LocalCheckpointRDD
- MapPartitionsRDD
- NarrowDependency
- NewHadoopRDD
- ParallelCollectionRDD
- RDD
- ReliableCheckpointRDD
- ShuffleDependency
- ShuffledRDD
- SubtractedRDD
- Operators
  Operators
- Partitioners
  Partitioners
- RDD Lineage
- Caching and Persistence
- Partitions and Partitioning
- Partition
- RDD Checkpointing
- RDDCheckpointData
- LocalRDDCheckpointData
- ReliableRDDCheckpointData
- Aggregator
Demos
Demos
- DiskBlockManager and Block Data
- Exercises
  Exercises
Web UIs
Web UIs
- Configuration Properties
- WebUI
- WebUIPage
- WebUITab
- Spark UI
  Spark UI
  - SparkUI
  - SparkUITab
  - Environment
    Environment
    
    EnvironmentTab
    
    EnvironmentPage
  - Executors
    Executors
    
    ExecutorsTab
    
    ExecutorsPage
    
    ExecutorThreadDumpPage
  - Jobs
    Jobs
    
    JobsTab
    
    AllJobsPage
    
    JobPage
  - Stages
    Stages
    
    StagesTab
    
    AllStagesPage
    
    StagePage
    
    PoolPage
  - Storage
    Storage
    
    StorageTab
    
    StoragePage
    
    RDDPage
- PrometheusResource
- UIUtils
- JettyUtils

Apache Spark
Internals
RPC

RPC System¶

RPC System is a communication system of Spark services.

The main abstractions are RpcEnv and RpcEndpoint.

RpcEnvironment with RpcEndpoints and RpcEndpointRefs

Copyright © 2023 Jacek Laskowski

Made with Material for MkDocs Insiders