Skip to content

The Internals of Apache Spark

RPC System

Initializing search

spark-internals

Home
Internals
Shared Variables
Monitoring
Tools
RDD
Demos
Web UIs
Features

The Internals of Apache Spark

spark-internals

Home
Internals
Internals
- Overview
- Tags
- SparkEnv
- SparkConf
- SparkContext
- Local Properties
- Inside Creating SparkContext
- SparkFiles
- Configuration Properties
- Spark Properties
- Executor
  Executor
- External Shuffle Service
  External Shuffle Service
- MapOutputTracker
  MapOutputTracker
- Shuffle System
  Shuffle System
  - BaseShuffleHandle
  - BlockStoreShuffleReader
  - BypassMergeSortShuffleHandle
  - BypassMergeSortShuffleWriter
  - DownloadFileManager
  - FetchFailedException
  - IndexShuffleBlockResolver
  - MigratableResolver
  - SerializedShuffleHandle
  - ShuffleBlockPusher
  - ShuffleBlockResolver
  - ShuffleExternalSorter
  - ShuffleHandle
  - ShuffleInMemorySorter
  - ShuffleManager
  - ShuffleReader
  - ShuffleWriteMetricsReporter
  - ShuffleWriteProcessor
  - ShuffleWriter
  - SortShuffleManager
  - SortShuffleWriter
  - UnsafeShuffleWriter
  - Spillable Collections
    Spillable Collections
    
    Spillable
    
    ExternalAppendOnlyMap
    
    ExternalSorter
  - ShuffleDataIOs
    ShuffleDataIOs
    
    ShuffleDataIO
    
    LocalDiskShuffleDataIO
    
    ShuffleDriverComponents
    
    ShuffleExecutorComponents
    
    ShuffleMapOutputWriter
    
    LocalDiskShuffleMapOutputWriter
    
    ExecutorDiskUtils
    
    SingleSpillShuffleMapOutputWriter
    
    LocalDiskShuffleExecutorComponents
    
    LocalDiskSingleSpillMapOutputWriter
    
    ShuffleDataIOUtils
- Dynamic Resource Allocation
  Dynamic Resource Allocation
- Core
  Core
- OutputCommitCoordinator
- Scheduler
  Scheduler
- RPC
  RPC
- Memory
  Memory
- Storage
  Storage
- Serialization
  Serialization
- Barrier Execution Mode
  Barrier Execution Mode
  - barrier-execution-mode
  - RDDBarrier
- Speculative Execution of Tasks
- Deployment Architecture
  Deployment Architecture
  - Architecture
  - Driver
  - Master
  - Workers
- Internal IO
  Internal IO
- Status
  Status
- Network
  Network
- Misc
  Misc
- Spark Tips and Tricks
  Spark Tips and Tricks
- Spark Local
  Spark Local
Shared Variables
Shared Variables
- Accumulators
  Accumulators
- Broadcast Variables
  Broadcast Variables
Monitoring
Monitoring
- ConsoleProgressBar
- Debugging Spark
- DriverLogger
- ListenerBus
- Logging
- SparkListener
- SparkListenerBus
- SparkListenerEvents
  SparkListenerEvents
  - SparkListenerEvent
  - SparkListenerTaskEnd
- SparkListenerInterface
- SparkStatusTracker
- SpillListener
- StatsReportListener
- TaskCompletionListener
- TaskFailureListener
- Spark History Server
  Spark History Server
- Status REST API
  Status REST API
- Metrics
  Metrics
  - Configuration Properties
  - MetricsSystem
  - MetricsConfig
  - Source
  - Sink
  - Sources
    Sources
    
    JvmSource
  - Sinks
    Sinks
    
    MetricsServlet
    
    PrometheusServlet
Tools
Tools
- spark-submit
  spark-submit
- spark-shell
- spark-class
- SparkLauncher
- Internals
  Internals
RDD
RDD
- CoalescedRDD
- CheckpointRDD
- CoGroupedRDD
- Dependency
- HadoopRDD
- LocalCheckpointRDD
- MapPartitionsRDD
- NarrowDependency
- NewHadoopRDD
- ParallelCollectionRDD
- RDD
- ReliableCheckpointRDD
- ShuffleDependency
- ShuffledRDD
- SubtractedRDD
- Operators
  Operators
- Partitioners
  Partitioners
- RDD Lineage
- Caching and Persistence
- Partitions and Partitioning
- Partition
- RDD Checkpointing
- RDDCheckpointData
- LocalRDDCheckpointData
- ReliableRDDCheckpointData
- Aggregator
Demos
Demos
- DiskBlockManager and Block Data
- Exercises
  Exercises
Web UIs
Web UIs
- Configuration Properties
- WebUI
- WebUIPage
- WebUITab
- Spark UI
  Spark UI
  - SparkUI
  - SparkUITab
  - Environment
    Environment
    
    EnvironmentTab
    
    EnvironmentPage
  - Executors
    Executors
    
    ExecutorsTab
    
    ExecutorsPage
    
    ExecutorThreadDumpPage
  - Jobs
    Jobs
    
    JobsTab
    
    AllJobsPage
    
    JobPage
  - Stages
    Stages
    
    StagesTab
    
    AllStagesPage
    
    StagePage
    
    PoolPage
  - Storage
    Storage
    
    StorageTab
    
    StoragePage
    
    RDDPage
- PrometheusResource
- UIUtils
- JettyUtils
Features
Features
- Plugin Framework
  Plugin Framework
- Push-Based Shuffle
- Stage-Level Scheduling
  Stage-Level Scheduling

RPC System¶

RPC System is a communication system of Spark services.

The main abstractions are RpcEnv and RpcEndpoint.

RpcEnvironment with RpcEndpoints and RpcEndpointRefs

Previous TaskSetManager

Copyright © 2022 Jacek Laskowski

Made with Material for MkDocs Insiders