The Internals of Apache Spark

ExecutionMemoryPool

Initializing search

spark-internals

Home
Internals
Spark Standalone
Monitoring
Tools
RDD
Metrics
Demos
Exercises
Web UIs

The Internals of Apache Spark

spark-internals

Home
Internals
Internals
- Overview
- SparkEnv
- SparkConf
- SparkContext
  SparkContext
- SparkFiles
- Configuration Properties
- Spark Properties
- External Shuffle Service
  External Shuffle Service
- MapOutputTracker
  MapOutputTracker
  - MapOutputTracker
  - MapOutputTrackerMaster
    MapOutputTrackerMaster
    
    MapOutputTrackerMaster
    
    ShuffleStatus
    
    MapOutputTrackerMasterEndpoint
  - MapOutputTrackerWorker
- Shuffle System
  Shuffle System
  - BaseShuffleHandle
  - BlockStoreShuffleReader
  - BypassMergeSortShuffleHandle
  - BypassMergeSortShuffleWriter
  - DownloadFileManager
  - FetchFailedException
  - IndexShuffleBlockResolver
  - MigratableResolver
  - SerializedShuffleHandle
  - ShuffleBlockResolver
  - ShuffleExternalSorter
  - ShuffleHandle
  - ShuffleInMemorySorter
  - ShuffleManager
  - ShuffleReader
  - ShuffleWriteMetricsReporter
  - ShuffleWriteProcessor
  - ShuffleWriter
  - SortShuffleManager
  - SortShuffleWriter
  - UnsafeShuffleWriter
  - Spillable Collections
    Spillable Collections
    
    Spillable
    
    ExternalAppendOnlyMap
    
    ExternalSorter
  - ShuffleDataIOs
    ShuffleDataIOs
    
    ShuffleDataIO
    
    LocalDiskShuffleDataIO
    
    ShuffleDriverComponents
    
    ShuffleExecutorComponents
    
    ShuffleMapOutputWriter
    
    LocalDiskShuffleMapOutputWriter
    
    ExecutorDiskUtils
    
    SingleSpillShuffleMapOutputWriter
    
    LocalDiskShuffleExecutorComponents
    
    LocalDiskSingleSpillMapOutputWriter
    
    ShuffleDataIOUtils
- Dynamic Resource Allocation
  Dynamic Resource Allocation
- Network
  Network
- Core
  Core
- Scheduler
  Scheduler
- RPC
  RPC
- Memory
  Memory
- Storage
  Storage
- Serialization
  Serialization
- Executor
  Executor
- Broadcast Variables
  Broadcast Variables
  - Broadcast
- Accumulators
  Accumulators
- Barrier Execution Mode
  Barrier Execution Mode
  - barrier-execution-mode
  - RDDBarrier
- Speculative Execution of Tasks
- Deployment Architecture
  Deployment Architecture
  - Architecture
  - Driver
  - Master
  - Workers
- Internal IO
  Internal IO
- Stage-Level Scheduling
  Stage-Level Scheduling
  - ResourceProfile
  - ResourceProfileBuilder
- Status
  Status
- Misc
  Misc
- Spark Tips and Tricks
  Spark Tips and Tricks
- Spark Local
  Spark Local
Spark Standalone
Spark Standalone
Monitoring
Monitoring
- Logging
- ListenerBus
- SparkListenerBus
- SparkListenerEvent
- SparkListener
- SparkListenerInterface
- SpillListener
- StatsReportListener
- AsyncEventQueue
- spark-debugging
- ConsoleProgressBar
- Spark History Server
  Spark History Server
- Status REST API
  Status REST API
- Plugin Framework
  Plugin Framework
Tools
Tools
- spark-submit
  spark-submit
- spark-shell
- spark-class
- SparkLauncher
- Internals
  Internals
RDD
RDD
- RDD
- Custom RDDs
  Custom RDDs
- Operators
  Operators
- Partitioners
  Partitioners
- RDD Lineage
- Caching and Persistence
- Partitions and Partitioning
- Partition
- RDD Checkpointing
- RDDCheckpointData
- LocalRDDCheckpointData
- ReliableRDDCheckpointData
- Shuffling
- Dependencies
- NarrowDependency
- ShuffleDependency
- Aggregator
Metrics
Metrics
- Configuration Properties
- MetricsSystem
- MetricsConfig
- Source
- Sink
- Sources
  Sources
  - JvmSource
- Sinks
  Sinks
  - MetricsServlet
Demos
Demos
- DiskBlockManager and Block Data
Exercises
Exercises
Web UIs
Web UIs
- Configuration Properties
- WebUI
- WebUIPage
- WebUITab
- Spark UI
  Spark UI
  - SparkUI
  - SparkUITab
  - Environment
    Environment
    
    EnvironmentTab
    
    EnvironmentPage
  - Executors
    Executors
    
    ExecutorsTab
    
    ExecutorsPage
    
    ExecutorThreadDumpPage
  - Jobs
    Jobs
    
    JobsTab
    
    AllJobsPage
    
    JobPage
  - Stages
    Stages
    
    StagesTab
    
    AllStagesPage
    
    StagePage
    
    PoolPage
  - Storage
    Storage
    
    StorageTab
    
    StoragePage
    
    RDDPage
- PrometheusResource
- UIUtils
- JettyUtils

ExecutionMemoryPool

= [[ExecutionMemoryPool]] ExecutionMemoryPool

ExecutionMemoryPool is a memory:MemoryPool.md[MemoryPool].

Last update: 2020-10-06

Previous MemoryPool

Next StorageMemoryPool

Copyright © 2021 Jacek Laskowski

Made with Material for MkDocs Insiders