Skip to content

The Internals of Apache Spark

Web UIs

Initializing search

spark-internals

Home
Internals
Shared Variables
Monitoring
Tools
RDD
Demos
Web UIs
Features

The Internals of Apache Spark

spark-internals

Home
Internals
Internals
- Overview
- Tags
- SparkEnv
- SparkConf
- SparkContext
- Local Properties
- Inside Creating SparkContext
- SparkFiles
- Configuration Properties
- Spark Properties
- Executor
  Executor
- External Shuffle Service
  External Shuffle Service
- MapOutputTracker
  MapOutputTracker
- Shuffle System
  Shuffle System
  - BaseShuffleHandle
  - BlockStoreShuffleReader
  - BypassMergeSortShuffleHandle
  - BypassMergeSortShuffleWriter
  - DownloadFileManager
  - FetchFailedException
  - IndexShuffleBlockResolver
  - MigratableResolver
  - SerializedShuffleHandle
  - ShuffleBlockPusher
  - ShuffleBlockResolver
  - ShuffleExternalSorter
  - ShuffleHandle
  - ShuffleInMemorySorter
  - ShuffleManager
  - ShuffleReader
  - ShuffleWriteMetricsReporter
  - ShuffleWriteProcessor
  - ShuffleWriter
  - SortShuffleManager
  - SortShuffleWriter
  - UnsafeShuffleWriter
  - Spillable Collections
    Spillable Collections
    
    Spillable
    
    ExternalAppendOnlyMap
    
    ExternalSorter
  - ShuffleDataIOs
    ShuffleDataIOs
    
    ShuffleDataIO
    
    LocalDiskShuffleDataIO
    
    ShuffleDriverComponents
    
    ShuffleExecutorComponents
    
    ShuffleMapOutputWriter
    
    LocalDiskShuffleMapOutputWriter
    
    ExecutorDiskUtils
    
    SingleSpillShuffleMapOutputWriter
    
    LocalDiskShuffleExecutorComponents
    
    LocalDiskSingleSpillMapOutputWriter
    
    ShuffleDataIOUtils
- Dynamic Resource Allocation
  Dynamic Resource Allocation
- Core
  Core
- OutputCommitCoordinator
- Scheduler
  Scheduler
- RPC
  RPC
- Memory
  Memory
- Storage
  Storage
- Serialization
  Serialization
- Barrier Execution Mode
  Barrier Execution Mode
  - barrier-execution-mode
  - RDDBarrier
- Speculative Execution of Tasks
- Deployment Architecture
  Deployment Architecture
  - Architecture
  - Driver
  - Master
  - Workers
- Internal IO
  Internal IO
- Status
  Status
- Network
  Network
- Misc
  Misc
- Spark Tips and Tricks
  Spark Tips and Tricks
- Spark Local
  Spark Local
Shared Variables
Shared Variables
- Accumulators
  Accumulators
- Broadcast Variables
  Broadcast Variables
Monitoring
Monitoring
- ConsoleProgressBar
- Debugging Spark
- DriverLogger
- ListenerBus
- Logging
- SparkListener
- SparkListenerBus
- SparkListenerEvents
  SparkListenerEvents
  - SparkListenerEvent
  - SparkListenerTaskEnd
- SparkListenerInterface
- SparkStatusTracker
- SpillListener
- StatsReportListener
- TaskCompletionListener
- TaskFailureListener
- Spark History Server
  Spark History Server
- Status REST API
  Status REST API
- Metrics
  Metrics
  - Configuration Properties
  - MetricsSystem
  - MetricsConfig
  - Source
  - Sink
  - Sources
    Sources
    
    JvmSource
  - Sinks
    Sinks
    
    MetricsServlet
    
    PrometheusServlet
Tools
Tools
- spark-submit
  spark-submit
- spark-shell
- spark-class
- SparkLauncher
- Internals
  Internals
RDD
RDD
- CoalescedRDD
- CheckpointRDD
- CoGroupedRDD
- Dependency
- HadoopRDD
- LocalCheckpointRDD
- MapPartitionsRDD
- NarrowDependency
- NewHadoopRDD
- ParallelCollectionRDD
- RDD
- ReliableCheckpointRDD
- ShuffleDependency
- ShuffledRDD
- SubtractedRDD
- Operators
  Operators
- Partitioners
  Partitioners
- RDD Lineage
- Caching and Persistence
- Partitions and Partitioning
- Partition
- RDD Checkpointing
- RDDCheckpointData
- LocalRDDCheckpointData
- ReliableRDDCheckpointData
- Aggregator
Demos
Demos
- DiskBlockManager and Block Data
- Exercises
  Exercises
Web UIs
Web UIs
- Configuration Properties
- WebUI
- WebUIPage
- WebUITab
- Spark UI
  Spark UI
  - SparkUI
  - SparkUITab
  - Environment
    Environment
    
    EnvironmentTab
    
    EnvironmentPage
  - Executors
    Executors
    
    ExecutorsTab
    
    ExecutorsPage
    
    ExecutorThreadDumpPage
  - Jobs
    Jobs
    
    JobsTab
    
    AllJobsPage
    
    JobPage
  - Stages
    Stages
    
    StagesTab
    
    AllStagesPage
    
    StagePage
    
    PoolPage
  - Storage
    Storage
    
    StorageTab
    
    StoragePage
    
    RDDPage
- PrometheusResource
- UIUtils
- JettyUtils
Features
Features
- Plugin Framework
  Plugin Framework
- Push-Based Shuffle
- Stage-Level Scheduling
  Stage-Level Scheduling

Web UIs¶

web UI is the web interface of Spark applications or infrastructure for monitoring and inspection.

The main abstraction is WebUI.

Previous Causing Stage to Fail

Next Configuration Properties

Copyright © 2021 Jacek Laskowski

Made with Material for MkDocs Insiders