Skip to content

The Internals of Spark Core

BlockReplicationPolicy

Initializing search

GitHub

Home
Features
Internals
Shared Variables
Monitoring
Tools
RDD
Demos
Web UIs

The Internals of Spark Core

GitHub

Home
Features
Features
- Barrier Execution Mode
  Barrier Scheduling
  Barrier Execution Mode
- Configuration Properties
- Developer API
- Logging
- Plugin Framework
  Plugin Framework
- Push-Based Shuffle
- Shuffle System
  Shuffle System
  - Shuffle Handles
    Shuffle Handles
    
    BaseShuffleHandle
    
    BypassMergeSortShuffleHandle
    
    SerializedShuffleHandle
    
    ShuffleHandle
  - Shuffle Readers
    Shuffle Readers
    
    BlockStoreShuffleReader
    
    ShuffleReader
  - BypassMergeSortShuffleWriter
  - DownloadFileManager
  - FetchFailedException
  - IndexShuffleBlockResolver
  - MigratableResolver
  - ShuffleBlockPusher
  - ShuffleBlockResolver
  - ShuffleExternalSorter
  - ShuffleInMemorySorter
  - ShuffleManager
  - ShuffleReadMetricsReporter
  - ShuffleWriteMetricsReporter
  - ShuffleWriteProcessor
  - ShuffleWriter
  - SortShuffleManager
  - SortShuffleWriter
  - UnsafeShuffleWriter
  - Spillable Collections
    Spillable Collections
    
    Spillable
    
    ExternalAppendOnlyMap
    
    ExternalSorter
  - ShuffleDataIOs
    ShuffleDataIOs
    
    ShuffleDataIO
    
    LocalDiskShuffleDataIO
    
    ShuffleDriverComponents
    
    ShuffleExecutorComponents
    
    ShuffleMapOutputWriter
    
    LocalDiskShuffleMapOutputWriter
    
    ExecutorDiskUtils
    
    SingleSpillShuffleMapOutputWriter
    
    LocalDiskShuffleExecutorComponents
    
    LocalDiskSingleSpillMapOutputWriter
    
    ShuffleDataIOUtils
- Stage-Level Scheduling
  Stage-Level Scheduling
Internals
Internals
- Overview
- SparkEnv
- SparkConf
- SparkContext
- SparkCoreErrors
- Local Properties
- Inside Creating SparkContext
- SparkFiles
- Spark Properties
- Executor
  Executor
- External Shuffle Service
  External Shuffle Service
- MapOutputTracker
  MapOutputTracker
- Dynamic Resource Allocation
  Dynamic Resource Allocation
- Core
  Core
- OutputCommitCoordinator
- Scheduler
  Scheduler
  - ActiveJob
  - BlacklistTracker
  - CoarseGrainedSchedulerBackend
  - DAGScheduler
  - DAGSchedulerEvent
  - DAGSchedulerEventProcessLoop
  - DAGSchedulerSource
  - DriverEndpoint
  - ExecutorData
  - ExternalClusterManager
  - FairSchedulableBuilder
  - FIFOSchedulableBuilder
  - JobListener
  - JobWaiter
  - LiveListenerBus
  - MapStatuses
    MapStatuses
    
    MapStatus
    
    CompressedMapStatus
    
    HighlyCompressedMapStatus
  - Pool
  - ResultStage
  - ResultTask
  - Schedulable
  - SchedulableBuilder
  - SchedulerBackend
  - SchedulerBackendUtils
  - SchedulingMode
  - ShuffleMapStage
  - ShuffleMapTask
  - Stage
  - StageInfo
  - TaskScheduler
  - TaskSchedulerImpl
  - Task
  - TaskContext
  - TaskContextImpl
  - TaskDescription
  - TaskInfo
  - TaskLocation
  - TaskResult
  - TaskResultGetter
  - TaskSet
  - TaskSetBlacklist
  - TaskSetManager
- RPC
  RPC
- Memory
  Memory
- Storage
  Storage
- Serialization
  Serialization
- Speculative Execution of Tasks
- Deployment Architecture
  Deployment Architecture
  - Architecture
  - Driver
  - Master
  - Workers
- Writers and Committers
  Writers and Committers
- Status
  Status
- Network
  Network
- Misc
  Misc
- Tips and Tricks
  Tips and Tricks
- Spark Local
  Spark Local
Shared Variables
Shared Variables
- Accumulators
  Accumulators
- Broadcast Variables
  Broadcast Variables
Monitoring
Monitoring
- ConsoleProgressBar
- Debugging Spark
- DriverLogger
- ListenerBus
- SparkListener
- SparkListenerBus
- SparkListenerEvents
  SparkListenerEvents
  - SparkListenerEvent
  - SparkListenerTaskEnd
- SparkListenerInterface
- SparkStatusTracker
- SpillListener
- StatsReportListener
- TaskCompletionListener
- TaskFailureListener
- Spark History Server
  Spark History Server
- Status REST API
  Status REST API
- Metrics
  Metrics
  - Configuration Properties
  - MetricsSystem
  - MetricsConfig
  - Source
  - Sink
  - Sources
    Sources
    
    JvmSource
  - Sinks
    Sinks
    
    MetricsServlet
    
    PrometheusServlet
Tools
Tools
RDD
RDD
- CoalescedRDD
- CheckpointRDD
- CoGroupedRDD
- Dependency
- HadoopRDD
- LocalCheckpointRDD
- MapPartitionsRDD
- NarrowDependency
- NewHadoopRDD
- ParallelCollectionRDD
- RDD
  Resilient Distributed Dataset
- RDDOperationScope
- ReliableCheckpointRDD
- ShuffleDependency
- ShuffledRDD
- Operators
  Operators
- Partitioners
  Partitioners
- RDD Lineage
- Caching and Persistence
- Partitions and Partitioning
- Partition
- RDD Checkpointing
- RDDCheckpointData
- LocalRDDCheckpointData
- MapPartitionsWithEvaluatorRDD
- ReliableRDDCheckpointData
- ZippedPartitionsWithEvaluatorRDD
- Aggregator
Demos
Demos
- DiskBlockManager and Block Data
- Exercises
  Exercises
Web UIs
Web UIs
- Configuration Properties
- WebUI
- WebUIPage
- WebUITab
- Spark UI
  Spark UI
  - SparkUI
  - SparkUITab
  - Environment
    Environment
    
    EnvironmentTab
    
    EnvironmentPage
  - Executors
    Executors
    
    ExecutorsTab
    
    ExecutorsPage
    
    ExecutorThreadDumpPage
  - Jobs
    Jobs
    
    JobsTab
    
    AllJobsPage
    
    JobPage
  - Stages
    Stages
    
    StagesTab
    
    AllStagesPage
    
    StagePage
    
    PoolPage
  - Storage
    Storage
    
    StorageTab
    
    StoragePage
    
    RDDPage
- PrometheusResource
- UIUtils
- JettyUtils

Home
Internals
Storage

BlockReplicationPolicy¶

BlockReplicationPolicy is...FIXME

Made with Material for MkDocs