Skip to content

The Internals of Spark SQL

FlowPlanner

Initializing search

spark-sql-internals

Spark SQL
Features
Query Execution
Internals
SQL
Connectors
High-Level APIs
Web UI
Demo
Misc

The Internals of Spark SQL

spark-sql-internals

Spark SQL
Features
Features
- Aggregate Queries
  Aggregate Queries
- Adaptive Query Execution
  Adaptive Query Execution
- Bloom Filter Join
  Bloom Filter Join
  - BloomFilter
  - BloomFilterImpl
- Bucketing
  Bucketing
  - BucketSpec
- Cache Serialization
  Cache Serialization
- Catalog Plugin API
  Catalog Plugin API
- Columnar Execution
  Columnar Execution
- Common Table Expressions
  Common Table Expressions
- Configuration Properties
- Connector Expressions
  Connector Expressions
  - Aggregation
  - SortOrder
- Cost-Based Optimization
  Cost-Based Optimization
- Default Columns
  Default Columns
- Declarative Pipelines
  ⚠️ 4.1.0-SNAPSHOT
  Declarative Pipelines
  - BatchTableWrite
  - DataflowGraph
  - DataflowGraphRegistry
  - DatasetManager
  - Flow
  - FlowExecution
  - FlowPlanner FlowPlanner
    Table of contents
    
    Plan
  - GraphExecution
  - GraphRegistrationContext
  - PipelineExecution
  - PipelineUpdateContext
  - PipelineUpdateContextImpl
  - PipelinesHandler
  - PipelinesTableProperties
  - SqlGraphRegistrationContext
  - StreamListener
  - StreamingTableWrite
  - Table
  - TriggeredGraphExecution
  - UnresolvedFlow
  - View
- Direct Queries on Files
  Direct Queries on Files
- Dynamic Partition Pruning
  Dynamic Partition Pruning
- File-Based Data Scanning
  File-Based Data Scanning
- Generated Columns
  Generated Columns
  - GeneratedColumn
- Hidden File Metadata
  Hidden File Metadata
  - MetadataAttribute
- Hints (SQL)
  Hints (SQL)
- Join Queries
  Join Queries
  - Joins
  - Broadcast Joins
- Logging
- Metadata Columns
  Metadata Columns
- Named Function Arguments
- Parameterized Queries
  Parameterized Queries
- Partition File Metadata Caching
  Partition File Metadata Caching
- Runtime Filtering
  Runtime Filtering
- Spark Connect
- Spark Thrift Server
  Spark Thrift Server
  - SparkSQLEnv
- Statistics
- Storage-Partitioned Joins
  Storage-Partitioned Joins
- Subexpression Elimination
  Subexpression Elimination
  - EquivalentExpressions
- Subqueries
  Subqueries
- Table-Valued Functions
  Table-Valued Functions
- Time Travel
  Time Travel
  - TimeTravelSpec
- Transactional Writes
  Transactional Writes
  - SQLHadoopMapReduceCommitProtocol
- User-Defined Functions
  User-Defined Functions
  - UDFRegistration
  - UserDefinedPythonFunction
- Vectorized Decoding
  Vectorized Decoding
- ANSI Intervals
- Catalog Plugin API and Multi-Catalog Support
- Explaining Query Plans Improved
- Observable Metrics
- Hive Integration
- Dynamic Partition Inserts
- Vectorized Query Execution
  Vectorized Query Execution
  - ColumnarBatch
- Whole-Stage Code Generation
  Whole-Stage Code Generation
- Catalyst DSL
  Catalyst DSL
  - DslLogicalPlan
- Variable Substitution
Query Execution
Query Execution
Internals
Internals
- Overview
- DataSource
  Pluggable Data Provider Framework
- Developer API
- ExecutionListenerBus
- ExecutionListenerManager
- SharedState
  State Shared Across SparkSessions
- SQLConf
- SQLConfHelper
- StaticSQLConf
  Static Configuration Properties
- SparkSession Registries
  SparkSession Registries
  - Catalog
    Catalog
    
    Catalog
    
    CatalogImpl
    
    CatalogStatistics
    
    CatalogUtils
  - ExperimentalMethods
  - ExternalCatalog
    ExternalCatalog
    
    ExternalCatalog
    
    InMemoryCatalog
    
    ExternalCatalogWithListener
  - FunctionRegistry
    FunctionRegistry
    
    FunctionRegistry
    
    FunctionRegistryBase
    
    SimpleFunctionRegistry
    
    SimpleFunctionRegistryBase
    
    SimpleTableFunctionRegistry
    
    TableFunctionRegistry
  - GlobalTempViewManager
  - SessionCatalog
    SessionCatalog
    
    SessionCatalog
    
    CatalogStorageFormat
    
    CatalogTable
    
    CatalogTablePartition
  - V2SessionCatalog
  - SessionState
    SessionState
    
    SessionState
    
    BaseSessionStateBuilder
    
    SessionStateBuilder
  - CacheManager
    CacheManager
    
    CacheManager
  - RuntimeConfig
- Encoder
  Encoder
- SQLExecution
- SQLMetric
- Tungsten Execution Backend
  Tungsten Execution Backend
- RDDs
  RDDs
SQL
SQL
Connectors
Connectors
High-Level APIs
High-Level APIs
- Column
- ColumnarRule
- Connector API
  Connector API
- Data Types
  Data Types
- DataFrame
- DataFrameNaFunctions
  Working With Missing Data
- DataFrameReader
- DataFrameStatFunctions
- DataFrameWriter
- DataFrameWriterV2
- Dataset
- Dataset API
  Dataset API
- DataSource V1 API
  DataSource V1 API
- Encoders
- KeyValueGroupedDataset
- Observation
- QueryExecutionListener
- RelationalGroupedDataset
- SparkSession
  The Entry Point to Spark SQL
- SparkSession.Builder
- SparkSessionExtensions
- Standard Functions
  org.apache.spark.sql.functions
  Standard Functions
- TypedColumn
- Window Functions
  Window Functions
Web UI
Web UI
Demo
Demo
Misc
Misc
- AggregatingAccumulator
- DistinctKeyVisitor
- FilterEvaluatorFactory
- JoinSelectionHelper
- PushDownUtils
- UnsafeExternalRowSorter
- BindReferences
- IntervalUtils
- ExplainUtils
- SerializerBuildHelper
- Dataset, DataFrame and RDD
- Dataset and SQL
- DDLUtils
- Implicits
- Row
- Data Source API
  Data Source API
  - CreateTableWriter
  - WriteConfigMethods
- Column Operators
- Caching and Persistence
  Caching and Persistence
  - Caching and Persistence
  - User-Friendly Names of Cached Queries in web UI
- Checkpointing
- Performance Tuning and Debugging
  Performance Tuning and Debugging
- CheckAnalysis
- CatalystTypeConverters
- SubExprUtils
- PredicateHelper
- ExtractEquiJoinKeys
- ExtractSingleColumnNullAwareAntiJoin
- ExtractJoinWithBuckets
- PhysicalOperation
- KnownSizeEstimation

Table of contents

Plan

Spark SQL
Features
Declarative Pipelines

FlowPlanner¶

Plan¶

plan(
  flow: ResolvedFlow): FlowExecution

plan...FIXME

plan is used when:

GraphExecution is requested to planAndStartFlow

Copyright © 2025 Jacek Laskowski

Made with Material for MkDocs Insiders