Skip to content

The Internals of Spark SQL

Developer API

Initializing search

spark-sql-internals

Spark SQL
Features
Query Execution
Internals
SQL
Connectors
High-Level APIs
Web UI
Demo
Misc

The Internals of Spark SQL

spark-sql-internals

Spark SQL
Features
Features
- Aggregate Queries
  Aggregate Queries
- Adaptive Query Execution
  Adaptive Query Execution
- Bloom Filter Join
  Bloom Filter Join
  - BloomFilter
  - BloomFilterImpl
- Bucketing
  Bucketing
  - BucketSpec
- Cache Serialization
  Cache Serialization
- Catalog Plugin API
  Catalog Plugin API
- Columnar Execution
  Columnar Execution
- Common Table Expressions
  Common Table Expressions
- Configuration Properties
- Connector API
  Connector API
- Connector Expressions
  Connector Expressions
  - Aggregation
  - SortOrder
- Cost-Based Optimization
  Cost-Based Optimization
- Default Columns
  Default Columns
- Dynamic Partition Pruning
  Dynamic Partition Pruning
- File-Based Data Scanning
  File-Based Data Scanning
- Generated Columns
  Generated Columns
  - GeneratedColumn
- Hints (SQL)
  Hints (SQL)
- Join Queries
  Join Queries
  - Joins
  - Broadcast Joins
- Logging
- Metadata Columns
  Metadata Columns
- Partition File Metadata Caching
  Partition File Metadata Caching
- Spark Connect
  Spark Connect
- Runtime Filtering
  Runtime Filtering
- Spark Thrift Server
  Spark Thrift Server
  - SparkSQLEnv
- Statistics
- Subqueries
  Subqueries
- Table-Valued Functions
  Table-Valued Functions
- Transactional Writes
  Transactional Writes
  - SQLHadoopMapReduceCommitProtocol
- User-Defined Functions
  User-Defined Functions
  - UDFRegistration
  - UserDefinedPythonFunction
- Vectorized Decoding
  Vectorized Decoding
- ANSI Intervals
- Catalog Plugin API and Multi-Catalog Support
- Explaining Query Plans Improved
- Observable Metrics
- Hive Integration
- Dynamic Partition Inserts
- Vectorized Query Execution
  Vectorized Query Execution
  - ColumnarBatch
- Whole-Stage Code Generation
  Whole-Stage Code Generation
  - Block
  - BufferedRowIterator
  - CodegenContext
  - CodeGenerator
  - GenerateUnsafeProjection
  - GenerateMutableProjection
  - GenerateColumnAccessor
  - GenerateOrdering
  - GeneratePredicate
  - GenerateSafeProjection
  - Subexpression Elimination
    Subexpression Elimination
    
    Subexpression Elimination
    
    EquivalentExpressions
- Catalyst DSL
  Catalyst DSL
  - DslLogicalPlan
- Variable Substitution
Query Execution
Query Execution
- Catalyst
  Catalyst
- Catalyst Expressions
  Catalyst Expressions
  - AggregateExpression
  - AggregateFunction
  - AggregateWindowFunction
  - Aggregator
  - ArrayFilter
  - AttributeSeq
  - Attribute
  - BasePredicate
  - BinaryComparison
  - BinaryOperator
  - BloomFilterAggregate
  - BloomFilterMightContain
  - BoundReference
  - CallMethodViaReflection
  - CodeGeneratorWithInterpretedFallback
  - CodegenFallback
  - Collect
  - CollectSet
  - Count
  - CreateNamedStruct
  - CreateStruct
  - CumeDist
  - DeclarativeAggregate
  - DecodeUsingSerializer
  - DynamicPruningExpression
  - DynamicPruningSubquery
  - EncodeUsingSerializer
  - EqualNullSafe
  - EqualTo
  - ExecSubqueryExpression
  - Exists
  - ExpectsInputTypes
  - ExplodeBase
  - Expression
  - First
  - Generator
  - HashExpression
  - HashPartitioning
  - Higher-Order Functions
    Higher-Order Functions
    
    HigherOrderFunction
    
    ArrayBasedSimpleHigherOrderFunction
    
    MapBasedSimpleHigherOrderFunction
    
    SimpleHigherOrderFunction
  - ImperativeAggregate
  - In
  - Inline
  - InSet
  - InSubquery
  - InSubqueryExec
  - InterpretedProjection
  - JsonToStructs
  - LessThanOrEqual
  - ListQuery
  - Literal
  - MonotonicallyIncreasingID
  - Murmur3Hash
  - MutableProjection
  - NamedExpression
  - Nondeterministic
  - OffsetWindowFunction
  - ParseToDate
  - ParseToTimestamp
  - PlanExpression
  - Predicate
  - Projection
  - PythonUDF
  - RowOrdering
  - RuntimeReplaceable
  - ScalaAggregator
  - ScalarSubquery
  - ScalarSubquery (ExecSubqueryExpression)
  - ScalaUDAF
  - ScalaUDF
  - SimpleTypedAggregateExpression
  - SortOrder
  - SparkUserDefinedFunction
  - Stateful
  - StaticInvoke
  - SubqueryExpression
  - TimeWindow
  - TypedImperativeAggregate
  - UnaryExpression
  - Unevaluable
  - UnixTimestamp
  - UnresolvedAttribute
  - UnresolvedFunction
  - UnresolvedGenerator
  - UnresolvedOrdinal
  - UnresolvedStar
  - UnsafeProjection
  - UserDefinedAggregator
  - User-Defined Functions
    User-Defined Functions
    
    User-Defined Functions
    
    UserDefinedExpression
    
    UserDefinedFunction
    
    UDFs are Blackbox
  - UserDefinedAggregateFunction
  - WindowExpression
  - WindowFunction
  - WindowSpecDefinition
- Execution Planning Strategies
  Execution Planning Strategies
- Logical Analyzer
- Logical Analysis Rules
  Logical Analysis Rules
- Logical Operators
  Logical Operators
- Logical Optimizations
  Logical Optimizations
- Physical Operators
  Physical Operators
  - AdaptiveSparkPlanExec
  - AggregateCodegenSupport
  - AliasAwareOutputExpression
  - AliasAwareQueryOutputOrdering
  - AlterTableExec
  - AQEShuffleReadExec
  - AtomicTableWriteExec
  - BaseAggregateExec
  - BaseJoinExec
  - BaseSubqueryExec
  - BatchWriteHelper
  - BatchScanExec
  - BroadcastExchangeExec
  - BroadcastExchangeLike
  - BroadcastHashJoinExec
  - BroadcastNestedLoopJoinExec
  - BroadcastQueryStageExec
  - CoalesceExec
  - CollectLimitExec
  - CollectMetricsExec
  - ColumnarToRowExec
  - ColumnarToRowTransition
  - CreateTableAsSelectExec
  - CodegenSupport
  - DataSourceScanExec
  - DataSourceV2ScanExecBase
  - DataWritingCommandExec
  - DebugExec
  - DeleteFromTableExec
  - DescribeTableExec
  - DeserializeToObjectExec
  - DropNamespaceExec
  - EvalPythonExec
  - Exchange
  - ExecutedCommandExec
  - ExternalRDDScanExec
  - FileSourceScanExec
  - FilterExec
  - GenerateExec
  - HashAggregateExec
  - HashedRelation
  - HashJoin
  - InMemoryTableScanExec
  - InputAdapter
  - JoinCodegenSupport
  - LocalTableScanExec
  - LongHashedRelation
  - ObjectConsumerExec
  - ObjectHashAggregateExec
  - ObjectProducerExec
  - OrderPreservingUnaryExecNode
  - OverwriteByExpressionExec
  - PartitioningPreservingUnaryExecNode
  - ProjectExec
  - QueryStageExec
  - RangeExec
  - ReusedExchangeExec
  - ReusedSubqueryExec
  - RowDataSourceScanExec
  - RowToColumnarExec
  - SerializeFromObjectExec
  - SetCatalogAndNamespaceExec
  - ShowCreateTableExec
  - ShowTablesExec
  - ShowTablePropertiesExec
  - ShuffleExchangeExec
  - ShuffleExchangeLike
  - ShuffledHashJoinExec
  - ShuffledJoin
  - ShuffleOrigin
  - ShuffleQueryStageExec
  - SortAggregateExec
  - SortMergeJoinExec
    SortMergeJoinExec
    
    SortMergeJoinExec
    
    SortMergeJoinScanner
  - SortExec
  - SparkPlan
  - SubqueryExec
  - TableWriteExecHelper
  - TruncateTableExec
  - UnaryExecNode
  - V2CommandExec
  - V2ExistingTableWriteExec
  - V2TableWriteExec
  - WholeStageCodegenExec
  - WindowExec
    WindowExec
    
    WindowExec
  - WindowExecBase
  - WriteFilesExec
  - Distribution and Partitioning
    Distribution and Partitioning
    
    Distribution
    
    Partitioning
    
    Distribution Specifications
    Distribution Specifications
    
    AllTuples
    
    BroadcastDistribution
    
    ClusteredDistribution
    
    HashClusteredDistribution
    
    OrderedDistribution
    
    UnspecifiedDistribution
    
    Broadcast Modes
    Broadcast Modes
    
    BroadcastMode
    
    HashedRelationBroadcastMode
    
    ShuffleSpec
- Physical Optimizations
  Physical Optimizations
- QueryExecution
- QueryPlanningTracker
- SparkOptimizer
- SparkPlanner
Internals
Internals
- Overview
- DataSource
- Developer API Developer API
  Table of contents
  - DeveloperApi
- DataSource V1 API
  DataSource V1 API
- ExecutionListenerBus
- ExecutionListenerManager
- SharedState
- SQLConf
- SQLConfHelper
- StaticSQLConf
- SparkSession Registries
  SparkSession Registries
  - Catalog
    Catalog
    
    Catalog
    
    CatalogImpl
    
    CatalogStatistics
    
    CatalogUtils
  - ExperimentalMethods
  - ExternalCatalog
    ExternalCatalog
    
    ExternalCatalog
    
    InMemoryCatalog
    
    ExternalCatalogWithListener
  - FunctionRegistry
    FunctionRegistry
    
    FunctionRegistry
    
    FunctionRegistryBase
    
    SimpleFunctionRegistry
    
    SimpleFunctionRegistryBase
    
    SimpleTableFunctionRegistry
    
    TableFunctionRegistry
  - GlobalTempViewManager
  - SessionCatalog
    SessionCatalog
    
    SessionCatalog
    
    CatalogStorageFormat
    
    CatalogTable
    
    CatalogTablePartition
  - V2SessionCatalog
  - SessionState
    SessionState
    
    SessionState
    
    BaseSessionStateBuilder
    
    SessionStateBuilder
  - CacheManager
    CacheManager
    
    CacheManager
  - RuntimeConfig
- Encoder
  Encoder
- SQLExecution
- SQLMetric
- Tungsten Execution Backend
  Tungsten Execution Backend
- RDDs
  RDDs
SQL
SQL
Connectors
Connectors
High-Level APIs
High-Level APIs
- Column
- ColumnarRule
- Dataset
- DataFrame
- DataFrameReader
- DataFrameWriter
- DataFrameWriterV2
- Encoders
- KeyValueGroupedDataset
- Observation
- QueryExecutionListener
- RelationalGroupedDataset
- SparkSession
- SparkSession.Builder
- SparkSessionExtensions
- Standard Functions
  Standard Functions
- TypedColumn
- Window Functions
  Window Functions
Web UI
Web UI
Demo
Demo
Misc
Misc
- AggregatingAccumulator
- JoinSelectionHelper
- PushDownUtils
- UnsafeExternalRowSorter
- BindReferences
- IntervalUtils
- ExplainUtils
- SerializerBuildHelper
- Dataset, DataFrame and RDD
- Dataset and SQL
- Implicits
- Row
- Data Source API
  Data Source API
  - CreateTableWriter
  - WriteConfigMethods
- Dataset API
  Dataset API
- Column Operators
- Data Types
  Data Types
- Caching and Persistence
  Caching and Persistence
  - Caching and Persistence
  - User-Friendly Names of Cached Queries in web UI
- Checkpointing
- Performance Tuning and Debugging
  Performance Tuning and Debugging
- CheckAnalysis
- CatalystTypeConverters
- SubExprUtils
- PredicateHelper
- ExtractEquiJoinKeys
- ExtractSingleColumnNullAwareAntiJoin
- ExtractJoinWithBuckets
- PhysicalOperation
- KnownSizeEstimation

Table of contents

DeveloperApi

Spark SQL
Internals

Developer API¶

DeveloperApi¶

Column
KeyValueGroupedDataset
RelationalGroupedDataset
SparkSessionExtensions
TypedColumn
CachedBatch
CachedBatchSerializer
SimpleMetricsCachedBatchSerializer
UserDefinedType
ColumnarBatch

Copyright © 2023 Jacek Laskowski

Made with Material for MkDocs Insiders