Skip to content
The Internals of Spark SQL
Demos
Initializing search
spark-sql-internals
Spark SQL
Features
Query Execution
Internals
SQL
Connectors
High-Level APIs
Web UI
Demo
Misc
The Internals of Spark SQL
spark-sql-internals
Spark SQL
Features
Features
Aggregations
Aggregations
Demo: Mult-Dimensional Aggregations
Adaptive Query Execution
Adaptive Query Execution
AdaptiveExecutionContext
AQEOptimizer
AQEUtils
CostEvaluator
ShufflePartitionsUtil
SimpleCostEvaluator
Bucketing
Bucketing
BucketSpec
Cache Serialization
Cache Serialization
CachedBatch
CachedBatchSerializer
CachedRDDBuilder
DefaultCachedBatchSerializer
SimpleMetricsCachedBatch
SimpleMetricsCachedBatchSerializer
Catalog Plugin API
Catalog Plugin API
CatalogExtension
CatalogHelper
CatalogManager
CatalogPlugin
Catalogs
CatalogV2Util
Column
DelegatingCatalogExtension
FunctionCatalog
MetadataColumn
StagingTableCatalog
SupportsNamespaces
SupportsCatalogOptions
TableCatalog
TableChange
V2TableWithV1Fallback
Columnar Execution
Columnar Execution
Common Table Expressions
Common Table Expressions
Configuration Properties
Connector API
Connector API
ApplyTransform
Batch
BatchWrite
CustomMetric
DataSourceV2Implicits
DataWriter
DataWriterFactory
Expression
InputPartition
LocalScan
OptionsHelper
PartitionReader
PartitionReaderFactory
PartitionSpecsHelper
Partitioning
Predicate
RewritableTransform
Scan
ScanBuilder
SessionConfigSupport
SimpleTableProvider
StagedTable
SupportsAtomicPartitionManagement
SupportsDelete
SupportsDeleteV2
SupportsDynamicOverwrite
SupportsMetadata
SupportsMetadataColumns
SupportsOverwrite
SupportsPartitionManagement
SupportsPushDownAggregates
SupportsPushDownFilters
SupportsPushDownRequiredColumns
SupportsPushDownV2Filters
SupportsRead
SupportsReportOrdering
SupportsReportPartitioning
SupportsReportStatistics
SupportsRowLevelOperations
SupportsStreamingUpdate
SupportsTruncate
SupportsWrite
Table
TableCapability
TableHelper
TableProvider
Transform
TransformHelper
TruncatableTable
V1Scan
V1Table
V1WriteBuilder
Write
WriteBuilder
Connector Expressions
Connector Expressions
Aggregation
SortOrder
Cost-Based Optimization
Cost-Based Optimization
BasicStatsPlanVisitor
CatalogColumnStat
ColumnStat
CommandUtils
EstimationUtils
JoinEstimation
LogicalPlanStats
LogicalPlanVisitor
SizeInBytesOnlyStatsPlanVisitor
Statistics
Default Columns
Default Columns
Dynamic Partition Pruning
Dynamic Partition Pruning
File-Based Data Scanning
File-Based Data Scanning
Generated Columns
Generated Columns
GeneratedColumn
Hints
Hints
HintErrorHandler
HintInfo
JoinHint
JoinStrategyHint
Join Strategy Hints
Join Queries
Join Queries
Joins
Broadcast Joins
Logging
Metadata Columns
Metadata Columns
FileSourceConstantMetadataAttribute
FileSourceGeneratedMetadataAttribute
FileSourceMetadataAttribute
MetadataAttribute
MetadataColumnHelper
MetadataColumnsHelper
Partition File Metadata Caching
Partition File Metadata Caching
Spark Connect
Spark Connect
CommandPlugin
ExpressionPlugin
RelationPlugin
SimpleSparkConnectService
SparkConnectAnalyzeHandler
SparkConnectInterceptorRegistry
SparkConnectPlanner
SparkConnectPlugin
SparkConnectServer
SparkConnectService
SparkConnectStreamHandler
Spark Thrift Server
Spark Thrift Server
SparkSQLEnv
Statistics
Subqueries
Subqueries
Table-Valued Functions
Table-Valued Functions
Transactional Writes
Transactional Writes
SQLHadoopMapReduceCommitProtocol
User-Defined Functions
User-Defined Functions
UDFRegistration
UserDefinedPythonFunction
Vectorized Decoding
Vectorized Decoding
ColumnVector
OffHeapColumnVector
OnHeapColumnVector
WritableColumnVector
ANSI Intervals
Catalog Plugin API and Multi-Catalog Support
Explaining Query Plans Improved
Observable Metrics
Hive Integration
Dynamic Partition Inserts
Vectorized Query Execution
Vectorized Query Execution
ColumnarBatch
Whole-Stage Code Generation
Whole-Stage Code Generation
Block
BufferedRowIterator
CodegenContext
CodeGenerator
GenerateUnsafeProjection
GenerateMutableProjection
GenerateColumnAccessor
GenerateOrdering
GeneratePredicate
GenerateSafeProjection
Subexpression Elimination
Subexpression Elimination
Subexpression Elimination
EquivalentExpressions
Catalyst DSL
Catalyst DSL
DslLogicalPlan
Variable Substitution
Query Execution
Query Execution
Catalyst
Catalyst
GenericStrategy
Optimizer
PlanChangeLogger
QueryPlan
QueryPlanner
Rule
RuleExecutor
TreeNode
TreePattern
TreePatternBits
Catalyst Expressions
Catalyst Expressions
AggregateExpression
AggregateFunction
AggregateWindowFunction
Aggregator
ArrayFilter
AttributeSeq
Attribute
BasePredicate
BinaryComparison
BinaryOperator
BloomFilterAggregate
BloomFilterMightContain
BoundReference
CallMethodViaReflection
CodeGeneratorWithInterpretedFallback
CodegenFallback
Collect
CollectSet
CreateNamedStruct
CreateStruct
CumeDist
DeclarativeAggregate
DecodeUsingSerializer
DynamicPruningExpression
DynamicPruningSubquery
EncodeUsingSerializer
EqualNullSafe
EqualTo
ExecSubqueryExpression
Exists
ExpectsInputTypes
ExplodeBase
Expression
First
Generator
HashExpression
HashPartitioning
Higher-Order Functions
Higher-Order Functions
HigherOrderFunction
ArrayBasedSimpleHigherOrderFunction
MapBasedSimpleHigherOrderFunction
SimpleHigherOrderFunction
ImperativeAggregate
In
Inline
InSet
InSubqueryExec
InterpretedProjection
JsonToStructs
LessThanOrEqual
ListQuery
Literal
MonotonicallyIncreasingID
Murmur3Hash
MutableProjection
NamedExpression
Nondeterministic
OffsetWindowFunction
ParseToDate
ParseToTimestamp
PlanExpression
Predicate
Projection
PythonUDF
RowOrdering
RuntimeReplaceable
ScalaAggregator
ScalarSubquery
ScalarSubquery (ExecSubqueryExpression)
ScalaUDF
SortOrder
SparkUserDefinedFunction
Stateful
StaticInvoke
SubqueryExpression
TimeWindow
TypedImperativeAggregate
UnaryExpression
Unevaluable
UnixTimestamp
UnresolvedAttribute
UnresolvedFunction
UnresolvedGenerator
UnresolvedOrdinal
UnresolvedStar
UnsafeProjection
UserDefinedAggregator
User-Defined Functions
User-Defined Functions
User-Defined Functions
UserDefinedExpression
UserDefinedFunction
UDFs are Blackbox
UserDefinedAggregateFunction
WindowExpression
WindowFunction
WindowSpecDefinition
Execution Planning Strategies
Execution Planning Strategies
SparkStrategy
Aggregation
BasicOperators
DataSourceStrategy
DataSourceV2Strategy
FileSourceStrategy
InMemoryScans
JoinSelection
LogicalQueryStageStrategy
SpecialLimits
SparkStrategies
Window
WithCTEStrategy
Logical Analyzer
Logical Analysis Rules
Logical Analysis Rules
AddMetadataColumns
AliasViewChild
CleanupAliases
CTESubstitution
DataSourceAnalysis
ExtractWindowExpressions
FindDataSourceTable
HandleNullInputsForUDF
LookupFunctions
PreprocessTableCreation
PreWriteCheck
RemoveAllHints
ResolveAggregateFunctions
ResolveAliases
ResolveCatalogs
ResolveCoalesceHints
ResolveCreateNamedStruct
ResolveDefaultColumns
ResolveFunctions
ResolveGroupingAnalytics
ResolveInlineTables
ResolveInsertInto
ResolveJoinStrategyHints
ResolveMissingReferences
ResolveOrdinalInOrderByAndGroupBy
ResolveOutputRelation
ResolveReferences
ResolveRelations
ResolveSessionCatalog
ResolveSQLOnFile
ResolveSubquery
ResolveTables
ResolveTempViews
ResolveTimeZone
ResolveWindowFrame
ResolveWindowOrder
ResolveWithCTE
RewriteDeleteFromTable
RewriteRowLevelCommand
TableCapabilityCheck
UpdateOuterReferences
WindowFrameCoercion
WidenSetOperationTypes
WindowsSubstitution
Logical Operators
Logical Operators
AddColumns
Aggregate
AlterTable
AlterTableAddColumnsCommand
AlterTableCommand
AnalyzeColumn
AnalyzeColumnCommand
AnalyzePartitionCommand
AnalyzeTable
AnalyzeTableCommand
AnalyzeTablesCommand
AppendData
ArrowEvalPython
BaseEvalPython
CacheTableCommand
ClearCacheCommand
CollectMetrics
Command
CommentOnTable
CreateDataSourceTableAsSelectCommand
CreateDataSourceTableCommand
CreateTable
CreateTableAsSelect
CreateTempViewUsing
CreateViewCommand
CTERelationDef
CTERelationRef
DataSourceV2Relation
DataSourceV2ScanRelation
DataWritingCommand
DeleteFromTable
DescribeColumnCommand
DescribeRelation
DescribeTableCommand
DeserializeToObject
Except
Expand
ExplainCommand
ExposesMetadataColumns
ExternalRDD
FlatMapGroupsWithState
Generate
GlobalLimit
GroupingSets
IgnoreCachedData
InMemoryRelation
InsertIntoDataSourceCommand
InsertIntoDir
InsertIntoHadoopFsRelationCommand
InsertIntoStatement
InsertIntoTable
Join
LeafNode
LeafRunnableCommand
LoadDataCommand
LocalRelation
LogicalPlan
LogicalQueryStage
LogicalRDD
LogicalRelation
MapPartitions
MergeIntoTable
MultiInstanceRelation
NamedRelation
OverwriteByExpression
OverwritePartitionsDynamic
ParsedStatement
Pivot
Project
RebalancePartitions
Repartition
RepartitionByExpression
RepartitionOperation
ReplaceData
ResolvedHint
ResolvedTable
RowLevelWrite
RunnableCommand
SaveIntoDataSourceCommand
SetCatalogAndNamespace
ShowCreateTable
ShowCreateTableCommand
ShowTableProperties
ShowTablePropertiesCommand
ShowTables
Sort
SubqueryAlias
SupportsSubquery
TruncateTableCommand
UnresolvedHaving
UnresolvedHint
UnresolvedRelation
UnresolvedTable
UnresolvedTableOrView
UnresolvedTableValuedFunction
UnresolvedWith
UpdateTable
V2CreateTablePlan
V2WriteCommand
View
Window
WithCTE
WithWindowDefinition
WriteDelta
WriteFiles
Logical Optimizations
Logical Optimizations
AQEPropagateEmptyRelation
CleanupDynamicPruningFilters
CollapseWindow
ColumnPruning
CombineTypedFilters
CombineUnions
ComputeCurrentTime
ConstantFolding
ConvertToLocalRelation
CostBasedJoinReorder
DecimalAggregates
DynamicJoinSelection
EliminateResolvedHint
EliminateSerialization
EliminateSubqueryAliases
EliminateView
ExtractPythonUDFs
ExtractPythonUDFFromAggregate
GetCurrentDatabase
GroupBasedRowLevelOperationScanPlanning
InferFiltersFromConstraints
InjectRuntimeFilter
InlineCTE
LimitPushDown
NullPropagation
OptimizeIn
OptimizeMetadataOnlyQuery
OptimizeSubqueries
PartitionPruning
PropagateEmptyRelation
PruneFileSourcePartitions
PruneFilters
PruneHiveTablePartitions
PullupCorrelatedPredicates
PushDownLeftSemiAntiJoin
PushDownOperatorsToDataSource
PushDownPredicate
PushDownPredicates
PushPredicateThroughJoin
ReorderJoin
ReplaceExpressions
ReplaceExceptWithAntiJoin
ReplaceExceptWithFilter
RewriteCorrelatedScalarSubquery
RewriteExceptAll
RewritePredicateSubquery
SchemaPruning
SimplifyCasts
UpdateCTERelationStats
UpdateAttributeNullability
V2ScanRelationPushDown
V2Writes
Physical Operators
Physical Operators
AdaptiveSparkPlanExec
AggregateCodegenSupport
AggregationIterators
AggregationIterators
AggregationIterator
ObjectAggregationIterator
ObjectAggregationMap
SortBasedAggregationIterator
TungstenAggregationIterator
AliasAwareOutputExpression
AliasAwareOutputOrdering
AliasAwareOutputPartitioning
AlterTableExec
AQEShuffleReadExec
AtomicTableWriteExec
BaseAggregateExec
BaseJoinExec
BaseSubqueryExec
BatchWriteHelper
BatchScanExec
BroadcastExchangeExec
BroadcastExchangeLike
BroadcastHashJoinExec
BroadcastNestedLoopJoinExec
BroadcastQueryStageExec
CoalesceExec
CollectLimitExec
CollectMetricsExec
ColumnarToRowExec
ColumnarToRowTransition
CreateTableAsSelectExec
CodegenSupport
DataSourceScanExec
DataSourceV2ScanExecBase
DataWritingCommandExec
DebugExec
DeleteFromTableExec
DescribeTableExec
DeserializeToObjectExec
DropNamespaceExec
EvalPythonExec
Exchange
ExecutedCommandExec
ExternalRDDScanExec
FileSourceScanExec
FilterExec
GenerateExec
HashAggregateExec
HashedRelation
HashJoin
InMemoryTableScanExec
InputAdapter
JoinCodegenSupport
LocalTableScanExec
LongHashedRelation
ObjectConsumerExec
ObjectHashAggregateExec
ObjectProducerExec
OverwriteByExpressionExec
ProjectExec
QueryStageExec
RangeExec
ReusedExchangeExec
ReusedSubqueryExec
RowDataSourceScanExec
RowToColumnarExec
SerializeFromObjectExec
SetCatalogAndNamespaceExec
ShowCreateTableExec
ShowTablesExec
ShowTablePropertiesExec
ShuffleExchangeExec
ShuffleExchangeLike
ShuffledHashJoinExec
ShuffledJoin
ShuffleOrigin
ShuffleQueryStageExec
SortAggregateExec
SortMergeJoinExec
SortMergeJoinExec
SortMergeJoinExec
SortMergeJoinScanner
SortExec
SparkPlan
SubqueryExec
TableWriteExecHelper
TruncateTableExec
UnaryExecNode
V2CommandExec
V2ExistingTableWriteExec
V2TableWriteExec
WholeStageCodegenExec
WindowExec
WindowExec
WindowExec
WindowExecBase
WriteFilesExec
Distribution and Partitioning
Distribution and Partitioning
Distribution
Partitioning
Distribution Specifications
Distribution Specifications
AllTuples
BroadcastDistribution
ClusteredDistribution
HashClusteredDistribution
OrderedDistribution
UnspecifiedDistribution
Broadcast Modes
Broadcast Modes
BroadcastMode
HashedRelationBroadcastMode
ShuffleSpec
Physical Optimizations
Physical Optimizations
AdjustShuffleExchangePosition
ApplyColumnarRulesAndInsertTransitions
AQEShuffleReadRule
CoalesceBucketsInJoin
CoalesceShufflePartitions
CollapseCodegenStages
DisableUnnecessaryBucketedScan
EnsureRequirements
InsertAdaptiveSparkPlan
OptimizeShuffleWithLocalRead
OptimizeSkewedJoin
OptimizeSkewInRebalancePartitions
PlanAdaptiveDynamicPruningFilters
PlanAdaptiveSubqueries
PlanDynamicPruningFilters
PlanSubqueries
RemoveRedundantProjects
RemoveRedundantSorts
ReplaceHashWithSortAgg
ReuseAdaptiveSubquery
ReuseExchange
ReuseExchangeAndSubquery
ReuseSubquery
ValidateSparkPlan
QueryExecution
QueryPlanningTracker
SparkOptimizer
SparkPlanner
Internals
Internals
Overview
DataSource
Developer API
DataSource V1 API
DataSource V1 API
DataSourceRegister
CreatableRelationProvider
RelationProvider
SchemaRelationProvider
BaseRelation
FileRelation
InsertableRelation
PrunedFilteredScan
PrunedScan
TableScan
Filter
ExecutionListenerBus
ExecutionListenerManager
SharedState
SQLConf
SQLConfHelper
StaticSQLConf
SparkSession Registries
SparkSession Registries
Catalog
Catalog
Catalog
CatalogImpl
CatalogStatistics
CatalogUtils
ExperimentalMethods
ExternalCatalog
ExternalCatalog
ExternalCatalog
InMemoryCatalog
ExternalCatalogWithListener
FunctionRegistry
FunctionRegistry
FunctionRegistry
FunctionRegistryBase
SimpleFunctionRegistry
SimpleFunctionRegistryBase
SimpleTableFunctionRegistry
TableFunctionRegistry
GlobalTempViewManager
SessionCatalog
SessionCatalog
SessionCatalog
CatalogStorageFormat
CatalogTable
CatalogTablePartition
V2SessionCatalog
SessionState
SessionState
SessionState
BaseSessionStateBuilder
SessionStateBuilder
CacheManager
CacheManager
CacheManager
RuntimeConfig
Encoder
Encoder
Encoder
ExpressionEncoder
RowEncoder
ScalaReflection
SQLExecution
SQLMetric
Tungsten Execution Backend
Tungsten Execution Backend
CatalystSerde
ExternalAppendOnlyUnsafeRowArray
HashMapGenerator
InternalRow
UnsafeFixedWidthAggregationMap
UnsafeHashedRelation
UnsafeRow
UnsafeRowSerializerInstance
RDDs
RDDs
DataSourceRDD
DataSourceRDDPartition
FileScanRDD
ShuffledRowRDD
SQLExecutionRDD
SQL
SQL
AbstractSqlParser
AstBuilder
CatalystSqlParser
ParserInterface
SparkSqlParser
SparkSqlAstBuilder
VariableSubstitution
Connectors
Connectors
Avro
Avro
AvroOptions
AvroFileFormat
CatalystDataToAvro
AvroDataToCatalyst
Files
Files
AggregatePushDownUtils
BasicWriteJobStatsTracker
BasicWriteTaskStats
CompressionCodecs
FileBatchWrite
FileDataSourceV2
FileFormat
FileFormatDataWriter
FileFormatWriter
FileIndex
FileIndex
FileIndex
CatalogFileIndex
InMemoryFileIndex
PartitioningAwareFileIndex
PrunedInMemoryFileIndex
FilePartition
FilePartitionReaderFactory
FileScan
FileScanBuilder
FileStatusCache
FileTable
FileWrite
FileWriterFactory
HadoopFileLinesReader
HadoopFsRelation
PartitionedFile
PartitionedFileUtil
RecordReaderIterator
SharedInMemoryCache
SchemaMergeUtils
SingleDirectoryDataWriter
WriteJobStatsTracker
WriteTaskStats
Hive
Hive
Configuration Properties
Hive Metastore
DataSinks
HiveClient
HiveClientImpl
HiveFileFormat
HiveUtils
IsolatedClientLoader
HiveTableRelation
CreateHiveTableAsSelectCommand
SaveAsHiveFile
InsertIntoHiveDirCommand
InsertIntoHiveTable
HiveTableScans
HiveTableScanExec
TableReader
HadoopTableReader
HiveSessionStateBuilder
HiveExternalCatalog
HiveSessionCatalog
HiveMetastoreCatalog
RelationConversions
ResolveHiveSerdeTable
DetermineTableStats
HiveAnalysis
JDBC
JDBC
JDBCOptions
JdbcRelationProvider
JDBCRelation
JDBCScan
JDBCScanBuilder
JDBCRDD
JdbcDialect
Kafka
Kafka
Configuration Properties
KafkaBatch
KafkaBatchWrite
KafkaBatchWriterFactory
KafkaDataConsumer
KafkaDataWriter
KafkaOffsetRangeLimit
KafkaOffsetReader
KafkaRelation
KafkaScan
KafkaSourceProvider
KafkaSourceRDD
KafkaSourceRDDPartition
KafkaTable
KafkaWrite
KafkaWriter
KafkaWriteTask
Misc
Misc
ConsumerStrategy
InternalKafkaConsumer
InternalKafkaProducerPool
JsonUtils
KafkaRowWriter
KafkaRecordToRowConverter
Options
Noop
Noop
NoopDataSource
NoopTable
NoopWriteBuilder
NoopBatchWrite
NoopStreamingWrite
Parquet
Parquet
ParquetDataSourceV2
ParquetFileFormat
ParquetFilters
ParquetOptions
ParquetPartitionReaderFactory
ParquetReadSupport
ParquetScan
ParquetScanBuilder
ParquetTable
ParquetUtils
ParquetWrite
ParquetWriteSupport
SparkToParquetSchemaConverter
SpecificParquetRecordReaderBase — Hadoop RecordReader
VectorizedColumnReader
VectorizedParquetRecordReader
BaseDynamicPartitionDataWriter
BasicWriteTaskStatsTracker
DataWritingSparkTask
DataSourceV2Utils
DynamicPartitionDataConcurrentWriter
DynamicPartitionDataSingleWriter
OutputWriter
WriteTaskStatsTracker
High-Level APIs
High-Level APIs
Column
ColumnarRule
Dataset
DataFrame
DataFrameReader
DataFrameWriter
DataFrameWriterV2
Encoders
KeyValueGroupedDataset
Observation
QueryExecutionListener
RelationalGroupedDataset
SparkSession
SparkSession.Builder
SparkSessionExtensions
Standard Functions
Standard Functions
Standard Aggregate Functions
Standard Collection Functions
Date and Time Functions
Regular Functions (Non-Aggregate Functions)
Standard Functions for Window Aggregation (Window Functions)
TypedColumn
Window Functions
Window Functions
AggregateProcessor
RangeFrame
Window
WindowFunctionFrame
WindowSpec
Web UI
Web UI
AllExecutionsPage
ExecutionPage
SparkListenerSQLExecutionEnd
SQLAppStatusListener
SQLAppStatusStore
SQLTab
Demo
Demo
Adaptive Query Execution
Connecting Spark SQL to Hive Metastore (with Remote Metastore Server)
Developing CatalogPlugin
Dynamic Partition Pruning
Hive Partitioned Parquet Table and Partition Pruning
ObjectHashAggregateExec and Sort-Based Fallback Tasks
Spilling
Using JDBC Data Source to Access PostgreSQL
Misc
Misc
AggregatingAccumulator
Bloom Filter
Bloom Filter
BloomFilter
BloomFilterImpl
JoinSelectionHelper
PushDownUtils
UnsafeExternalRowSorter
BindReferences
IntervalUtils
ExplainUtils
SerializerBuildHelper
Dataset, DataFrame and RDD
Dataset and SQL
Implicits
Row
Data Source API
Data Source API
CreateTableWriter
WriteConfigMethods
Dataset API
Dataset API
Dataset API — Dataset Operators
Typed Transformations
Untyped Transformations
Dataset API — Basic Actions
Actions
DataFrameNaFunctions — Working With Missing Data
DataFrameStatFunctions
Column Operators
Data Types
Data Types
AbstractDataType
ArrayType
AtomicType
CalendarInterval
DataType
StructField
StructType
UserDefinedType
Caching and Persistence
Caching and Persistence
Caching and Persistence
User-Friendly Names of Cached Queries in web UI
Checkpointing
Performance Tuning and Debugging
Performance Tuning and Debugging
Debugging Query Execution
Performance Tuning
Case Study
CheckAnalysis
CatalystTypeConverters
SubExprUtils
PredicateHelper
AggUtils
ExtractEquiJoinKeys
ExtractSingleColumnNullAwareAntiJoin
ExtractJoinWithBuckets
PhysicalAggregation
PhysicalOperation
KnownSizeEstimation
Spark SQL
Demo
Demos
¶
Back to top