Skip to content
The Internals of Spark SQL
Demos
Initializing search
spark-sql-internals
Home
Internals
SQL
Features
Data Sources
High-Level APIs
Web UI
Demo
Misc
The Internals of Spark SQL
spark-sql-internals
Home
Internals
Internals
Overview
Configuration Properties
DataSource
Developer API
Connector API
Connector API
ApplyTransform
Batch
BatchWrite
CustomMetric
DataSourceV2Implicits
DataWriter
DataWriterFactory
InputPartition
FileTable
MetadataColumn
MetadataColumnHelper
MetadataColumnsHelper
OptionsHelper
Partitioning
PartitionReader
PartitionReaderFactory
PartitionSpecsHelper
Predicate
RewritableTransform
Scan
ScanBuilder
SessionConfigSupport
SimpleTableProvider
StagedTable
SupportsAtomicPartitionManagement
SupportsDelete
SupportsDynamicOverwrite
SupportsMetadata
SupportsMetadataColumns
SupportsOverwrite
SupportsPartitionManagement
SupportsPushDownFilters
SupportsPushDownRequiredColumns
SupportsPushDownV2Filters
SupportsRead
SupportsReportStatistics
SupportsReportPartitioning
SupportsRowLevelOperations
SupportsStreamingUpdate
SupportsTruncate
SupportsWrite
Table
TableCapability
TableHelper
TableProvider
Transform
TruncatableTable
V1Scan
V1Table
V1WriteBuilder
Write
WriteBuilder
DataSource V1 API
DataSource V1 API
DataSourceRegister
CreatableRelationProvider
RelationProvider
SchemaRelationProvider
BaseRelation
FileRelation
InsertableRelation
PrunedFilteredScan
PrunedScan
TableScan
Filter
ExecutionListenerBus
ExecutionListenerManager
Query Execution
Query Execution
Logical Analyzer
QueryExecution
QueryPlanningTracker
SparkOptimizer
SparkPlanner
SharedState
SQLConf
StaticSQLConf
Statistics and Hints
Statistics and Hints
Statistics
LogicalPlanStats
LogicalPlanVisitor
SizeInBytesOnlyStatsPlanVisitor
BasicStatsPlanVisitor
JoinEstimation
Logical Operators
Logical Operators
AddColumns
Aggregate
AlterTable
AlterTableAddColumnsCommand
AlterTableCommand
AnalyzeColumn
AnalyzeColumnCommand
AnalyzePartitionCommand
AnalyzeTable
AnalyzeTableCommand
AnalyzeTablesCommand
AppendData
CacheTableCommand
ClearCacheCommand
CollectMetrics
Command
CommentOnTable
CreateDataSourceTableAsSelectCommand
CreateDataSourceTableCommand
CreateTable
CreateTableAsSelect
CreateTempViewUsing
CreateViewCommand
CTERelationDef
CTERelationRef
DataSourceV2Relation
DataSourceV2ScanRelation
DataWritingCommand
DeleteFromTable
DescribeColumnCommand
DescribeRelation
DescribeTableCommand
DeserializeToObject
Except
Expand
ExplainCommand
ExternalRDD
FlatMapGroupsWithState
Generate
GlobalLimit
GroupingSets
IgnoreCachedData
InMemoryRelation
InsertIntoDataSourceCommand
InsertIntoDir
InsertIntoHadoopFsRelationCommand
InsertIntoStatement
InsertIntoTable
Join
LeafNode
LeafRunnableCommand
LoadDataCommand
LocalRelation
LogicalPlan
LogicalQueryStage
LogicalRDD
LogicalRelation
MapPartitions
MergeIntoTable
MultiInstanceRelation
NamedRelation
OverwriteByExpression
OverwritePartitionsDynamic
ParsedStatement
Pivot
Project
RebalancePartitions
Repartition
RepartitionByExpression
RepartitionOperation
ReplaceData
ResolvedHint
ResolvedTable
RowLevelWrite
RunnableCommand
SaveIntoDataSourceCommand
SetCatalogAndNamespace
ShowCreateTableCommand
ShowTableProperties
ShowTablePropertiesCommand
ShowTables
Sort
SubqueryAlias
SupportsSubquery
TruncateTableCommand
UnresolvedHaving
UnresolvedHint
UnresolvedTable
UnresolvedTableOrView
UnresolvedWith
UpdateTable
V2CreateTablePlan
V2WriteCommand
View
Window
WithCTE
WithWindowDefinition
SparkSession Registries
SparkSession Registries
Catalog
Catalog
Catalog
CatalogImpl
CatalogStatistics
ExperimentalMethods
ExternalCatalog
ExternalCatalog
ExternalCatalog
InMemoryCatalog
FunctionRegistry
FunctionRegistry
FunctionRegistry
FunctionRegistryBase
SimpleFunctionRegistry
SimpleFunctionRegistryBase
SimpleTableFunctionRegistry
TableFunctionRegistry
GlobalTempViewManager
SessionCatalog
SessionCatalog
SessionCatalog
BucketSpec
CatalogStorageFormat
CatalogTable
CatalogTablePartition
V2SessionCatalog
SessionState
SessionState
SessionState
BaseSessionStateBuilder
SessionStateBuilder
CacheManager
CacheManager
CacheManager
CachedRDDBuilder
RuntimeConfig
UDFRegistration
Catalyst
Catalyst
GenericStrategy
Optimizer
PlanChangeLogger
QueryPlan
QueryPlanner
Rule
RuleExecutor
TreeNode
TreePattern
TreePatternBits
Encoder
Encoder
Encoder
ExpressionEncoder
RowEncoder
ScalaReflection
Expressions
Expressions
AggregateExpression
AggregateFunction
AggregateWindowFunction
Aggregator
ArrayFilter
AttributeSeq
Attribute
BasePredicate
BinaryComparison
BinaryOperator
BloomFilterAggregate
BloomFilterMightContain
BoundReference
CallMethodViaReflection
CodeGeneratorWithInterpretedFallback
CodegenFallback
Collect
CollectSet
CreateNamedStruct
CreateStruct
CumeDist
DeclarativeAggregate
DecodeUsingSerializer
DynamicPruningExpression
DynamicPruningSubquery
EncodeUsingSerializer
EqualNullSafe
EqualTo
ExecSubqueryExpression
Exists
ExpectsInputTypes
ExplodeBase
Expression
First
Generator
HashExpression
HashPartitioning
Higher-Order Functions
Higher-Order Functions
HigherOrderFunction
ArrayBasedSimpleHigherOrderFunction
MapBasedSimpleHigherOrderFunction
SimpleHigherOrderFunction
ImperativeAggregate
In
Inline
InSet
InSubqueryExec
InterpretedProjection
JsonToStructs
LessThanOrEqual
ListQuery
Literal
MonotonicallyIncreasingID
Murmur3Hash
MutableProjection
NamedExpression
Nondeterministic
OffsetWindowFunction
ParseToDate
ParseToTimestamp
PlanExpression
Predicate
Projection
RowOrdering
RuntimeReplaceable
ScalaAggregator
ScalarSubquery
ScalarSubquery (ExecSubqueryExpression)
ScalaUDF
SortOrder
SparkUserDefinedFunction
Stateful
StaticInvoke
SubqueryExpression
TimeWindow
TypedImperativeAggregate
UnaryExpression
Unevaluable
UnixTimestamp
UnresolvedAttribute
UnresolvedFunction
UnresolvedGenerator
UnresolvedOrdinal
UnresolvedStar
UnsafeProjection
UserDefinedAggregator
User-Defined Functions
User-Defined Functions
User-Defined Functions
UserDefinedExpression
UserDefinedFunction
UDFs are Blackbox
UserDefinedAggregateFunction
WindowExpression
WindowFunction
WindowSpecDefinition
Physical Operators
Physical Operators
AdaptiveSparkPlanExec
AggregateCodegenSupport
AggregationIterators
AggregationIterators
AggregationIterator
ObjectAggregationIterator
ObjectAggregationMap
SortBasedAggregationIterator
TungstenAggregationIterator
AliasAwareOutputExpression
AliasAwareOutputOrdering
AliasAwareOutputPartitioning
AlterTableExec
AQEShuffleReadExec
AtomicTableWriteExec
BaseAggregateExec
BaseJoinExec
BaseSubqueryExec
BatchWriteHelper
BatchScanExec
BroadcastExchangeExec
BroadcastExchangeLike
BroadcastHashJoinExec
BroadcastNestedLoopJoinExec
BroadcastQueryStageExec
CoalesceExec
CollectLimitExec
CollectMetricsExec
ColumnarToRowExec
ColumnarToRowTransition
CreateTableAsSelectExec
CodegenSupport
ColumnarBatchScan
DataSourceScanExec
DataSourceV2ScanExec
DataSourceV2ScanExecBase
DataWritingCommandExec
DebugExec
DeleteFromTableExec
DescribeTableExec
DeserializeToObjectExec
DropNamespaceExec
EvalPythonExec
Exchange
ExecutedCommandExec
ExternalRDDScanExec
FileSourceScanExec
FilterExec
GenerateExec
HashAggregateExec
HashedRelation
HashJoin
InMemoryTableScanExec
InputAdapter
JoinCodegenSupport
LocalTableScanExec
LongHashedRelation
ObjectConsumerExec
ObjectHashAggregateExec
ObjectProducerExec
OverwriteByExpressionExec
ProjectExec
QueryStageExec
RangeExec
ReusedExchangeExec
ReusedSubqueryExec
RowDataSourceScanExec
SerializeFromObjectExec
SetCatalogAndNamespaceExec
ShowTablesExec
ShowTablePropertiesExec
ShuffleExchangeExec
ShuffleExchangeLike
ShuffledHashJoinExec
ShuffledJoin
ShuffleOrigin
ShuffleQueryStageExec
SortAggregateExec
SortMergeJoinExec
SortMergeJoinExec
SortMergeJoinExec
SortMergeJoinScanner
SortExec
SparkPlan
SubqueryExec
TableWriteExecHelper
UnaryExecNode
V2CommandExec
V2ExistingTableWriteExec
V2TableWriteExec
WholeStageCodegenExec
WindowExec
WindowExec
WindowExec
WindowExecBase
Distribution and Partitioning
Distribution and Partitioning
Distribution
Partitioning
Distribution Specifications
Distribution Specifications
AllTuples
BroadcastDistribution
ClusteredDistribution
HashClusteredDistribution
OrderedDistribution
UnspecifiedDistribution
Broadcast Modes
Broadcast Modes
BroadcastMode
HashedRelationBroadcastMode
Logical Analysis Rules
Logical Analysis Rules
AddMetadataColumns
AliasViewChild
CleanupAliases
CTESubstitution
DataSourceAnalysis
ExtractWindowExpressions
FindDataSourceTable
HandleNullInputsForUDF
LookupFunctions
PreprocessTableCreation
PreWriteCheck
RemoveAllHints
ResolveAggregateFunctions
ResolveAliases
ResolveCatalogs
ResolveCoalesceHints
ResolveCreateNamedStruct
ResolveFunctions
ResolveGroupingAnalytics
ResolveInlineTables
ResolveInsertInto
ResolveJoinStrategyHints
ResolveMissingReferences
ResolveOrdinalInOrderByAndGroupBy
ResolveOutputRelation
ResolveReferences
ResolveRelations
ResolveSessionCatalog
ResolveSQLOnFile
ResolveSubquery
ResolveTables
ResolveTempViews
ResolveTimeZone
ResolveWindowFrame
ResolveWindowOrder
ResolveWithCTE
RewriteDeleteFromTable
RewriteRowLevelCommand
TableCapabilityCheck
UpdateOuterReferences
WindowFrameCoercion
WidenSetOperationTypes
WindowsSubstitution
Logical Optimizations
Logical Optimizations
AQEPropagateEmptyRelation
CleanupDynamicPruningFilters
CollapseWindow
ColumnPruning
CombineTypedFilters
CombineUnions
ComputeCurrentTime
ConstantFolding
ConvertToLocalRelation
CostBasedJoinReorder
DecimalAggregates
DynamicJoinSelection
EliminateResolvedHint
EliminateSerialization
EliminateSubqueryAliases
EliminateView
ExtractPythonUDFFromAggregate
GetCurrentDatabase
GroupBasedRowLevelOperationScanPlanning
InferFiltersFromConstraints
InjectRuntimeFilter
InlineCTE
LimitPushDown
NullPropagation
OptimizeIn
OptimizeMetadataOnlyQuery
OptimizeSubqueries
PartitionPruning
PropagateEmptyRelation
PruneFileSourcePartitions
PruneFilters
PruneHiveTablePartitions
PullupCorrelatedPredicates
PushDownOperatorsToDataSource
PushDownPredicate
PushDownPredicates
PushPredicateThroughJoin
ReorderJoin
ReplaceExpressions
ReplaceExceptWithAntiJoin
ReplaceExceptWithFilter
RewriteCorrelatedScalarSubquery
RewriteExceptAll
RewritePredicateSubquery
SchemaPruning
SimplifyCasts
UpdateCTERelationStats
UpdateAttributeNullability
V2ScanRelationPushDown
V2Writes
Execution Planning Strategies
Execution Planning Strategies
SparkStrategy
Aggregation
BasicOperators
DataSourceStrategy
DataSourceV2Strategy
FileSourceStrategy
InMemoryScans
JoinSelection
LogicalQueryStageStrategy
SpecialLimits
SparkStrategies
Window
WithCTEStrategy
Physical Optimizations
Physical Optimizations
ApplyColumnarRulesAndInsertTransitions
AQEShuffleReadRule
CoalesceBucketsInJoin
CoalesceShufflePartitions
CollapseCodegenStages
DisableUnnecessaryBucketedScan
EnsureRequirements
ExtractPythonUDFs
InsertAdaptiveSparkPlan
OptimizeShuffleWithLocalRead
OptimizeSkewedJoin
OptimizeSkewInRebalancePartitions
PlanAdaptiveDynamicPruningFilters
PlanAdaptiveSubqueries
PlanDynamicPruningFilters
PlanSubqueries
RemoveRedundantProjects
RemoveRedundantSorts
ReuseAdaptiveSubquery
ReuseExchange
ReuseExchangeAndSubquery
ReuseSubquery
SQLExecution
SQLMetric
Tungsten Execution Backend
Tungsten Execution Backend
CatalystSerde
ExternalAppendOnlyUnsafeRowArray
HashMapGenerator
InternalRow
UnsafeFixedWidthAggregationMap
UnsafeHashedRelation
UnsafeRow
UnsafeRowSerializerInstance
SQL
SQL
AbstractSqlParser
AstBuilder
CatalystSqlParser
ParserInterface
SparkSqlParser
SparkSqlAstBuilder
VariableSubstitution
Features
Features
Adaptive Query Execution
Adaptive Query Execution
AdaptiveExecutionContext
AQEOptimizer
AQEUtils
CostEvaluator
ShufflePartitionsUtil
SimpleCostEvaluator
Demo
Bucketing
Catalog Plugin API
Catalog Plugin API
CatalogExtension
CatalogHelper
CatalogManager
CatalogPlugin
Catalogs
CatalogV2Util
DelegatingCatalogExtension
FunctionCatalog
StagingTableCatalog
SupportsNamespaces
SupportsCatalogOptions
TableCatalog
TableChange
V2TableWithV1Fallback
Columnar Execution
Columnar Execution
Common Table Expressions
Cost-Based Optimization
Cost-Based Optimization
CatalogColumnStat
ColumnStat
CommandUtils
EstimationUtils
Join Queries
Join Queries
Joins
Broadcast Joins
Statistics
Vectorized Decoding
Vectorized Decoding
ColumnVector
OffHeapColumnVector
OnHeapColumnVector
WritableColumnVector
Metadata Columns
Multi-Dimensional Aggregation
ANSI Intervals
Catalog Plugin API and Multi-Catalog Support
Dynamic Partition Pruning
Explaining Query Plans Improved
Hints
Hints
Hint Framework
JoinHint
HintInfo
Join Strategy Hints
HintErrorHandler
Observable Metrics
DataSource V2
Hive Integration
Dynamic Partition Inserts
Whole-Stage CodeGen
Whole-Stage CodeGen
BufferedRowIterator
CodegenContext
CodeGenerator
Block
GenerateUnsafeProjection
GenerateMutableProjection
GenerateColumnAccessor
GenerateOrdering
GeneratePredicate
GenerateSafeProjection
Subexpression Elimination
Subexpression Elimination
Subexpression Elimination
EquivalentExpressions
Vectorized Query Execution
Vectorized Query Execution
Vectorized Query Execution
ColumnarBatch
Subqueries
Catalyst DSL
Catalyst DSL
DslLogicalPlan
Variable Substitution
Data Sources
Data Sources
Avro
Avro
AvroOptions
AvroFileFormat
CatalystDataToAvro
AvroDataToCatalyst
Console
Console
ConsoleSinkProvider
CSV
CSV
CSVFileFormat
CSVScanBuilder
Files
Files
BasicWriteJobStatsTracker
BasicWriteTaskStats
FileBatchWrite
FileDataSourceV2
FileFormat
FileFormatDataWriter
FileFormatWriter
FileIndex
FileIndex
FileIndex
CatalogFileIndex
InMemoryFileIndex
PartitioningAwareFileIndex
PrunedInMemoryFileIndex
FilePartition
FilePartitionReaderFactory
FileScan
FileScanBuilder
FileWrite
FileWriterFactory
HadoopFileLinesReader
HadoopFsRelation
PartitionedFile
RecordReaderIterator
SingleDirectoryDataWriter
SQLHadoopMapReduceCommitProtocol
TextBasedFileFormat
WriteJobStatsTracker
WriteTaskStats
Hive
Hive
Configuration Properties
Hive Metastore
DataSinks
HiveClient
HiveClientImpl
HiveFileFormat
HiveUtils
IsolatedClientLoader
HiveTableRelation
CreateHiveTableAsSelectCommand
SaveAsHiveFile
InsertIntoHiveDirCommand
InsertIntoHiveTable
HiveTableScans
HiveTableScanExec
TableReader
HadoopTableReader
HiveSessionStateBuilder
HiveExternalCatalog
HiveSessionCatalog
HiveMetastoreCatalog
RelationConversions
ResolveHiveSerdeTable
DetermineTableStats
HiveAnalysis
JDBC
JDBC
JDBCOptions
JdbcRelationProvider
JDBCRelation
JDBCScan
JDBCScanBuilder
JDBCRDD
JdbcDialect
JSON
JSON
JsonFileFormat
JsonDataSource
Kafka
Kafka
Configuration Properties
KafkaBatch
KafkaBatchWrite
KafkaBatchWriterFactory
KafkaDataConsumer
KafkaDataWriter
KafkaOffsetRangeLimit
KafkaOffsetReader
KafkaRelation
KafkaScan
KafkaSourceProvider
KafkaSourceRDD
KafkaSourceRDDPartition
KafkaTable
KafkaWrite
KafkaWriter
KafkaWriteTask
Misc
Misc
ConsumerStrategy
InternalKafkaConsumer
InternalKafkaProducerPool
JsonUtils
KafkaRowWriter
KafkaRecordToRowConverter
Options
Noop
Noop
NoopDataSource
NoopTable
NoopWriteBuilder
NoopBatchWrite
NoopStreamingWrite
ORC
ORC
OrcScanBuilder
OrcFileFormat
Parquet
Parquet
ParquetDataSourceV2
ParquetFileFormat
ParquetFilters
ParquetOptions
ParquetPartitionReaderFactory
ParquetReadSupport
ParquetScan
ParquetScanBuilder
ParquetTable
ParquetUtils
ParquetWrite
ParquetWriteSupport
SparkToParquetSchemaConverter
SpecificParquetRecordReaderBase
VectorizedColumnReader
VectorizedParquetRecordReader
Text
Text
TextFileFormat
BaseDynamicPartitionDataWriter
BasicWriteTaskStatsTracker
DataWritingSparkTask
DataSourceV2Utils
DynamicPartitionDataConcurrentWriter
DynamicPartitionDataSingleWriter
OutputWriter
WriteTaskStatsTracker
High-Level APIs
High-Level APIs
Basic Aggregation
Basic Aggregation
KeyValueGroupedDataset
RelationalGroupedDataset
ColumnarRule
Dataset
DataFrame
DataFrameReader
DataFrameWriter
DataFrameWriterV2
Encoders
Observation
QueryExecutionListener
SparkSession
SparkSession.Builder
SparkSessionExtensions
Standard Functions
Standard Functions
Standard Aggregate Functions
Standard Collection Functions
Date and Time Functions
Regular Functions (Non-Aggregate Functions)
Standard Functions for Window Aggregation (Window Functions)
Window Functions
Window Functions
AggregateProcessor
RangeFrame
Window
WindowFunctionFrame
WindowSpec
Web UI
Web UI
AllExecutionsPage
ExecutionPage
SparkListenerSQLExecutionEnd
SQLAppStatusListener
SQLAppStatusStore
SQLTab
Demo
Demo
Connecting Spark SQL to Hive Metastore (with Remote Metastore Server)
Developing CatalogPlugin
Hive Partitioned Parquet Table and Partition Pruning
ObjectHashAggregateExec and Sort-Based Fallback Tasks
Spilling
Using JDBC Data Source to Access PostgreSQL
Misc
Misc
AggregatingAccumulator
Bloom Filter
Bloom Filter
BloomFilter
BloomFilterImpl
PushDownUtils
UnsafeExternalRowSorter
BindReferences
IntervalUtils
ExplainUtils
PartitionedFileUtil
SerializerBuildHelper
SQLExecutionRDD
DataSourceRDD
DataSourceRDDPartition
Dataset, DataFrame and RDD
Dataset and SQL
Implicits
Row
Data Source API
Data Source API
DataSource API
CreateTableWriter
WriteConfigMethods
Dataset API
Dataset API
Dataset API — Dataset Operators
Typed Transformations
Untyped Transformations
Dataset API — Basic Actions
Actions
DataFrameNaFunctions — Working With Missing Data
DataFrameStatFunctions
Column
Column
Column
Column Operators
TypedColumn
Data Types
Data Types
AbstractDataType
ArrayType
AtomicType
CalendarInterval
DataType
StructField
StructType
Caching and Persistence
Caching and Persistence
Caching and Persistence
User-Friendly Names of Cached Queries in web UI
Checkpointing
FileScanRDD
Logging
Performance Tuning and Debugging
Performance Tuning and Debugging
Debugging Query Execution
Performance Tuning
Case Study
Spark Thrift Server
Spark Thrift Server
HiveThriftServer2
SparkSQLEnv
RDDConversions
ShuffledRowRDD
CheckAnalysis
CatalystTypeConverters
SubExprUtils
PredicateHelper
AggUtils
ExtractEquiJoinKeys
ExtractSingleColumnNullAwareAntiJoin
ExtractJoinWithBuckets
PhysicalAggregation
PhysicalOperation
KnownSizeEstimation
CompressionCodecs
JoinStrategyHint
Demos
¶
Back to top