Skip to content
The Internals of Spark SQL
AnalysisOnlyCommand
Initializing search
spark-sql-internals
Spark SQL
Features
Query Execution
Internals
SQL
Connectors
High-Level APIs
Web UI
Demo
Misc
The Internals of Spark SQL
spark-sql-internals
Spark SQL
Features
Features
Aggregate Queries
Aggregate Queries
AggUtils
AggregationIterator
KVSorterIterator
ObjectAggregationIterator
ObjectAggregationMap
PhysicalAggregation
SortBasedAggregationIterator
TungstenAggregationIterator
UnsafeFixedWidthAggregationMap
UnsafeKVExternalSorter
Adaptive Query Execution
Adaptive Query Execution
AQEOptimizer
AQEUtils
AdaptiveExecutionContext
AdaptiveRulesHolder
CostEvaluator
ShufflePartitionsUtil
SimpleCostEvaluator
Bloom Filter Join
Bloom Filter Join
BloomFilter
BloomFilterImpl
Bucketing
Bucketing
BucketSpec
Cache Serialization
Cache Serialization
CachedBatch
CachedBatchSerializer
CachedRDDBuilder
DefaultCachedBatchSerializer
SimpleMetricsCachedBatch
SimpleMetricsCachedBatchSerializer
Catalog Plugin API
Catalog Plugin API
CatalogExtension
CatalogHelper
CatalogManager
CatalogPlugin
Catalogs
CatalogV2Util
Column
DelegatingCatalogExtension
FunctionCatalog
MetadataColumn
StagingTableCatalog
SupportsNamespaces
SupportsCatalogOptions
TableCatalog
TableCatalogCapability
TableChange
V2TableWithV1Fallback Tables
Columnar Execution
Columnar Execution
Common Table Expressions
Common Table Expressions
Configuration Properties
Connector Expressions
Connector Expressions
Aggregation Expression
SortOrder Expression
Cost-Based Optimization
Cost-Based Optimization
BasicStatsPlanVisitor — Computing Statistics for Cost-Based Optimization
CatalogColumnStat
ColumnStat
CommandUtils
EstimationUtils
JoinEstimation
LogicalPlanStats — Statistics Estimates and Query Hints of Logical Operators
LogicalPlanVisitor
SizeInBytesOnlyStatsPlanVisitor — LogicalPlanVisitor for Total Size (in Bytes) Statistic Only
Statistics
Default Columns
Default Columns
Direct Queries on Files
Direct Queries on Files
Dynamic Partition Pruning
Dynamic Partition Pruning
File-Based Data Scanning
File-Based Data Scanning
Generated Columns
Generated Columns
GeneratedColumn
Hidden File Metadata
Hidden File Metadata
MetadataAttribute
Hints (SQL)
Hints (SQL)
HintErrorHandler
HintInfo
JoinHint
JoinStrategyHint
Join Strategy Hints
Join Queries
Join Queries
Join Queries
Broadcast Joins
Logging
Metadata Columns
Metadata Columns
FileSourceConstantMetadataAttribute
FileSourceGeneratedMetadataAttribute
FileSourceMetadataAttribute
MetadataColumnHelper
MetadataColumnsHelper
Named Function Arguments
Parameterized Queries
Parameterized Queries
Partition File Metadata Caching
Partition File Metadata Caching
Spark Connect
Spark Connect
CommandPlugin
ExpressionPlugin
RelationPlugin
SimpleSparkConnectService
SparkConnectAnalyzeHandler
SparkConnectInterceptorRegistry
SparkConnectPlanner
SparkConnectPlugin
SparkConnectServer
SparkConnectService
SparkConnectStreamHandler
Runtime Filtering
Runtime Filtering
Spark Thrift Server
Spark Thrift Server
SparkSQLEnv
Statistics
Subexpression Elimination
Subexpression Elimination
EquivalentExpressions
Subqueries
Subqueries
Table-Valued Functions
Table-Valued Functions
Time Travel
Time Travel
TimeTravelSpec
Transactional Writes
Transactional Writes
SQLHadoopMapReduceCommitProtocol
User-Defined Functions
User-Defined Functions
UDFRegistration
UserDefinedPythonFunction
Vectorized Decoding
Vectorized Decoding
ColumnVector
OffHeapColumnVector
OnHeapColumnVector
WritableColumnVector
ANSI Intervals
Catalog Plugin API and Multi-Catalog Support
Explaining Query Plans Improved
Observable Metrics
Hive Integration
Dynamic Partition Inserts
Vectorized Query Execution
Vectorized Query Execution
ColumnarBatch
Whole-Stage Code Generation
Whole-Stage Code Generation
Block
BufferedRowIterator
CodeGenerator
CodegenContext
GenerateColumnAccessor
GenerateMutableProjection
GenerateOrdering
GeneratePredicate
GenerateSafeProjection
GenerateUnsafeProjection
Catalyst DSL
Catalyst DSL
DslLogicalPlan
Variable Substitution
Query Execution
Query Execution
Catalyst
Catalyst
GenericStrategy
Optimizer
PlanChangeLogger
QueryPlan
QueryPlanner
Rule
RuleExecutors
TreeNode
TreePattern
TreePatternBits
Catalyst Expressions
Catalyst Expressions
AggregateExpression
AggregateFunction
AggregateWindowFunction Expressions
Aggregator Expressions
ArrayFilter Expression
AttributeSeq
Attribute Expressions
BasePredicate Expressions
BinaryComparison Expressions
BinaryOperator
BloomFilterAggregate
BloomFilterMightContain
BoundReference
CallMethodViaReflection
CodeGeneratorWithInterpretedFallback
CodegenFallback Expressions
Collect Expressions
CollectSet Expression
Count Expression
CreateNamedStruct
CreateStruct Function Builder
CumeDist
DeclarativeAggregate
DecodeUsingSerializer
DynamicPruningExpression Unary Expression
DynamicPruningSubquery Unevaluable Subquery Unary Expression
EncodeUsingSerializer
EqualNullSafe
EqualTo Predicate Expression
ExecSubqueryExpression Expressions
Exists — Correlated Predicate Subquery Expression
ExpectsInputTypes Expressions
ExplodeBase
Expression
First Aggregate Function Expression
Generator
HashExpression
HashPartitioning
Higher-Order Functions
Higher-Order Functions
HigherOrderFunction
ArrayBasedSimpleHigherOrderFunction
MapBasedSimpleHigherOrderFunction
SimpleHigherOrderFunction
ImperativeAggregate Expressions
In
Inline
InSet
InSubquery
InSubqueryExec
InterpretedProjection
JsonToStructs
LessThanOrEqual
ListQuery
Literal
MonotonicallyIncreasingID
MaxBy
Murmur3Hash
MutableProjection Expression
NamedExpression
Nondeterministic Expressions
OffsetWindowFunction Expressions
ParseToDate
ParseToTimestamp
PlanExpression Expressions
Predicate Expressions
Projection Functions
PythonUDF
RowNumber
RowNumberLike
RowOrdering
RuntimeReplaceable
ScalaAggregator
ScalarSubquery
ScalarSubquery (ExecSubqueryExpression)
ScalaUDAF
ScalaUDF Expression
SimpleTypedAggregateExpression
SortOrder
SparkUserDefinedFunction
Stateful
StaticInvoke
SubqueryExpression
TimeWindow
TypedImperativeAggregate Expressions
UnaryExpression
Unevaluable
UnixTimestamp
UnresolvedAttribute
UnresolvedFunction
UnresolvedGenerator
UnresolvedOrdinal
UnresolvedStar
UnsafeProjection
UserDefinedAggregator
User-Defined Functions
User-Defined Functions
User-Defined Functions
UserDefinedExpression
UserDefinedFunction
UDFs are Blackbox — Don't Use Them Unless You've Got No Choice
UserDefinedAggregateFunction — User-Defined Untyped Aggregate Functions (UDAFs)
WindowExpression
WindowFunction Expressions
WindowSpecDefinition
Execution Planning Strategies
Execution Planning Strategies
Aggregation
BasicOperators
DataSourceStrategy
DataSourceV2Strategy
FileSourceStrategy
InMemoryScans
JoinSelection
LogicalQueryStageStrategy
SparkStrategies
SparkStrategy
SpecialLimits
Window
WithCTEStrategy
Logical Query Plan Analyzer
Logical Analysis Rules
Logical Analysis Rules
AddMetadataColumns
AliasViewChild
BindParameters
CTESubstitution
CleanupAliases
DataSourceAnalysis
ExtractWindowExpressions
FindDataSourceTable
LookupFunctions
PreWriteCheck
PreprocessTableCreation
ResolveAggregateFunctions
ResolveAliases
ResolveCatalogs
ResolveCoalesceHints
ResolveCreateNamedStruct
ResolveDefaultColumns
ResolveFunctions
ResolveGroupingAnalytics
ResolveInlineTables
ResolveInsertInto
ResolveJoinStrategyHints
ResolveOrdinalInOrderByAndGroupBy
ResolveReferences
ResolveRelations
ResolveSQLOnFile
ResolveSessionCatalog
ResolveSubquery
ResolveWindowFrame
ResolveWithCTE
RewriteDeleteFromTable
RewriteRowLevelCommand
TableCapabilityCheck
WindowFrameCoercion
WindowsSubstitution
Logical Operators
Logical Operators
AddColumns
Aggregate
AlterTable
AlterTableAddColumnsCommand
AlterTableCommand
AnalysisOnlyCommand
AnalyzeColumn
AnalyzeColumnCommand
AnalyzePartitionCommand
AnalyzeTable
AnalyzeTableCommand
AnalyzeTablesCommand
AppendData
ArrowEvalPython
BaseEvalPython
CTERelationDef
CTERelationRef
CacheTableCommand
ClearCacheCommand
CollectMetrics
Command
CommentOnTable
CreateDataSourceTableAsSelectCommand
CreateDataSourceTableCommand
CreateTable
CreateTableAsSelect
CreateTempViewUsing
CreateView
CreateViewCommand
DataSourceV2Relation
DataSourceV2ScanRelation
DataWritingCommand
DeleteFromTable
DescribeColumnCommand
DescribeRelation
DescribeTableCommand
DeserializeToObject
DropNamespace
Except
Expand
ExplainCommand
ExposesMetadataColumns
ExternalRDD
FlatMapGroupsWithState
Generate
GlobalLimit
GroupingSets
IgnoreCachedData
InMemoryRelation
InsertIntoDataSourceCommand
InsertIntoDir
InsertIntoHadoopFsRelationCommand
InsertIntoStatement
InsertIntoTable
Intersect
Join
LeafNodes
LeafRunnableCommand
LoadDataCommand
LocalRelation
LogicalPlan
LogicalPlanDistinctKeys
LogicalQueryStage
LogicalRDD
LogicalRelation
MapPartitions
MergeIntoTable
MultiInstanceRelation
NameParameterizedQuery
NamedRelation
Offset
OrderPreservingUnaryNode
OverwriteByExpression
OverwritePartitionsDynamic
ParameterizedQuery
ParsedStatement
Pivot
Project
RebalancePartitions
RelationTimeTravel
Repartition
RepartitionByExpression
RepartitionOperation
ReplaceData
ResolvedHint
ResolvedTable
RowLevelWrite
RunnableCommand
SaveIntoDataSourceCommand
SetCatalogAndNamespace
ShowColumns
ShowColumnsCommand
ShowCreateTable
ShowCreateTableCommand
ShowTableProperties
ShowTablePropertiesCommand
ShowTables
Sort
SubqueryAlias
SupportsSubquery
TruncateTableCommand
Union
UnresolvedHaving
UnresolvedHint
UnresolvedRelation
UnresolvedTable
UnresolvedTableOrView
UnresolvedTableValuedFunction
UnresolvedWith
UpdateTable
V1WriteCommand
V2CreateTablePlan
V2WriteCommand
View
Window
WithCTE
WithWindowDefinition
WriteDelta
WriteFiles
Logical Optimizations
Logical Optimizations
AQEPropagateEmptyRelation Adaptive Logical Optimization
CleanupDynamicPruningFilters Logical Optimization
CollapseWindow Logical Optimization
ColumnPruning Logical Optimization
CombineTypedFilters Logical Optimization
CombineUnions Logical Optimization
ComputeCurrentTime Logical Optimization
ConstantFolding Logical Optimization
ConvertToLocalRelation Logical Optimization
CostBasedJoinReorder Logical Optimization -- Join Reordering in Cost-Based Optimization
DecimalAggregates Logical Optimization
DynamicJoinSelection Adaptive Logical Optimization
EliminateResolvedHint Logical Optimization
EliminateSerialization Logical Optimization
EliminateSubqueryAliases Logical Optimization
EliminateView Logical Optimization
ExtractPythonUDFFromAggregate Logical Optimization
ExtractPythonUDFs
GetCurrentDatabase Logical Optimization
GroupBasedRowLevelOperationScanPlanning Logical Optimization
InferFiltersFromConstraints Logical Optimization Rule
InjectRuntimeFilter
InlineCTE Logical Optimization
LimitPushDown Logical Optimization
NullPropagation Logical Optimization -- Nullability (NULL Value) Propagation
OptimizeIn Logical Optimization
OptimizeMetadataOnlyQuery Logical Optimization
OptimizeSubqueries Logical Optimization
PartitionPruning
PropagateEmptyRelation Logical Optimization
PruneFileSourcePartitions Logical Optimization
PruneFilters
PruneHiveTablePartitions Logical Optimization
PullupCorrelatedPredicates Logical Optimization
PushDownLeftSemiAntiJoin
PushDownOperatorsToDataSource Logical Optimization
PushDownPredicate Logical Optimization
PushDownPredicates Logical Optimization
PushPredicateThroughJoin Logical Optimization
ReorderJoin Logical Optimization -- Reordering Inner and Cross Joins
ReplaceExceptWithAntiJoin Logical Optimization Rule -- Rewriting Except (DISTINCT) Operators
ReplaceExceptWithFilter Logical Optimization Rule -- Rewriting Except (DISTINCT) Operators
ReplaceExpressions Logical Optimization
RewriteCorrelatedScalarSubquery Logical Optimization
RewriteExceptAll Logical Optimization Rule -- Rewriting Except (ALL) Operators
RewritePredicateSubquery Logical Optimization
SchemaPruning Logical Optimization
SimplifyCasts Logical Optimization
UpdateAttributeNullability Logical Optimization
UpdateCTERelationStats Logical Optimization
V2ScanRelationPushDown Logical Optimization
V2Writes Logical Optimization
Physical Operators
Physical Operators
AdaptiveSparkPlanExec Leaf Physical Operator
AggregateCodegenSupport Physical Operators
AliasAwareOutputExpression Physical Operators
AliasAwareQueryOutputOrdering Unary Physical Operators
AlterTableExec
AQEShuffleReadExec Physical Operator
AtomicTableWriteExec Physical Commands
BaseAggregateExec Unary Physical Operators
BaseJoinExec Physical Operators
BaseSubqueryExec Physical Operators
BatchWriteHelper Physical Operators
BatchScanExec
BroadcastExchangeExec Unary Physical Operator for Broadcast Joins
BroadcastExchangeLike Physical Operators
BroadcastHashJoinExec Physical Operator
BroadcastNestedLoopJoinExec Binary Physical Operator
BroadcastQueryStageExec Physical Operator
CacheTableAsSelectExec
CoalesceExec
CollectLimitExec Physical Operator
CollectMetricsExec Physical Operator
ColumnarToRowExec Physical Operator
ColumnarToRowTransition Unary Physical Operators
CreateTableAsSelectExec Physical Command
CodegenSupport
DataSourceScanExec Leaf Physical Operators
DataSourceV2ScanExecBase
DataWritingCommandExec Physical Operator
DebugExec Unary Physical Operator
DeleteFromTableExec
DescribeTableExec Physical Command
DeserializeToObjectExec
DropNamespaceExec
EvalPythonExec Physical Operators
Exchange Unary Physical Operators
ExecutedCommandExec Physical Operator
ExpandExec
ExternalRDDScanExec Leaf Physical Operator
FileSourceScanExec Physical Operator
FilterExec
GenerateExec
HashAggregateExec
HashedRelation
HashJoin — Hash-Based Join Physical Operators
InMemoryTableScanExec Leaf Physical Operator
InputAdapter Unary Physical Operator
JoinCodegenSupport
LocalTableScanExec Leaf Physical Operator
LongHashedRelation
ObjectConsumerExec -- Unary Physical Operators with Child Physical Operator with One-Attribute Output Schema
ObjectHashAggregateExec Physical Operator
ObjectProducerExec
OrderPreservingUnaryExecNode
OverwriteByExpressionExec Physical Command
PartitioningPreservingUnaryExecNode Unary Physical Operators
ProjectExec Unary Physical Operator
QueryStageExec Leaf Physical Operators
RangeExec Leaf Physical Operator
ReusedExchangeExec Leaf Physical Operator
ReusedSubqueryExec Physical Operator
RowDataSourceScanExec Leaf Physical Operator
RowToColumnarExec Physical Operator
SerializeFromObjectExec Unary Physical Operator
SetCatalogAndNamespaceExec Physical Command
ShowCreateTableExec Physical Command
ShowTablesExec Physical Command
ShowTablePropertiesExec Physical Command
ShuffleExchangeExec
ShuffleExchangeLike Physical Operators
ShuffledHashJoinExec Physical Operator
ShuffledJoin Physical Operators
ShuffleOrigin
ShuffleQueryStageExec Adaptive Leaf Physical Operator
SortAggregateExec
SortMergeJoinExec
SortMergeJoinExec
SortMergeJoinExec Physical Operator
SortMergeJoinScanner
SortExec Physical Operator
SparkPlan
SubqueryExec Unary Physical Operator
TableWriteExecHelper Unary Physical Commands
TruncateTableExec Physical Operator
UnaryExecNode
V2CommandExec
V2ExistingTableWriteExec
V2TableWriteExec
WholeStageCodegenExec Physical Operator
WindowExec
WindowExecBase
WriteDeltaExec
WriteFilesExec
Distribution and Partitioning
Distribution and Partitioning
Distribution
Partitioning (Catalyst)
Distribution Specifications
Distribution Specifications
AllTuples
BroadcastDistribution
ClusteredDistribution
HashClusteredDistribution
OrderedDistribution
UnspecifiedDistribution
Broadcast Modes
Broadcast Modes
BroadcastModes
HashedRelationBroadcastMode
ShuffleSpec
Physical Optimizations
Physical Optimizations
AQEShuffleReadRule
AdjustShuffleExchangePosition
ApplyColumnarRulesAndInsertTransitions
CoalesceBucketsInJoin
CoalesceShufflePartitions
CollapseCodegenStages
DisableUnnecessaryBucketedScan
EnsureRequirements
InsertAdaptiveSparkPlan
OptimizeShuffleWithLocalRead
OptimizeSkewInRebalancePartitions
OptimizeSkewedJoin
PlanAdaptiveDynamicPruningFilters
PlanAdaptiveSubqueries
PlanDynamicPruningFilters
PlanSubqueries
RemoveRedundantProjects
RemoveRedundantSorts
ReplaceHashWithSortAgg
ReuseAdaptiveSubquery
ReuseExchange
ReuseExchangeAndSubquery
ReuseSubquery
ValidateSparkPlan
QueryExecution
Structured Query Execution Pipeline
QueryPlanningTracker
SparkOptimizer — Logical Query Plan Optimizer
SparkPlanner — Spark Query Planner
Internals
Internals
Spark SQL
DataSource — Pluggable Data Provider Framework
Developer API
ExecutionListenerBus
ExecutionListenerManager
SharedState — State Shared Across SparkSessions
SQLConf
SQLConfHelper
StaticSQLConf — Static Configuration Properties
SparkSession Registries
SparkSession Registries
Catalog
Catalog
Catalog — Metastore Management Interface
CatalogImpl
CatalogStatistics
CatalogUtils
ExperimentalMethods
ExternalCatalog
ExternalCatalog
ExternalCatalog
InMemoryCatalog
ExternalCatalogWithListener
FunctionRegistry
FunctionRegistry
FunctionRegistry
FunctionRegistryBase
SimpleFunctionRegistry
SimpleFunctionRegistryBase
SimpleTableFunctionRegistry
TableFunctionRegistry
GlobalTempViewManager
SessionCatalog
SessionCatalog
SessionCatalog
CatalogStorageFormat
CatalogTable
CatalogTablePartition
V2SessionCatalog
SessionState
SessionState
SessionState
BaseSessionStateBuilder
SessionStateBuilder
CacheManager
CacheManager
CacheManager
RuntimeConfig
Encoder
Encoder
Encoder
ExpressionEncoder
RowEncoder
ScalaReflection
SQLExecution
SQLMetric
Tungsten Execution Backend
Tungsten Execution Backend
CatalystSerde Utility
ExternalAppendOnlyUnsafeRowArray
HashMapGenerator
InternalRow
UnsafeHashedRelation
UnsafeRow
UnsafeRowSerializerInstance
RDDs
RDDs
DataSourceRDD
DataSourceRDDPartition
FileScanRDD
ShuffledRowRDD
SQLExecutionRDD
SQL
SQL
AbstractSqlParser
AstBuilder
ANTLR-based SQL Parser
CatalystSqlParser
ParserInterface
SparkSqlParser — Default SQL Parser
SparkSqlAstBuilder
VariableSubstitution
Connectors
Connectors
Avro
Avro
AvroOptions
AvroFileFormat
CatalystDataToAvro Unary Expression
AvroDataToCatalyst Unary Expression
Files
Files
AggregatePushDownUtils
BaseDynamicPartitionDataWriter
BasicWriteJobStatsTracker
BasicWriteTaskStats
BasicWriteTaskStatsTracker
CompressionCodecs
DynamicPartitionDataSingleWriter
DynamicPartitionDataConcurrentWriter
FileBatchWrite
FileDataSourceV2 Table Providers
FileFormat
FileFormatDataWriter
FileFormatWriter
FileIndex
FileIndex
FileIndex
CatalogFileIndex
InMemoryFileIndex
PartitioningAwareFileIndex
PrunedInMemoryFileIndex
FilePartition
FilePartitionReaderFactory
FileScan
FileScanBuilder
FileStatusCache
FileTable
FileWrite
FileWriterFactory
HadoopFileLinesReader
HadoopFsRelation
PartitionedFile
PartitionedFileUtil
RecordReaderIterator
SharedInMemoryCache
SchemaMergeUtils
SingleDirectoryDataWriter
WriteJobStatsTracker
WriteTaskStats
WriteTaskStatsTracker
Hive
Hive
Configuration Properties
Hive Metastore
DataSinks
HiveClient
HiveClientImpl
HiveFileFormat
HiveUtils
IsolatedClientLoader Utility
HiveTableRelation Leaf Logical Operator
CreateHiveTableAsSelectCommand Logical Command
SaveAsHiveFile
InsertIntoHiveDirCommand Logical Command
InsertIntoHiveTable
HiveTableScans
HiveTableScanExec Leaf Physical Operator
TableReader
HadoopTableReader
HiveSessionStateBuilder
HiveExternalCatalog
HiveSessionCatalog -- Hive-Specific Catalog of Relational Entities
HiveMetastoreCatalog — Legacy SessionCatalog for Converting Hive Metastore Relations to Data Source Relations
RelationConversions
ResolveHiveSerdeTable Logical Resolution Rule
DetermineTableStats Logical PostHoc Resolution Rule -- Computing Total Size Table Statistic for HiveTableRelations
HiveAnalysis PostHoc Logical Resolution Rule
JDBC
JDBC
AggregatedDialect
JDBCOptions
JDBCRDD
JDBCRelation
JDBCScan
JDBCScanBuilder
JDBCTableCatalog
JdbcDialect
JdbcDialects
JdbcRelationProvider
JdbcUtils
Kafka
Kafka
Configuration Properties
KafkaBatch
KafkaBatchWrite
KafkaBatchWriterFactory
KafkaDataConsumer
KafkaDataWriter
KafkaOffsetRangeLimit
KafkaOffsetReader
KafkaRelation
KafkaScan
KafkaSourceProvider
KafkaSourceRDD
KafkaSourceRDDPartition
KafkaTable
KafkaWrite
KafkaWriter Utility
KafkaWriteTask
Misc
Misc
ConsumerStrategy — Kafka Consumer Providers
InternalKafkaConsumer
InternalKafkaProducerPool
JsonUtils
KafkaRowWriter
KafkaRecordToRowConverter
Options
Noop
Noop
NoopDataSource
NoopTable
NoopWriteBuilder
NoopBatchWrite
NoopStreamingWrite
Parquet
Parquet
ParquetDataSourceV2
ParquetFileFormat
ParquetFilters
ParquetOptions
ParquetPartitionReaderFactory
ParquetReadSupport
ParquetScan
ParquetScanBuilder
ParquetTable
ParquetUtils
ParquetWrite
ParquetWriteSupport
SparkToParquetSchemaConverter
SpecificParquetRecordReaderBase — Hadoop RecordReader
VectorizedColumnReader
VectorizedParquetRecordReader
DataWritingSparkTask Utility
DataSourceV2Utils Utility
OutputWriter
High-Level APIs
High-Level APIs
Column
ColumnarRule
Connector API
Connector API
ApplyTransform
Batch
BatchWrite
CustomMetric
DataSourceV2Implicits
DataWriter
DataWriterFactory
Expression
InputPartition
LocalScan
OptionsHelper
PartitionReader
PartitionReaderFactory
PartitionSpecsHelper
Partitioning
Predicate
RewritableTransform
RowLevelOperation
Scan
ScanBuilder
SessionConfigSupport
SimpleTableProvider
StagedTable
SupportsAtomicPartitionManagement
SupportsDelete
SupportsDeleteV2
SupportsDelta
SupportsDynamicOverwrite
SupportsMetadata
SupportsMetadataColumns
SupportsOverwrite
SupportsPartitionManagement
SupportsPushDownAggregates
SupportsPushDownFilters
SupportsPushDownRequiredColumns
SupportsPushDownV2Filters
SupportsRead
SupportsReportOrdering
SupportsReportPartitioning
SupportsReportStatistics
SupportsRowLevelOperations
SupportsRuntimeFiltering
SupportsRuntimeV2Filtering
SupportsStreamingUpdate
SupportsTruncate
SupportsWrite
Table
TableCapability
TableHelper
TableProvider
Transform
TransformHelper
TruncatableTable
V1Scan
V1Table
V1WriteBuilder
Write
WriteBuilder
Data Types
Data Types
AbstractDataType
ArrayType
AtomicType
CalendarInterval
DataType
Metadata
MetadataBuilder
StructField
StructType
UserDefinedType
Dataset
DataFrame
DataFrameReader
DataFrameWriter
DataFrameWriterV2
DataSource V1 API
DataSource V1 API
DataSourceRegister
CreatableRelationProvider
RelationProvider — Relations with Pre-Defined Schema
SchemaRelationProvider — Relation Providers With Mandatory User-Defined Schema
BaseRelation — Collection of Tuples with Schema
FileRelation
InsertableRelation
PrunedFilteredScan
PrunedScan
TableScan — Relations with Column Pruning
Data Source Filter Predicate
Encoders Utility
KeyValueGroupedDataset
Observation
QueryExecutionListener
RelationalGroupedDataset
SparkSession
The Entry Point to Spark SQL
SparkSession.Builder
SparkSessionExtensions
Standard Functions
org.apache.spark.sql.functions
Standard Functions
Aggregate
Collection
Date time
Non-aggregate (Normal)
Window
TypedColumn
Window Functions
Window Functions
AggregateProcessor
RangeFrame
Window
WindowFunctionFrame
WindowSpec
Web UI
Web UI
AllExecutionsPage
ExecutionPage
SQLAppStatusListener
SQLAppStatusStore
SQLTab
SparkListenerSQLExecutionEnd
Demo
Demo
Adaptive Query Execution
Connecting Spark SQL to Hive Metastore
Mult-Dimensional Aggregations
Developing CatalogPlugin
Dynamic Partition Pruning
Hive Partitioned Parquet Table and Partition Pruning
ObjectHashAggregateExec and Sort-Based Fallback Tasks
Spilling
Using JDBC Data Source to Access PostgreSQL
Misc
Misc
AggregatingAccumulator
DistinctKeyVisitor
FilterEvaluatorFactory
JoinSelectionHelper
PushDownUtils
UnsafeExternalRowSorter
BindReferences
IntervalUtils
ExplainUtils
SerializerBuildHelper
Datasets, DataFrames and RDDs
Dataset API and SQL
DDLUtils
implicits Object -- Implicits Conversions
Row
Data Source API
Data Source API
CreateTableWriter
WriteConfigMethods
Dataset API
Dataset API
Operators
Dataset API — Typed Transformations
Dataset API — Untyped Transformations
Basic Actions
Actions
DataFrameNaFunctions
DataFrameStatFunctions
Column Operators
Caching and Persistence
Caching and Persistence
Caching and Persistence
User-Friendly Names of Cached Queries in web UI
Checkpointing
Performance Tuning and Debugging
Performance Tuning and Debugging
Debugging Query Execution
Spark SQL's Performance Tuning Tips and Tricks (aka Case Studies)
Case Study: Number of Partitions for groupBy Aggregation
CheckAnalysis — Analysis Validation
CatalystTypeConverters Helper Object
SubExprUtils Utility
PredicateHelper
ExtractEquiJoinKeys Scala Extractor
ExtractSingleColumnNullAwareAntiJoin Scala Extractor
ExtractJoinWithBuckets Scala Extractor
PhysicalOperation
KnownSizeEstimation
Spark SQL
Query Execution
Logical Operators
AnalysisOnlyCommand Logical Operators
¶
AnalysisOnlyCommand
is...FIXME
Back to top