Skip to content
The Internals of Spark SQL
ValidateSparkPlan
Initializing search
spark-sql-internals
Spark SQL
Features
Query Execution
Internals
SQL
Connectors
High-Level APIs
Web UI
Demo
Misc
The Internals of Spark SQL
spark-sql-internals
Spark SQL
Features
Features
Aggregate Queries
Aggregate Queries
AggUtils
AggregationIterator
KVSorterIterator
ObjectAggregationIterator
ObjectAggregationMap
PhysicalAggregation
SortBasedAggregationIterator
TungstenAggregationIterator
UnsafeFixedWidthAggregationMap
UnsafeKVExternalSorter
Adaptive Query Execution
Adaptive Query Execution
AQEOptimizer
AQEUtils
AdaptiveExecutionContext
AdaptiveRulesHolder
CostEvaluator
ShufflePartitionsUtil
SimpleCostEvaluator
Bloom Filter Join
Bloom Filter Join
BloomFilter
BloomFilterImpl
Bucketing
Bucketing
BucketSpec
Cache Serialization
Cache Serialization
CachedBatch
CachedBatchSerializer
CachedRDDBuilder
DefaultCachedBatchSerializer
SimpleMetricsCachedBatch
SimpleMetricsCachedBatchSerializer
Catalog Plugin API
Catalog Plugin API
CatalogExtension
CatalogHelper
CatalogManager
CatalogPlugin
Catalogs
CatalogV2Util
Column
DelegatingCatalogExtension
FunctionCatalog
MetadataColumn
StagingTableCatalog
SupportsNamespaces
SupportsCatalogOptions
TableCatalog
TableCatalogCapability
TableChange
V2TableWithV1Fallback
Columnar Execution
Columnar Execution
Common Table Expressions
Common Table Expressions
Configuration Properties
Connector Expressions
Connector Expressions
Aggregation
SortOrder
Cost-Based Optimization
Cost-Based Optimization
BasicStatsPlanVisitor
CatalogColumnStat
ColumnStat
CommandUtils
EstimationUtils
JoinEstimation
LogicalPlanStats
LogicalPlanVisitor
SizeInBytesOnlyStatsPlanVisitor
Statistics
Default Columns
Default Columns
Direct Queries on Files
Direct Queries on Files
Dynamic Partition Pruning
Dynamic Partition Pruning
File-Based Data Scanning
File-Based Data Scanning
Generated Columns
Generated Columns
GeneratedColumn
Hidden File Metadata
Hidden File Metadata
MetadataAttribute
Hints (SQL)
Hints (SQL)
HintErrorHandler
HintInfo
JoinHint
JoinStrategyHint
Join Strategy Hints
Join Queries
Join Queries
Joins
Broadcast Joins
Logging
Metadata Columns
Metadata Columns
FileSourceConstantMetadataAttribute
FileSourceGeneratedMetadataAttribute
FileSourceMetadataAttribute
MetadataColumnHelper
MetadataColumnsHelper
Named Function Arguments
Parameterized Queries
Parameterized Queries
Partition File Metadata Caching
Partition File Metadata Caching
Spark Connect
Spark Connect
CommandPlugin
ExpressionPlugin
RelationPlugin
SimpleSparkConnectService
SparkConnectAnalyzeHandler
SparkConnectInterceptorRegistry
SparkConnectPlanner
SparkConnectPlugin
SparkConnectServer
SparkConnectService
SparkConnectStreamHandler
Runtime Filtering
Runtime Filtering
Spark Thrift Server
Spark Thrift Server
SparkSQLEnv
Statistics
Subexpression Elimination
Subexpression Elimination
EquivalentExpressions
Subqueries
Subqueries
Table-Valued Functions
Table-Valued Functions
Time Travel
Time Travel
TimeTravelSpec
Transactional Writes
Transactional Writes
SQLHadoopMapReduceCommitProtocol
User-Defined Functions
User-Defined Functions
UDFRegistration
UserDefinedPythonFunction
Vectorized Decoding
Vectorized Decoding
ColumnVector
OffHeapColumnVector
OnHeapColumnVector
WritableColumnVector
ANSI Intervals
Catalog Plugin API and Multi-Catalog Support
Explaining Query Plans Improved
Observable Metrics
Hive Integration
Dynamic Partition Inserts
Vectorized Query Execution
Vectorized Query Execution
ColumnarBatch
Whole-Stage Code Generation
Whole-Stage Code Generation
Block
BufferedRowIterator
CodeGenerator
CodegenContext
GenerateColumnAccessor
GenerateMutableProjection
GenerateOrdering
GeneratePredicate
GenerateSafeProjection
GenerateUnsafeProjection
Catalyst DSL
Catalyst DSL
DslLogicalPlan
Variable Substitution
Query Execution
Query Execution
Catalyst
Catalyst
GenericStrategy
Optimizer
PlanChangeLogger
QueryPlan
QueryPlanner
Rule
RuleExecutors
TreeNode
TreePattern
TreePatternBits
Catalyst Expressions
Catalyst Expressions
AggregateExpression
AggregateFunction
AggregateWindowFunction
Aggregator
ArrayFilter
AttributeSeq
Attribute
BasePredicate
BinaryComparison
BinaryOperator
BloomFilterAggregate
BloomFilterMightContain
BoundReference
CallMethodViaReflection
CodeGeneratorWithInterpretedFallback
CodegenFallback
Collect
CollectSet
Count
CreateNamedStruct
CreateStruct
CumeDist
DeclarativeAggregate
DecodeUsingSerializer
DynamicPruningExpression
DynamicPruningSubquery
EncodeUsingSerializer
EqualNullSafe
EqualTo
ExecSubqueryExpression
Exists
ExpectsInputTypes
ExplodeBase
Expression
First
Generator
HashExpression
HashPartitioning
Higher-Order Functions
Higher-Order Functions
HigherOrderFunction
ArrayBasedSimpleHigherOrderFunction
MapBasedSimpleHigherOrderFunction
SimpleHigherOrderFunction
ImperativeAggregate
In
Inline
InSet
InSubquery
InSubqueryExec
InterpretedProjection
JsonToStructs
LessThanOrEqual
ListQuery
Literal
MonotonicallyIncreasingID
MaxBy
Murmur3Hash
MutableProjection
NamedExpression
Nondeterministic
OffsetWindowFunction
ParseToDate
ParseToTimestamp
PlanExpression
Predicate
Projection
PythonUDF
RowNumber
RowNumberLike
RowOrdering
RuntimeReplaceable
ScalaAggregator
ScalarSubquery
ScalarSubquery (ExecSubqueryExpression)
ScalaUDAF
ScalaUDF
SimpleTypedAggregateExpression
SortOrder
SparkUserDefinedFunction
Stateful
StaticInvoke
SubqueryExpression
TimeWindow
TypedImperativeAggregate
UnaryExpression
Unevaluable
UnixTimestamp
UnresolvedAttribute
UnresolvedFunction
UnresolvedGenerator
UnresolvedOrdinal
UnresolvedStar
UnsafeProjection
UserDefinedAggregator
User-Defined Functions
User-Defined Functions
User-Defined Functions
UserDefinedExpression
UserDefinedFunction
UDFs are Blackbox
UserDefinedAggregateFunction
WindowExpression
WindowFunction
WindowSpecDefinition
Execution Planning Strategies
Execution Planning Strategies
Aggregation
BasicOperators
DataSourceStrategy
DataSourceV2Strategy
FileSourceStrategy
InMemoryScans
JoinSelection
LogicalQueryStageStrategy
SparkStrategies
SparkStrategy
SpecialLimits
Window
WithCTEStrategy
Logical Analyzer
Logical Analysis Rules
Logical Analysis Rules
AddMetadataColumns
AliasViewChild
BindParameters
CTESubstitution
CleanupAliases
DataSourceAnalysis
ExtractWindowExpressions
FindDataSourceTable
LookupFunctions
PreWriteCheck
PreprocessTableCreation
ResolveAggregateFunctions
ResolveAliases
ResolveCatalogs
ResolveCoalesceHints
ResolveCreateNamedStruct
ResolveDefaultColumns
ResolveFunctions
ResolveGroupingAnalytics
ResolveInlineTables
ResolveInsertInto
ResolveJoinStrategyHints
ResolveOrdinalInOrderByAndGroupBy
ResolveReferences
ResolveRelations
ResolveSQLOnFile
ResolveSessionCatalog
ResolveSubquery
ResolveWindowFrame
ResolveWithCTE
RewriteDeleteFromTable
RewriteRowLevelCommand
TableCapabilityCheck
WindowFrameCoercion
WindowsSubstitution
Logical Operators
Logical Operators
AddColumns
Aggregate
AlterTable
AlterTableAddColumnsCommand
AlterTableCommand
AnalysisOnlyCommand
AnalyzeColumn
AnalyzeColumnCommand
AnalyzePartitionCommand
AnalyzeTable
AnalyzeTableCommand
AnalyzeTablesCommand
AppendData
ArrowEvalPython
BaseEvalPython
CTERelationDef
CTERelationRef
CacheTableCommand
ClearCacheCommand
CollectMetrics
Command
CommentOnTable
CreateDataSourceTableAsSelectCommand
CreateDataSourceTableCommand
CreateTable
CreateTableAsSelect
CreateTempViewUsing
CreateView
CreateViewCommand
DataSourceV2Relation
DataSourceV2ScanRelation
DataWritingCommand
DeleteFromTable
DescribeColumnCommand
DescribeRelation
DescribeTableCommand
DeserializeToObject
Except
Expand
ExplainCommand
ExposesMetadataColumns
ExternalRDD
FlatMapGroupsWithState
Generate
GlobalLimit
GroupingSets
IgnoreCachedData
InMemoryRelation
InsertIntoDataSourceCommand
InsertIntoDir
InsertIntoHadoopFsRelationCommand
InsertIntoStatement
InsertIntoTable
Join
LeafNodes
LeafRunnableCommand
LoadDataCommand
LocalRelation
LogicalPlan
LogicalPlanDistinctKeys
LogicalQueryStage
LogicalRDD
LogicalRelation
MapPartitions
MergeIntoTable
MultiInstanceRelation
NameParameterizedQuery
NamedRelation
Offset
OrderPreservingUnaryNode
OverwriteByExpression
OverwritePartitionsDynamic
ParameterizedQuery
ParsedStatement
Pivot
Project
RebalancePartitions
RelationTimeTravel
Repartition
RepartitionByExpression
RepartitionOperation
ReplaceData
ResolvedHint
ResolvedTable
RowLevelWrite
RunnableCommand
SaveIntoDataSourceCommand
SetCatalogAndNamespace
ShowColumns
ShowColumnsCommand
ShowCreateTable
ShowCreateTableCommand
ShowTableProperties
ShowTablePropertiesCommand
ShowTables
Sort
SubqueryAlias
SupportsSubquery
TruncateTableCommand
UnresolvedHaving
UnresolvedHint
UnresolvedRelation
UnresolvedTable
UnresolvedTableOrView
UnresolvedTableValuedFunction
UnresolvedWith
UpdateTable
V2CreateTablePlan
V2WriteCommand
View
Window
WithCTE
WithWindowDefinition
WriteDelta
WriteFiles
Logical Optimizations
Logical Optimizations
AQEPropagateEmptyRelation Adaptive Logical Optimization
CleanupDynamicPruningFilters Logical Optimization
CollapseWindow Logical Optimization
ColumnPruning Logical Optimization
CombineTypedFilters Logical Optimization
CombineUnions Logical Optimization
ComputeCurrentTime Logical Optimization
ConstantFolding Logical Optimization
ConvertToLocalRelation Logical Optimization
CostBasedJoinReorder Logical Optimization -- Join Reordering in Cost-Based Optimization
DecimalAggregates Logical Optimization
DynamicJoinSelection Adaptive Logical Optimization
EliminateResolvedHint Logical Optimization
EliminateSerialization Logical Optimization
EliminateSubqueryAliases Logical Optimization
EliminateView Logical Optimization
ExtractPythonUDFFromAggregate Logical Optimization
ExtractPythonUDFs
GetCurrentDatabase Logical Optimization
GroupBasedRowLevelOperationScanPlanning Logical Optimization
InferFiltersFromConstraints Logical Optimization Rule
InjectRuntimeFilter
InlineCTE Logical Optimization
LimitPushDown Logical Optimization
NullPropagation Logical Optimization -- Nullability (NULL Value) Propagation
OptimizeIn Logical Optimization
OptimizeMetadataOnlyQuery Logical Optimization
OptimizeSubqueries Logical Optimization
PartitionPruning
PropagateEmptyRelation Logical Optimization
PruneFileSourcePartitions Logical Optimization
PruneFilters
PruneHiveTablePartitions Logical Optimization
PullupCorrelatedPredicates Logical Optimization
PushDownLeftSemiAntiJoin
PushDownOperatorsToDataSource Logical Optimization
PushDownPredicate Logical Optimization
PushDownPredicates Logical Optimization
PushPredicateThroughJoin Logical Optimization
ReorderJoin Logical Optimization -- Reordering Inner and Cross Joins
ReplaceExceptWithAntiJoin Logical Optimization Rule -- Rewriting Except (DISTINCT) Operators
ReplaceExceptWithFilter Logical Optimization Rule -- Rewriting Except (DISTINCT) Operators
ReplaceExpressions Logical Optimization
RewriteCorrelatedScalarSubquery Logical Optimization
RewriteExceptAll Logical Optimization Rule -- Rewriting Except (ALL) Operators
RewritePredicateSubquery Logical Optimization
SchemaPruning Logical Optimization
SimplifyCasts Logical Optimization
UpdateAttributeNullability Logical Optimization
UpdateCTERelationStats Logical Optimization
V2ScanRelationPushDown Logical Optimization
V2Writes Logical Optimization
Physical Operators
Physical Operators
AdaptiveSparkPlanExec
AggregateCodegenSupport
AliasAwareOutputExpression
AliasAwareQueryOutputOrdering
AlterTableExec
AQEShuffleReadExec
AtomicTableWriteExec
BaseAggregateExec
BaseJoinExec
BaseSubqueryExec
BatchWriteHelper
BatchScanExec
BroadcastExchangeExec
BroadcastExchangeLike
BroadcastHashJoinExec
BroadcastNestedLoopJoinExec
BroadcastQueryStageExec
CacheTableAsSelectExec
CoalesceExec
CollectLimitExec
CollectMetricsExec
ColumnarToRowExec
ColumnarToRowTransition
CreateTableAsSelectExec
CodegenSupport
DataSourceScanExec
DataSourceV2ScanExecBase
DataWritingCommandExec
DebugExec
DeleteFromTableExec
DescribeTableExec
DeserializeToObjectExec
DropNamespaceExec
EvalPythonExec
Exchange
ExecutedCommandExec
ExpandExec
ExternalRDDScanExec
FileSourceScanExec
FilterExec
GenerateExec
HashAggregateExec
HashedRelation
HashJoin
InMemoryTableScanExec
InputAdapter
JoinCodegenSupport
LocalTableScanExec
LongHashedRelation
ObjectConsumerExec
ObjectHashAggregateExec
ObjectProducerExec
OrderPreservingUnaryExecNode
OverwriteByExpressionExec
PartitioningPreservingUnaryExecNode
ProjectExec
QueryStageExec
RangeExec
ReusedExchangeExec
ReusedSubqueryExec
RowDataSourceScanExec
RowToColumnarExec
SerializeFromObjectExec
SetCatalogAndNamespaceExec
ShowCreateTableExec
ShowTablesExec
ShowTablePropertiesExec
ShuffleExchangeExec
ShuffleExchangeLike
ShuffledHashJoinExec
ShuffledJoin
ShuffleOrigin
ShuffleQueryStageExec
SortAggregateExec
SortMergeJoinExec
SortMergeJoinExec
SortMergeJoinExec
SortMergeJoinScanner
SortExec
SparkPlan
SubqueryExec
TableWriteExecHelper
TruncateTableExec
UnaryExecNode
V2CommandExec
V2ExistingTableWriteExec
V2TableWriteExec
WholeStageCodegenExec
WindowExec
WindowExecBase
WriteDeltaExec
WriteFilesExec
Distribution and Partitioning
Distribution and Partitioning
Distribution
Partitioning
Distribution Specifications
Distribution Specifications
AllTuples
BroadcastDistribution
ClusteredDistribution
HashClusteredDistribution
OrderedDistribution
UnspecifiedDistribution
Broadcast Modes
Broadcast Modes
BroadcastMode
HashedRelationBroadcastMode
ShuffleSpec
Physical Optimizations
Physical Optimizations
AQEShuffleReadRule
AdjustShuffleExchangePosition
ApplyColumnarRulesAndInsertTransitions
CoalesceBucketsInJoin
CoalesceShufflePartitions
CollapseCodegenStages
DisableUnnecessaryBucketedScan
EnsureRequirements
InsertAdaptiveSparkPlan
OptimizeShuffleWithLocalRead
OptimizeSkewInRebalancePartitions
OptimizeSkewedJoin
PlanAdaptiveDynamicPruningFilters
PlanAdaptiveSubqueries
PlanDynamicPruningFilters
PlanSubqueries
RemoveRedundantProjects
RemoveRedundantSorts
ReplaceHashWithSortAgg
ReuseAdaptiveSubquery
ReuseExchange
ReuseExchangeAndSubquery
ReuseSubquery
ValidateSparkPlan
QueryExecution
QueryPlanningTracker
SparkOptimizer
SparkPlanner
Internals
Internals
Overview
DataSource
Developer API
ExecutionListenerBus
ExecutionListenerManager
SharedState
SQLConf
SQLConfHelper
StaticSQLConf
SparkSession Registries
SparkSession Registries
Catalog
Catalog
Catalog
CatalogImpl
CatalogStatistics
CatalogUtils
ExperimentalMethods
ExternalCatalog
ExternalCatalog
ExternalCatalog
InMemoryCatalog
ExternalCatalogWithListener
FunctionRegistry
FunctionRegistry
FunctionRegistry
FunctionRegistryBase
SimpleFunctionRegistry
SimpleFunctionRegistryBase
SimpleTableFunctionRegistry
TableFunctionRegistry
GlobalTempViewManager
SessionCatalog
SessionCatalog
SessionCatalog
CatalogStorageFormat
CatalogTable
CatalogTablePartition
V2SessionCatalog
SessionState
SessionState
SessionState
BaseSessionStateBuilder
SessionStateBuilder
CacheManager
CacheManager
CacheManager
RuntimeConfig
Encoder
Encoder
Encoder
ExpressionEncoder
RowEncoder
ScalaReflection
SQLExecution
SQLMetric
Tungsten Execution Backend
Tungsten Execution Backend
CatalystSerde
ExternalAppendOnlyUnsafeRowArray
HashMapGenerator
InternalRow
UnsafeHashedRelation
UnsafeRow
UnsafeRowSerializerInstance
RDDs
RDDs
DataSourceRDD
DataSourceRDDPartition
FileScanRDD
ShuffledRowRDD
SQLExecutionRDD
SQL
SQL
AbstractSqlParser
AstBuilder
CatalystSqlParser
ParserInterface
SparkSqlParser
SparkSqlAstBuilder
VariableSubstitution
Connectors
Connectors
Avro
Avro
AvroOptions
AvroFileFormat
CatalystDataToAvro
AvroDataToCatalyst
Files
Files
AggregatePushDownUtils
BaseDynamicPartitionDataWriter
BasicWriteJobStatsTracker
BasicWriteTaskStats
BasicWriteTaskStatsTracker
CompressionCodecs
DynamicPartitionDataSingleWriter
DynamicPartitionDataConcurrentWriter
FileBatchWrite
FileDataSourceV2
FileFormat
FileFormatDataWriter
FileFormatWriter
FileIndex
FileIndex
FileIndex
CatalogFileIndex
InMemoryFileIndex
PartitioningAwareFileIndex
PrunedInMemoryFileIndex
FilePartition
FilePartitionReaderFactory
FileScan
FileScanBuilder
FileStatusCache
FileTable
FileWrite
FileWriterFactory
HadoopFileLinesReader
HadoopFsRelation
PartitionedFile
PartitionedFileUtil
RecordReaderIterator
SharedInMemoryCache
SchemaMergeUtils
SingleDirectoryDataWriter
WriteJobStatsTracker
WriteTaskStats
WriteTaskStatsTracker
Hive
Hive
Configuration Properties
Hive Metastore
DataSinks
HiveClient
HiveClientImpl
HiveFileFormat
HiveUtils
IsolatedClientLoader
HiveTableRelation
CreateHiveTableAsSelectCommand
SaveAsHiveFile
InsertIntoHiveDirCommand
InsertIntoHiveTable
HiveTableScans
HiveTableScanExec
TableReader
HadoopTableReader
HiveSessionStateBuilder
HiveExternalCatalog
HiveSessionCatalog
HiveMetastoreCatalog
RelationConversions
ResolveHiveSerdeTable
DetermineTableStats
HiveAnalysis
JDBC
JDBC
AggregatedDialect
JDBCOptions
JDBCRDD
JDBCRelation
JDBCScan
JDBCScanBuilder
JDBCTableCatalog
JdbcDialect
JdbcDialects
JdbcRelationProvider
JdbcUtils
Kafka
Kafka
Configuration Properties
KafkaBatch
KafkaBatchWrite
KafkaBatchWriterFactory
KafkaDataConsumer
KafkaDataWriter
KafkaOffsetRangeLimit
KafkaOffsetReader
KafkaRelation
KafkaScan
KafkaSourceProvider
KafkaSourceRDD
KafkaSourceRDDPartition
KafkaTable
KafkaWrite
KafkaWriter
KafkaWriteTask
Misc
Misc
ConsumerStrategy
InternalKafkaConsumer
InternalKafkaProducerPool
JsonUtils
KafkaRowWriter
KafkaRecordToRowConverter
Options
Noop
Noop
NoopDataSource
NoopTable
NoopWriteBuilder
NoopBatchWrite
NoopStreamingWrite
Parquet
Parquet
ParquetDataSourceV2
ParquetFileFormat
ParquetFilters
ParquetOptions
ParquetPartitionReaderFactory
ParquetReadSupport
ParquetScan
ParquetScanBuilder
ParquetTable
ParquetUtils
ParquetWrite
ParquetWriteSupport
SparkToParquetSchemaConverter
SpecificParquetRecordReaderBase — Hadoop RecordReader
VectorizedColumnReader
VectorizedParquetRecordReader
DataWritingSparkTask
DataSourceV2Utils
OutputWriter
High-Level APIs
High-Level APIs
Column
ColumnarRule
Connector API
Connector API
ApplyTransform
Batch
BatchWrite
CustomMetric
DataSourceV2Implicits
DataWriter
DataWriterFactory
Expression
InputPartition
LocalScan
OptionsHelper
PartitionReader
PartitionReaderFactory
PartitionSpecsHelper
Partitioning
Predicate
RewritableTransform
RowLevelOperation
Scan
ScanBuilder
SessionConfigSupport
SimpleTableProvider
StagedTable
SupportsAtomicPartitionManagement
SupportsDelete
SupportsDeleteV2
SupportsDelta
SupportsDynamicOverwrite
SupportsMetadata
SupportsMetadataColumns
SupportsOverwrite
SupportsPartitionManagement
SupportsPushDownAggregates
SupportsPushDownFilters
SupportsPushDownRequiredColumns
SupportsPushDownV2Filters
SupportsRead
SupportsReportOrdering
SupportsReportPartitioning
SupportsReportStatistics
SupportsRowLevelOperations
SupportsRuntimeFiltering
SupportsRuntimeV2Filtering
SupportsStreamingUpdate
SupportsTruncate
SupportsWrite
Table
TableCapability
TableHelper
TableProvider
Transform
TransformHelper
TruncatableTable
V1Scan
V1Table
V1WriteBuilder
Write
WriteBuilder
Data Types
Data Types
AbstractDataType
ArrayType
AtomicType
CalendarInterval
DataType
Metadata
MetadataBuilder
StructField
StructType
UserDefinedType
Dataset
DataFrame
DataFrameReader
DataFrameWriter
DataFrameWriterV2
DataSource V1 API
DataSource V1 API
DataSourceRegister
CreatableRelationProvider
RelationProvider
SchemaRelationProvider
BaseRelation
FileRelation
InsertableRelation
PrunedFilteredScan
PrunedScan
TableScan
Filter
Encoders
KeyValueGroupedDataset
Observation
QueryExecutionListener
RelationalGroupedDataset
SparkSession
SparkSession.Builder
SparkSessionExtensions
Standard Functions
org.apache.spark.sql.functions
Standard Functions
Aggregate
Collection
Date time
Non-aggregate (Normal)
Window
TypedColumn
Window Functions
Window Functions
AggregateProcessor
RangeFrame
Window
WindowFunctionFrame
WindowSpec
Web UI
Web UI
AllExecutionsPage
ExecutionPage
SQLAppStatusListener
SQLAppStatusStore
SQLTab
SparkListenerSQLExecutionEnd
Demo
Demo
Adaptive Query Execution
Connecting Spark SQL to Hive Metastore
Mult-Dimensional Aggregations
Developing CatalogPlugin
Dynamic Partition Pruning
Hive Partitioned Parquet Table and Partition Pruning
ObjectHashAggregateExec and Sort-Based Fallback Tasks
Spilling
Using JDBC Data Source to Access PostgreSQL
Misc
Misc
AggregatingAccumulator
DistinctKeyVisitor
FilterEvaluatorFactory
JoinSelectionHelper
PushDownUtils
UnsafeExternalRowSorter
BindReferences
IntervalUtils
ExplainUtils
SerializerBuildHelper
Dataset, DataFrame and RDD
Dataset and SQL
DDLUtils
Implicits
Row
Data Source API
Data Source API
CreateTableWriter
WriteConfigMethods
Dataset API
Dataset API
Operators
Typed Transformations
Untyped Transformations
Basic Actions
Actions
DataFrameNaFunctions
DataFrameStatFunctions
Column Operators
Caching and Persistence
Caching and Persistence
Caching and Persistence
User-Friendly Names of Cached Queries in web UI
Checkpointing
Performance Tuning and Debugging
Performance Tuning and Debugging
Debugging Query Execution
Performance Tuning
Case Study
CheckAnalysis
CatalystTypeConverters
SubExprUtils
PredicateHelper
ExtractEquiJoinKeys
ExtractSingleColumnNullAwareAntiJoin
ExtractJoinWithBuckets
PhysicalOperation
KnownSizeEstimation
Spark SQL
Query Execution
Physical Optimizations
ValidateSparkPlan Physical Optimization
¶
ValidateSparkPlan
is...FIXME
Back to top