The Internals of Spark SQL
Metadata
Initializing search
    spark-sql-internals
    • Spark SQL
    • Features
    • Query Execution
    • Internals
    • SQL
    • Connectors
    • High-Level APIs
    • Web UI
    • Demo
    • Misc
    spark-sql-internals
    • Spark SQL
    • Features
      • Aggregate Queries
        • AggUtils
        • AggregationIterator
        • KVSorterIterator
        • ObjectAggregationIterator
        • ObjectAggregationMap
        • PhysicalAggregation
        • SortBasedAggregationIterator
        • TungstenAggregationIterator
        • UnsafeFixedWidthAggregationMap
        • UnsafeKVExternalSorter
      • Adaptive Query Execution
        • AQEOptimizer
        • AQEUtils
        • AdaptiveExecutionContext
        • AdaptiveRulesHolder
        • CostEvaluator
        • ShufflePartitionsUtil
        • SimpleCostEvaluator
      • Bloom Filter Join
        • BloomFilter
        • BloomFilterImpl
      • Bucketing
        • BucketSpec
      • Cache Serialization
        • CachedBatch
        • CachedBatchSerializer
        • CachedRDDBuilder
        • DefaultCachedBatchSerializer
        • SimpleMetricsCachedBatch
        • SimpleMetricsCachedBatchSerializer
      • Catalog Plugin API
        • CatalogExtension
        • CatalogHelper
        • CatalogManager
        • CatalogPlugin
        • Catalogs
        • CatalogV2Util
        • Column
        • DelegatingCatalogExtension
        • FunctionCatalog
        • MetadataColumn
        • StagingTableCatalog
        • SupportsNamespaces
        • SupportsCatalogOptions
        • TableCatalog
        • TableCatalogCapability
        • TableChange
        • V2TableWithV1Fallback
      • Columnar Execution
      • Common Table Expressions
      • Configuration Properties
      • Connector Expressions
        • Aggregation
        • SortOrder
      • Cost-Based Optimization
        • BasicStatsPlanVisitor
        • CatalogColumnStat
        • ColumnStat
        • CommandUtils
        • EstimationUtils
        • JoinEstimation
        • LogicalPlanStats
        • LogicalPlanVisitor
        • SizeInBytesOnlyStatsPlanVisitor
        • Statistics
      • Default Columns
      • Direct Queries on Files
      • Dynamic Partition Pruning
      • File-Based Data Scanning
      • Generated Columns
        • GeneratedColumn
      • Hidden File Metadata
        • MetadataAttribute
      • Hints (SQL)
        • HintErrorHandler
        • HintInfo
        • JoinHint
        • JoinStrategyHint
        • Join Strategy Hints
        • Joins
        • Broadcast Joins
      • Logging
      • Metadata Columns
        • FileSourceConstantMetadataAttribute
        • FileSourceGeneratedMetadataAttribute
        • FileSourceMetadataAttribute
        • MetadataColumnHelper
        • MetadataColumnsHelper
      • Named Function Arguments
      • Parameterized Queries
      • Partition File Metadata Caching
      • Runtime Filtering
      • Spark Connect
      • Spark Thrift Server
        • SparkSQLEnv
      • Statistics
      • Storage-Partitioned Joins
      • Subexpression Elimination
        • EquivalentExpressions
      • Subqueries
      • Table-Valued Functions
      • Time Travel
        • TimeTravelSpec
      • Transactional Writes
        • SQLHadoopMapReduceCommitProtocol
      • User-Defined Functions
        • UDFRegistration
        • UserDefinedPythonFunction
      • Vectorized Decoding
        • ColumnVector
        • OffHeapColumnVector
        • OnHeapColumnVector
        • WritableColumnVector
      • ANSI Intervals
      • Catalog Plugin API and Multi-Catalog Support
      • Explaining Query Plans Improved
      • Observable Metrics
      • Hive Integration
      • Dynamic Partition Inserts
      • Vectorized Query Execution
        • ColumnarBatch
      • Whole-Stage Code Generation
        • Block
        • BufferedRowIterator
        • CodeGenerator
        • CodegenContext
        • GenerateColumnAccessor
        • GenerateMutableProjection
        • GenerateOrdering
        • GeneratePredicate
        • GenerateSafeProjection
        • GenerateUnsafeProjection
      • Catalyst DSL
        • DslLogicalPlan
      • Variable Substitution
    • Query Execution
      • Catalyst
        • GenericStrategy
        • Optimizer
        • PlanChangeLogger
        • QueryPlan
        • QueryPlanner
        • Rule
        • RuleExecutors
        • TreeNode
        • TreePattern
        • TreePatternBits
      • Catalyst Expressions
        • AggregateExpression
        • AggregateFunction
        • AggregateWindowFunction
        • Aggregator
        • ArrayFilter
        • AttributeSeq
        • Attribute
        • BasePredicate
        • BinaryComparison
        • BinaryOperator
        • BloomFilterAggregate
        • BloomFilterMightContain
        • BoundReference
        • CallMethodViaReflection
        • CodeGeneratorWithInterpretedFallback
        • CodegenFallback
        • Collect
        • CollectSet
        • Count
        • CreateNamedStruct
        • CreateStruct
        • CumeDist
        • DeclarativeAggregate
        • DecodeUsingSerializer
        • DynamicPruningExpression
        • DynamicPruningSubquery
        • EncodeUsingSerializer
        • EqualNullSafe
        • EqualTo
        • ExecSubqueryExpression
        • Exists
        • ExpectsInputTypes
        • ExplodeBase
        • Expression
        • First
        • Generator
        • HashExpression
        • HashPartitioning
          • HigherOrderFunction
          • ArrayBasedSimpleHigherOrderFunction
          • MapBasedSimpleHigherOrderFunction
          • SimpleHigherOrderFunction
        • ImperativeAggregate
        • In
        • Inline
        • InSet
        • InSubquery
        • InSubqueryExec
        • InterpretedProjection
        • JsonToStructs
        • LessThanOrEqual
        • ListQuery
        • Literal
        • MonotonicallyIncreasingID
        • MaxBy
        • Murmur3Hash
        • MutableProjection
        • NamedExpression
        • Nondeterministic
        • OffsetWindowFunction
        • ParseToDate
        • ParseToTimestamp
        • PlanExpression
        • Predicate
        • Projection
        • PythonUDF
        • RowNumber
        • RowNumberLike
        • RowOrdering
        • RuntimeReplaceable
        • ScalaAggregator
        • ScalarSubquery
        • ScalarSubquery (ExecSubqueryExpression)
        • ScalaUDAF
        • ScalaUDF
        • SimpleTypedAggregateExpression
        • SortOrder
        • SparkUserDefinedFunction
        • Stateful
        • StaticInvoke
        • SubqueryExpression
        • TimeWindow
        • TypedImperativeAggregate
        • UnaryExpression
        • Unevaluable
        • UnixTimestamp
        • UnresolvedAttribute
        • UnresolvedFunction
        • UnresolvedGenerator
        • UnresolvedOrdinal
        • UnresolvedStar
        • UnsafeProjection
        • UserDefinedAggregator
          • User-Defined Functions
          • UserDefinedExpression
          • UserDefinedFunction
          • UDFs are Blackbox
        • UserDefinedAggregateFunction
        • WindowExpression
        • WindowFunction
        • WindowSpecDefinition
      • Execution Planning Strategies
        • Aggregation
        • BasicOperators
        • DataSourceStrategy
        • DataSourceV2Strategy
        • FileSourceStrategy
        • InMemoryScans
        • JoinSelection
        • LogicalQueryStageStrategy
        • SparkStrategies
        • SparkStrategy
        • SpecialLimits
        • Window
        • WithCTEStrategy
      • Logical Analyzer
      • Logical Analysis Rules
        • AddMetadataColumns
        • AliasViewChild
        • BindParameters
        • CTESubstitution
        • CleanupAliases
        • DataSourceAnalysis
        • ExtractWindowExpressions
        • FindDataSourceTable
        • LookupFunctions
        • PreWriteCheck
        • PreprocessTableCreation
        • ResolveAggregateFunctions
        • ResolveAliases
        • ResolveCatalogs
        • ResolveCoalesceHints
        • ResolveCreateNamedStruct
        • ResolveDefaultColumns
        • ResolveFunctions
        • ResolveGroupingAnalytics
        • ResolveInlineTables
        • ResolveInsertInto
        • ResolveJoinStrategyHints
        • ResolveOrdinalInOrderByAndGroupBy
        • ResolveReferences
        • ResolveRelations
        • ResolveSQLOnFile
        • ResolveSessionCatalog
        • ResolveSubquery
        • ResolveWindowFrame
        • ResolveWithCTE
        • RewriteDeleteFromTable
        • RewriteRowLevelCommand
        • TableCapabilityCheck
        • WindowFrameCoercion
        • WindowsSubstitution
      • Logical Operators
        • AddColumns
        • Aggregate
        • AlterTable
        • AlterTableAddColumnsCommand
        • AlterTableCommand
        • AnalysisOnlyCommand
        • AnalyzeColumn
        • AnalyzeColumnCommand
        • AnalyzePartitionCommand
        • AnalyzeTable
        • AnalyzeTableCommand
        • AnalyzeTablesCommand
        • AppendData
        • ArrowEvalPython
        • BaseEvalPython
        • CTERelationDef
        • CTERelationRef
        • CacheTableCommand
        • ClearCacheCommand
        • CollectMetrics
        • Command
        • CommentOnTable
        • CreateDataSourceTableAsSelectCommand
        • CreateDataSourceTableCommand
        • CreateNamespace
        • CreateTable
        • CreateTableAsSelect
        • CreateTempViewUsing
        • CreateView
        • CreateViewCommand
        • DataSourceV2Relation
        • DataSourceV2ScanRelation
        • DataWritingCommand
        • DeleteFromTable
        • DescribeColumnCommand
        • DescribeRelation
        • DescribeTableCommand
        • DeserializeToObject
        • DropNamespace
        • Except
        • Expand
        • ExplainCommand
        • ExposesMetadataColumns
        • ExternalRDD
        • FlatMapGroupsWithState
        • Generate
        • GlobalLimit
        • GroupingSets
        • IgnoreCachedData
        • InMemoryRelation
        • InsertIntoDataSourceCommand
        • InsertIntoDir
        • InsertIntoHadoopFsRelationCommand
        • InsertIntoStatement
        • InsertIntoTable
        • Intersect
        • Join
        • LeafNodes
        • LeafRunnableCommand
        • LoadDataCommand
        • LocalRelation
        • LogicalPlan
        • LogicalPlanDistinctKeys
        • LogicalQueryStage
        • LogicalRDD
        • LogicalRelation
        • MapPartitions
        • MergeIntoTable
        • MultiInstanceRelation
        • NameParameterizedQuery
        • NamedRelation
        • Offset
        • OrderPreservingUnaryNode
        • OverwriteByExpression
        • OverwritePartitionsDynamic
        • ParameterizedQuery
        • ParsedStatement
        • Pivot
        • Project
        • RebalancePartitions
        • RelationTimeTravel
        • Repartition
        • RepartitionByExpression
        • RepartitionOperation
        • ReplaceData
        • ResolvedHint
        • ResolvedTable
        • RowLevelWrite
        • RunnableCommand
        • SaveIntoDataSourceCommand
        • SetCatalogAndNamespace
        • ShowColumns
        • ShowColumnsCommand
        • ShowCreateTable
        • ShowCreateTableCommand
        • ShowTableProperties
        • ShowTablePropertiesCommand
        • ShowTables
        • Sort
        • SubqueryAlias
        • SupportsSubquery
        • TruncateTableCommand
        • Union
        • UnresolvedCatalogRelation
        • UnresolvedHaving
        • UnresolvedHint
        • UnresolvedRelation
        • UnresolvedTable
        • UnresolvedTableOrView
        • UnresolvedTableValuedFunction
        • UnresolvedWith
        • UpdateTable
        • V1WriteCommand
        • V2CreateTablePlan
        • V2WriteCommand
        • View
        • Window
        • WithCTE
        • WithWindowDefinition
        • WriteDelta
        • WriteFiles
      • Logical Optimizations
        • AQEPropagateEmptyRelation Adaptive Logical Optimization
        • CleanupDynamicPruningFilters Logical Optimization
        • CollapseWindow Logical Optimization
        • ColumnPruning Logical Optimization
        • CombineTypedFilters Logical Optimization
        • CombineUnions Logical Optimization
        • ComputeCurrentTime Logical Optimization
        • ConstantFolding Logical Optimization
        • ConvertToLocalRelation Logical Optimization
        • CostBasedJoinReorder Logical Optimization -- Join Reordering in Cost-Based Optimization
        • DecimalAggregates Logical Optimization
        • DynamicJoinSelection Adaptive Logical Optimization
        • EliminateResolvedHint Logical Optimization
        • EliminateSerialization Logical Optimization
        • EliminateSubqueryAliases Logical Optimization
        • EliminateView Logical Optimization
        • ExtractPythonUDFFromAggregate Logical Optimization
        • ExtractPythonUDFs
        • GetCurrentDatabase Logical Optimization
        • GroupBasedRowLevelOperationScanPlanning Logical Optimization
        • InferFiltersFromConstraints Logical Optimization Rule
        • InjectRuntimeFilter
        • InlineCTE Logical Optimization
        • LimitPushDown Logical Optimization
        • NullPropagation Logical Optimization -- Nullability (NULL Value) Propagation
        • OptimizeIn Logical Optimization
        • OptimizeMetadataOnlyQuery Logical Optimization
        • OptimizeSubqueries Logical Optimization
        • PartitionPruning
        • PropagateEmptyRelation Logical Optimization
        • PruneFileSourcePartitions Logical Optimization
        • PruneFilters
        • PruneHiveTablePartitions Logical Optimization
        • PullupCorrelatedPredicates Logical Optimization
        • PushDownLeftSemiAntiJoin
        • PushDownOperatorsToDataSource Logical Optimization
        • PushDownPredicate Logical Optimization
        • PushDownPredicates Logical Optimization
        • PushPredicateThroughJoin Logical Optimization
        • ReorderJoin Logical Optimization -- Reordering Inner and Cross Joins
        • ReplaceExceptWithAntiJoin Logical Optimization Rule -- Rewriting Except (DISTINCT) Operators
        • ReplaceExceptWithFilter Logical Optimization Rule -- Rewriting Except (DISTINCT) Operators
        • ReplaceExpressions Logical Optimization
        • RewriteCorrelatedScalarSubquery Logical Optimization
        • RewriteExceptAll Logical Optimization Rule -- Rewriting Except (ALL) Operators
        • RewritePredicateSubquery Logical Optimization
        • SchemaPruning Logical Optimization
        • SimplifyCasts Logical Optimization
        • UpdateAttributeNullability Logical Optimization
        • UpdateCTERelationStats Logical Optimization
        • V2ScanRelationPushDown Logical Optimization
        • V2Writes Logical Optimization
      • Physical Operators
        • AdaptiveSparkPlanExec
        • AggregateCodegenSupport
        • AliasAwareOutputExpression
        • AliasAwareQueryOutputOrdering
        • AlterTableExec
        • AQEShuffleReadExec
        • AtomicTableWriteExec
        • BaseAggregateExec
        • BaseCacheTableExec
        • BaseJoinExec
        • BaseSubqueryExec
        • BatchWriteHelper
        • BatchScanExec
        • BroadcastExchangeExec
        • BroadcastExchangeLike
        • BroadcastHashJoinExec
        • BroadcastNestedLoopJoinExec
        • BroadcastQueryStageExec
        • CacheTableAsSelectExec
        • CacheTableExec
        • CoalesceExec
        • CollectLimitExec
        • CollectMetricsExec
        • ColumnarToRowExec
        • ColumnarToRowTransition
        • CreateTableAsSelectExec
        • CodegenSupport
        • DataSourceScanExec
        • DataSourceV2ScanExecBase
        • DataWritingCommandExec
        • DebugExec
        • DeleteFromTableExec
        • DescribeTableExec
        • DeserializeToObjectExec
        • DropNamespaceExec
        • EvalPythonExec
        • Exchange
        • ExecutedCommandExec
        • ExpandExec
        • ExternalRDDScanExec
        • FileSourceScanExec
        • FilterExec
        • GenerateExec
        • HashAggregateExec
        • HashedRelation
        • HashJoin
        • InMemoryTableScanExec
        • InputAdapter
        • JoinCodegenSupport
        • LocalTableScanExec
        • LongHashedRelation
        • ObjectConsumerExec
        • ObjectHashAggregateExec
        • ObjectProducerExec
        • OrderPreservingUnaryExecNode
        • OverwriteByExpressionExec
        • PartitioningPreservingUnaryExecNode
        • ProjectExec
        • QueryStageExec
        • RangeExec
        • ReusedExchangeExec
        • ReusedSubqueryExec
        • RowDataSourceScanExec
        • RowToColumnarExec
        • SerializeFromObjectExec
        • SetCatalogAndNamespaceExec
        • ShowCreateTableExec
        • ShowTablesExec
        • ShowTablePropertiesExec
        • ShuffleExchangeExec
        • ShuffleExchangeLike
        • ShuffledHashJoinExec
        • ShuffledJoin
        • ShuffleOrigin
        • ShuffleQueryStageExec
        • SortAggregateExec
          • SortMergeJoinExec
          • SortMergeJoinScanner
        • SortExec
        • SparkPlan
        • SubqueryExec
        • TableWriteExecHelper
        • TruncateTableExec
        • UnaryExecNode
        • V2CommandExec
        • V2ExistingTableWriteExec
        • V2TableWriteExec
        • WholeStageCodegenExec
        • WindowExec
        • WindowExecBase
        • WriteDeltaExec
        • WriteFilesExec
          • Distribution
          • Partitioning
            • AllTuples
            • BroadcastDistribution
            • ClusteredDistribution
            • HashClusteredDistribution
            • OrderedDistribution
            • UnspecifiedDistribution
            • BroadcastMode
            • HashedRelationBroadcastMode
          • ShuffleSpec
      • Physical Optimizations
        • AQEShuffleReadRule
        • AdjustShuffleExchangePosition
        • ApplyColumnarRulesAndInsertTransitions
        • CoalesceBucketsInJoin
        • CoalesceShufflePartitions
        • CollapseCodegenStages
        • DisableUnnecessaryBucketedScan
        • EnsureRequirements
        • InsertAdaptiveSparkPlan
        • OptimizeShuffleWithLocalRead
        • OptimizeSkewInRebalancePartitions
        • OptimizeSkewedJoin
        • PlanAdaptiveDynamicPruningFilters
        • PlanAdaptiveSubqueries
        • PlanDynamicPruningFilters
        • PlanSubqueries
        • RemoveRedundantProjects
        • RemoveRedundantSorts
        • ReplaceHashWithSortAgg
        • ReuseAdaptiveSubquery
        • ReuseExchange
        • ReuseExchangeAndSubquery
        • ReuseSubquery
        • ValidateSparkPlan
      • QueryExecution
        Structured Query Execution Pipeline
      • QueryPlanningTracker
      • SparkOptimizer
        Logical Query Plan Optimizer
      • SparkPlanner
        Spark Query Planner
      • Overview
      • DataSource
        Pluggable Data Provider Framework
      • Developer API
      • ExecutionListenerBus
      • ExecutionListenerManager
      • SharedState
        State Shared Across SparkSessions
      • SQLConf
      • SQLConfHelper
      • StaticSQLConf
        Static Configuration Properties
          • Catalog
          • CatalogImpl
          • CatalogStatistics
          • CatalogUtils
        • ExperimentalMethods
          • ExternalCatalog
          • InMemoryCatalog
          • ExternalCatalogWithListener
          • FunctionRegistry
          • FunctionRegistryBase
          • SimpleFunctionRegistry
          • SimpleFunctionRegistryBase
          • SimpleTableFunctionRegistry
          • TableFunctionRegistry
        • GlobalTempViewManager
          • SessionCatalog
          • CatalogStorageFormat
          • CatalogTable
          • CatalogTablePartition
        • V2SessionCatalog
          • SessionState
          • BaseSessionStateBuilder
          • SessionStateBuilder
          • CacheManager
        • RuntimeConfig
        • Encoder
        • ExpressionEncoder
        • RowEncoder
        • ScalaReflection
      • SQLExecution
      • SQLMetric
      • Tungsten Execution Backend
        • CatalystSerde
        • ExternalAppendOnlyUnsafeRowArray
        • HashMapGenerator
        • InternalRow
        • UnsafeHashedRelation
        • UnsafeRow
        • UnsafeRowSerializerInstance
        • DataSourceRDD
        • DataSourceRDDPartition
        • FileScanRDD
        • ShuffledRowRDD
        • SQLExecutionRDD
    • SQL
      • AbstractSqlParser
      • AstBuilder
        ANTLR-based SQL Parser
      • CatalystSqlParser
      • ParserInterface
      • SparkSqlParser
        Default SQL Parser
      • SparkSqlAstBuilder
      • VariableSubstitution
    • Connectors
      • Avro
        • AvroOptions
        • AvroFileFormat
        • CatalystDataToAvro
        • AvroDataToCatalyst
      • Files
        • AggregatePushDownUtils
        • BaseDynamicPartitionDataWriter
        • BasicWriteJobStatsTracker
        • BasicWriteTaskStats
        • BasicWriteTaskStatsTracker
        • CompressionCodecs
        • DynamicPartitionDataSingleWriter
        • DynamicPartitionDataConcurrentWriter
        • FileBatchWrite
        • FileDataSourceV2
        • FileFormat
        • FileFormatDataWriter
        • FileFormatWriter
          • FileIndex
          • CatalogFileIndex
          • InMemoryFileIndex
          • PartitioningAwareFileIndex
          • PrunedInMemoryFileIndex
        • FilePartition
        • FilePartitionReaderFactory
        • FileScan
        • FileScanBuilder
        • FileStatusCache
        • FileTable
        • FileWrite
        • FileWriterFactory
        • HadoopFileLinesReader
        • HadoopFsRelation
        • PartitionedFile
        • PartitionedFileUtil
        • RecordReaderIterator
        • SharedInMemoryCache
        • SchemaMergeUtils
        • SingleDirectoryDataWriter
        • WriteJobStatsTracker
        • WriteTaskStats
        • WriteTaskStatsTracker
      • Hive
        • Configuration Properties
        • Hive Metastore
        • DataSinks
        • HiveClient
        • HiveClientImpl
        • HiveFileFormat
        • HiveUtils
        • IsolatedClientLoader
        • HiveTableRelation
        • CreateHiveTableAsSelectCommand
        • SaveAsHiveFile
        • InsertIntoHiveDirCommand
        • InsertIntoHiveTable
        • HiveTableScans
        • HiveTableScanExec
        • TableReader
        • HadoopTableReader
        • HiveSessionStateBuilder
        • HiveExternalCatalog
        • HiveSessionCatalog
        • HiveMetastoreCatalog
        • RelationConversions
        • ResolveHiveSerdeTable
        • DetermineTableStats
        • HiveAnalysis
      • JDBC
        • AggregatedDialect
        • JDBCOptions
        • JDBCRDD
        • JDBCRelation
        • JDBCScan
        • JDBCScanBuilder
        • JDBCTableCatalog
        • JdbcDialect
        • JdbcDialects
        • JdbcRelationProvider
        • JdbcUtils
      • Kafka
        • Configuration Properties
        • KafkaBatch
        • KafkaBatchWrite
        • KafkaBatchWriterFactory
        • KafkaDataConsumer
        • KafkaDataWriter
        • KafkaOffsetRangeLimit
        • KafkaOffsetReader
        • KafkaRelation
        • KafkaScan
        • KafkaSourceProvider
        • KafkaSourceRDD
        • KafkaSourceRDDPartition
        • KafkaTable
        • KafkaWrite
        • KafkaWriter
        • KafkaWriteTask
          • ConsumerStrategy
          • InternalKafkaConsumer
          • InternalKafkaProducerPool
          • JsonUtils
          • KafkaRowWriter
          • KafkaRecordToRowConverter
        • Options
      • Noop
        • NoopDataSource
        • NoopTable
        • NoopWriteBuilder
        • NoopBatchWrite
        • NoopStreamingWrite
      • Parquet
        • ParquetDataSourceV2
        • ParquetFileFormat
        • ParquetFilters
        • ParquetOptions
        • ParquetPartitionReaderFactory
        • ParquetReadSupport
        • ParquetScan
        • ParquetScanBuilder
        • ParquetTable
        • ParquetUtils
        • ParquetWrite
        • ParquetWriteSupport
        • SparkToParquetSchemaConverter
        • SpecificParquetRecordReaderBase — Hadoop RecordReader
        • VectorizedColumnReader
        • VectorizedParquetRecordReader
      • DataWritingSparkTask
      • DataSourceV2Utils
      • OutputWriter
      • Column
      • ColumnarRule
      • Connector API
        • ApplyTransform
        • Batch
        • BatchWrite
        • CustomMetric
        • DataSourceV2Implicits
        • DataWriter
        • DataWriterFactory
        • Expression
        • InputPartition
        • KeyGroupedPartitioning
        • LocalScan
        • OptionsHelper
        • PartitionReader
        • PartitionReaderFactory
        • PartitionSpecsHelper
        • Partitioning
        • Predicate
        • RewritableTransform
        • RowLevelOperation
        • Scan
        • ScanBuilder
        • SessionConfigSupport
        • SimpleTableProvider
        • StagedTable
        • SupportsAtomicPartitionManagement
        • SupportsDelete
        • SupportsDeleteV2
        • SupportsDelta
        • SupportsDynamicOverwrite
        • SupportsMetadata
        • SupportsMetadataColumns
        • SupportsOverwrite
        • SupportsPartitionManagement
        • SupportsPushDownAggregates
        • SupportsPushDownFilters
        • SupportsPushDownRequiredColumns
        • SupportsPushDownV2Filters
        • SupportsRead
        • SupportsReportOrdering
        • SupportsReportPartitioning
        • SupportsReportStatistics
        • SupportsRowLevelOperations
        • SupportsRuntimeFiltering
        • SupportsRuntimeV2Filtering
        • SupportsStreamingUpdate
        • SupportsTruncate
        • SupportsWrite
        • Table
        • TableCapability
        • TableHelper
        • TableProvider
        • Transform
        • TransformHelper
        • TruncatableTable
        • V1Scan
        • V1Table
        • V1WriteBuilder
        • Write
        • WriteBuilder
      • Data Types
        • AbstractDataType
        • ArrayType
        • AtomicType
        • CalendarInterval
        • DataType
        • Metadata
        • MetadataBuilder
        • StructField
        • StructType
        • UserDefinedType
      • DataFrame
      • DataFrameNaFunctions
        Working With Missing Data
      • DataFrameReader
      • DataFrameStatFunctions
      • DataFrameWriter
      • DataFrameWriterV2
      • Dataset
      • Dataset API
        • Actions
        • Basic Actions
        • Typed Transformations
        • Untyped Transformations
        • DataSourceRegister
        • CreatableRelationProvider
        • RelationProvider
        • SchemaRelationProvider
        • BaseRelation
        • FileRelation
        • InsertableRelation
        • PrunedFilteredScan
        • PrunedScan
        • TableScan
        • Filter
      • Encoders
      • KeyValueGroupedDataset
      • Observation
      • QueryExecutionListener
      • RelationalGroupedDataset
      • SparkSession
        The Entry Point to Spark SQL
      • SparkSession.Builder
      • SparkSessionExtensions
      • Standard Functions
        org.apache.spark.sql.functions
        • Aggregate
        • Collection
        • Date time
        • Non-aggregate (Normal)
        • Window
      • TypedColumn
      • Window Functions
        • AggregateProcessor
        • RangeFrame
        • Window
        • WindowFunctionFrame
        • WindowSpec
    • Web UI
      • AllExecutionsPage
      • ExecutionPage
      • SQLAppStatusListener
      • SQLAppStatusStore
      • SQLTab
      • SparkListenerSQLExecutionEnd
    • Demo
      • Adaptive Query Execution
      • Connecting Spark SQL to Hive Metastore
      • Mult-Dimensional Aggregations
      • Developing CatalogPlugin
      • Dynamic Partition Pruning
      • Hive Partitioned Parquet Table and Partition Pruning
      • ObjectHashAggregateExec and Sort-Based Fallback Tasks
      • Spilling
      • Using JDBC Data Source to Access PostgreSQL
      • AggregatingAccumulator
      • DistinctKeyVisitor
      • FilterEvaluatorFactory
      • JoinSelectionHelper
      • PushDownUtils
      • UnsafeExternalRowSorter
      • BindReferences
      • IntervalUtils
      • ExplainUtils
      • SerializerBuildHelper
      • Dataset, DataFrame and RDD
      • Dataset and SQL
      • DDLUtils
      • Implicits
      • Row
        • CreateTableWriter
        • WriteConfigMethods
      • Column Operators
        • Caching and Persistence
        • User-Friendly Names of Cached Queries in web UI
      • Checkpointing
        • Debugging Query Execution
        • Performance Tuning
        • Case Study
      • CheckAnalysis
      • CatalystTypeConverters
      • SubExprUtils
      • PredicateHelper
      • ExtractEquiJoinKeys
      • ExtractSingleColumnNullAwareAntiJoin
      • ExtractJoinWithBuckets
      • PhysicalOperation
      • KnownSizeEstimation
    1. Spark SQL
    2. High-Level APIs
    3. Data Types

    Metadata

    Copyright © 2024-2025 Jacek Laskowski
    Made with Material for MkDocs Insiders