Skip to content


CommandUtils is a helper class for logical commands to manage table statistics.

Analyzing Table

  sparkSession: SparkSession,
  tableIdent: TableIdentifier,
  noScan: Boolean): Unit

analyzeTable requests the SessionCatalog for the table metadata.

analyzeTable branches off based on the type of the table: a view and the other types.


For CatalogTableType.VIEWs, analyzeTable requests the CacheManager to lookupCachedData. If available and the given noScan flag is disabled, analyzeTable requests the table to count the number of rows (that materializes the underlying columnar RDD).

Other Table Types

For other types, analyzeTable calculateTotalSize for the table. With the given noScan flag disabled, analyzeTable creates a DataFrame for the table and counts the number of rows (that triggers a Spark job). In case the table stats have changed, analyzeTable requests the SessionCatalog to alterTableStats.


analyzeTable is used when the following commands are executed:

Updating Existing Table Statistics

  sparkSession: SparkSession,
  table: CatalogTable): Unit

updateTableStats updates the table statistics of the input CatalogTable (only if the statistics are available in the metastore already).

updateTableStats requests SessionCatalog to alterTableStats with the <> (when spark.sql.statistics.size.autoUpdate.enabled property is enabled) or empty statistics (that effectively removes the recorded statistics completely).


updateTableStats uses spark.sql.statistics.size.autoUpdate.enabled property to auto-update table statistics and can be expensive (and slow down data change commands) if the total number of files of a table is very large.

updateTableStats is used when the following logical commands are executed:


  oldStats: Option[CatalogStatistics],
  newTotalSize: BigInt,
  newRowCount: Option[BigInt]): Option[CatalogStatistics]


compareAndGetNewStats is used when:


Enable ALL logging level for org.apache.spark.sql.execution.command.CommandUtils logger to see what happens inside.

Add the following line to conf/

Refer to Logging.