Skip to content


:hadoop-version: 2.10.0 :url-hadoop-javadoc:{hadoop-version}/api

PrunedInMemoryFileIndex is a[InMemoryFileIndex] for a <> at an <>.

PrunedInMemoryFileIndex may be given the <>.

PrunedInMemoryFileIndex is <> when CatalogFileIndex is requested to[filter the partitions of a partitioned table].

[[logging]] [TIP] ==== Enable ALL logging level for org.apache.spark.sql.execution.datasources.PrunedInMemoryFileIndex logger to see what happens inside.

Add the following line to conf/

Refer to[Logging].

=== [[creating-instance]] Creating PrunedInMemoryFileIndex Instance

PrunedInMemoryFileIndex takes the following to be created:

  • [[sparkSession]][SparkSession]
  • [[tableBasePath]] Location of the Hive metastore table (as a Hadoop {url-hadoop-javadoc}/org/apache/hadoop/fs/Path.html[Path])
  • [[fileStatusCache]] FileStatusCache
  • [[partitionSpec]] PartitionSpec (from a Hive metastore)
  • [[metadataOpsTimeNs]] Optional time of the partition metadata listing