Skip to content

FilePartition

maxSplitBytes

maxSplitBytes(
  sparkSession: SparkSession,
  selectedPartitions: Seq[PartitionDirectory]): Long

maxSplitBytes reads the following properties:

maxSplitBytes uses the given selectedPartitions to calculate totalBytes based on the size of the files with spark.sql.files.openCostInBytes added (for each file).

maxSplitBytes calculates bytesPerCore to be totalBytes divided by filesMinPartitionNum.

In the end, maxSplitBytes takes the minimum of the following:


maxSplitBytes is used when: