ShuffleInMemorySorter is used by ShuffleExternalSorter to sort pointers of key-value records and partition IDs using radix or tim sort algorithms.
ShuffleInMemorySorter takes the following to be created:
ShuffleInMemorySorter requests the given MemoryConsumer to allocate an array of the given initial size for the Unsafe LongArray of Record Pointers and Partition IDs.
ShuffleInMemorySorter is created for a ShuffleExternalSorter.
getSortedIterator is used when ShuffleExternalSorter is requested to writeSortedFile.
getUsableCapacity calculates the capacity that is a half or two-third of the memory used for the LongArray.
getUsableCapacity is used when…FIXME
ALL logging level for
org.apache.spark.shuffle.sort.ShuffleExternalSorter logger to see what happens inside.
Add the following line to
Refer to Logging.