SerializerManager

SerializerManager is used to select a serializer for shuffle blocks (either the default JavaSerializer or KryoSerializer based on the key and value of a record).

Creating Instance

SerializerManager takes the following to be created:

SerializerManager is created when SparkEnv utility is used to create a SparkEnv for the driver and executors.

Accessing SerializerManager Using SparkEnv

SerializerManager is available using SparkEnv on the driver and executors.

import org.apache.spark.SparkEnv
SparkEnv.get.serializerManager

KryoSerializer

SerializerManager creates a KryoSerializer when created.

KryoSerializer is used as a serializer when the type of a given key and value is compatible with Kryo.

Wrapping Input or Output Stream of Block for Compression

wrapForCompression(
  blockId: BlockId,
  s: OutputStream): OutputStream
wrapForCompression(
  blockId: BlockId,
  s: InputStream): InputStream

wrapForCompression…​FIXME

wrapForCompression is used when:

Wrapping Input or Output Stream for Block

wrapStream(
  blockId: BlockId,
  s: InputStream): InputStream
wrapStream(
  blockId: BlockId,
  s: OutputStream): OutputStream

wrapStream…​FIXME

wrapStream is used when:

dataSerializeStream Method

dataSerializeStream[T: ClassTag](
  blockId: BlockId,
  outputStream: OutputStream,
  values: Iterator[T]): Unit

dataSerializeStream…​FIXME

dataSerializeStream is used when BlockManager is requested to doPutIterator and dropFromMemory.

dataSerializeWithExplicitClassTag Method

dataSerializeWithExplicitClassTag(
  blockId: BlockId,
  values: Iterator[_],
  classTag: ClassTag[_]): ChunkedByteBuffer

dataSerializeWithExplicitClassTag…​FIXME

dataSerializeWithExplicitClassTag is used when BlockManager is requested to doGetLocalBytes.

dataDeserializeStream Method

dataDeserializeStream[T](
  blockId: BlockId,
  inputStream: InputStream)
  (classTag: ClassTag[T]): Iterator[T]

dataDeserializeStream…​FIXME

dataDeserializeStream is used when:

Selecting Serializer

getSerializer(
  ct: ClassTag[_],
  autoPick: Boolean): Serializer
getSerializer(
  keyClassTag: ClassTag[_],
  valueClassTag: ClassTag[_]): Serializer

getSerializer returns the KryoSerializer when the given arguments are compatible with Kryo. Otherwise, getSerializer returns the Serializer.

getSerializer is used when:

Checking Whether Kryo Serializer Could Be Used

canUseKryo(
  ct: ClassTag[_]): Boolean

canUseKryo is true when the given ClassTag is a primitive, an array of primitives or a String. Otherwise, canUseKryo is false.

canUseKryo is used when SerializerManager is requested for a Serializer.

shouldCompress Method

shouldCompress(
  blockId: BlockId): Boolean

shouldCompress…​FIXME

shouldCompress is used when SerializerManager is requested to wrapForCompression.