SerializationStream¶

SerializationStream is an abstraction of serialized streams for writing out serialized key-value records.

Contract¶

Closing Stream¶

close(): Unit

Flushing Stream¶

flush(): Unit

Used when:

• UnsafeShuffleWriter is requested to insert a record into a ShuffleExternalSorter
• DiskBlockObjectWriter is requested to commitAndGet

Writing Out Object¶

writeObject[T: ClassTag](
  t: T): SerializationStream

Used when:

• MemoryStore is requested to putIteratorAsBytes
• JavaSerializerInstance is requested to serialize
• RequestMessage is requested to serialize (for NettyRpcEnv)
• ParallelCollectionPartition is requested to writeObject (for ParallelCollectionRDD)
• ReliableRDDCheckpointData is requested to doCheckpoint
• TorrentBroadcast is created (and requested to writeBlocks)
• RangePartitioner is requested to writeObject
• SerializationStream is requested to writeKey, writeValue or writeAll
• FileSystemPersistenceEngine is requested to serializeIntoFile (for Spark Standalone's Master)

Implementations¶

• JavaSerializationStream
• KryoSerializationStream

Writing Out All Records¶

writeAll[T: ClassTag](
  iter: Iterator[T]): SerializationStream

writeAll writes out records of the given iterator (one by one as objects).

writeAll is used when:

• ReliableCheckpointRDD is requested to doCheckpoint
• SerializerManager is requested to dataSerializeStream and dataSerializeWithExplicitClassTag

Writing Out Key¶

writeKey[T: ClassTag](
  key: T): SerializationStream

Writes out the key

writeKey is used when:

• UnsafeShuffleWriter is requested to insert a record into a ShuffleExternalSorter
• DiskBlockObjectWriter is requested to write the key and value of a record

Writing Out Value¶

writeValue[T: ClassTag](
  value: T): SerializationStream

Writes out the value

writeValue is used when:

• UnsafeShuffleWriter is requested to insert a record into a ShuffleExternalSorter
• DiskBlockObjectWriter is requested to write the key and value of a record