Skip to content


DiskBlockObjectWriter is a custom that BlockManager offers for writing data blocks to disk.

DiskBlockObjectWriter is used when:

  • BypassMergeSortShuffleWriter is requested for[partition writers]

  • UnsafeSorterSpillWriter is requested for a[partition writer]

  • ShuffleExternalSorter is requested to[writeSortedFile]

  • ExternalSorter is requested to[spillMemoryIteratorToDisk]

== [[creating-instance]] Creating Instance

DiskBlockObjectWriter takes the following to be created:

  • [[file]] Java {java-javadoc-url}/java/io/File.html[File]
  • [[serializerManager]][]
  • [[serializerInstance]][]
  • [[bufferSize]] Buffer size
  • [[syncWrites]] syncWrites flag
  • [[writeMetrics]][]
  • [[blockId]][] (default: null)

DiskBlockObjectWriter is created when:

  • BlockManager is requested for[one]

  • BypassMergeSortShuffleWriter is requested to[write records] (as[partition writers])

== [[objOut]] SerializationStream

DiskBlockObjectWriter manages a[SerializationStream] for <>:

  • Opens it when requested to <>

  • Closes it when requested to <>

  • Dereferences it (nulls it) when <>

== [[states]][[streamOpen]] States

DiskBlockObjectWriter can be in the following states (that match the state of the underlying output streams):

. Initialized . Open . Closed

== [[write]] Writing Key and Value (of Record)

[source, scala]

write( key: Any, value: Any): Unit

write <> unless <> already.

write requests the <> to[write the key] and then the[value].

In the end, write <>.

write is used when:

  • BypassMergeSortShuffleWriter is requested to[write records of a partition]

  • ExternalAppendOnlyMap is requested to[spillMemoryIteratorToDisk]

  • ExternalSorter is requested to[write all records into a partitioned file] ** SpillableIterator is requested to spill

  • WritablePartitionedPairCollection is requested for a destructiveSortedWritablePartitionedIterator

== [[commitAndGet]] commitAndGet Method

[source, scala]

commitAndGet(): FileSegment


commitAndGet is used when...FIXME

== [[close]] Committing Writes and Closing Resources

[source, scala]

close(): Unit


close is used when...FIXME

== [[revertPartialWritesAndClose]] revertPartialWritesAndClose Method

[source, scala]

revertPartialWritesAndClose(): File


revertPartialWritesAndClose is used when...FIXME

== [[updateBytesWritten]] updateBytesWritten Method


== [[initialize]] initialize Method


== [[write-bytes]] Writing Bytes (From Byte Array Starting From Offset)

[source, scala]

write(kvBytes: Array[Byte], offs: Int, len: Int): Unit



== [[recordWritten]] recordWritten Method


== [[open]] Opening DiskBlockObjectWriter

[source, scala]

open(): DiskBlockObjectWriter

open opens DiskBlockObjectWriter, i.e. <> and re-sets <> and <> internal output streams.

Internally, open makes sure that DiskBlockObjectWriter is not closed (i.e. <> flag is disabled). If it was, open throws a IllegalStateException:

Writer already closed. Cannot be reopened.

Unless DiskBlockObjectWriter has already been initialized (i.e. <> flag is enabled), open <> it (and turns <> flag on).

Regardless of whether DiskBlockObjectWriter was already initialized or not, open[requests SerializerManager to wrap mcs output stream for encryption and compression] (for <>) and sets it as <>.

open requests the <> to[serialize bs output stream] and sets it as <>.

NOTE: open uses SerializerInstance that was specified when <>

In the end, open turns <> flag on.

NOTE: open is used exclusively when DiskBlockObjectWriter <> or <> but the <>.

== [[internal-properties]] Internal Properties

[cols="30m,70",options="header",width="100%"] |=== | Name | Description

| initialized | [[initialized]] Internal flag...FIXME

Used when...FIXME

| hasBeenClosed | [[hasBeenClosed]] Internal flag...FIXME

Used when...FIXME

| mcs | [[mcs]] FIXME

Used when...FIXME

| bs | [[bs]] FIXME

Used when...FIXME


Last update: 2020-10-14