Skip to content

InputPartition

InputPartition is an abstraction of input partitions in DataSource V2 with optional location preferences.

InputPartition is a Java Serializable.

Contract

preferredLocations

String[] preferredLocations()

Specifies preferred locations (executor hosts)

By default, preferredLocations defines no location preferences (is simply empty).

Used when:

  • FileScanRDD is requested for preferred locations

  • DataSourceRDD is requested for preferred locations

  • ContinuousDataSourceRDD (Spark Structured Streaming) is requested for preferred locations

Implementations

  • ContinuousMemoryStreamInputPartition
  • FilePartition
  • KafkaBatchInputPartition
  • KafkaContinuousInputPartition
  • MemoryStreamInputPartition
  • RateStreamContinuousInputPartition
  • RateStreamMicroBatchInputPartition
  • TextSocketContinuousInputPartition
  • TextSocketInputPartition