InputPartition¶

InputPartition is an abstraction of input partitions in DataSource V2 with optional location preferences.

InputPartition is a Java Serializable.

Contract¶

preferredLocations¶

String[] preferredLocations()

Specifies preferred locations (executor hosts)

By default, preferredLocations defines no location preferences (is simply empty).

Used when:

• FileScanRDD is requested for preferred locations

• DataSourceRDD is requested for preferred locations

• ContinuousDataSourceRDD (Spark Structured Streaming) is requested for preferred locations

Implementations¶

• ContinuousMemoryStreamInputPartition
• FilePartition
• KafkaBatchInputPartition
• KafkaContinuousInputPartition
• MemoryStreamInputPartition
• RateStreamContinuousInputPartition
• RateStreamMicroBatchInputPartition
• TextSocketContinuousInputPartition
• TextSocketInputPartition