InputPartition¶
InputPartition
is an abstraction of input partitions in DataSource V2 with optional location preferences.
InputPartition
is a Java Serializable.
Contract¶
preferredLocations¶
String[] preferredLocations()
Specifies preferred locations (executor hosts)
By default, preferredLocations
defines no location preferences (is simply empty).
Used when:
-
FileScanRDD
is requested for preferred locations -
DataSourceRDD
is requested for preferred locations -
ContinuousDataSourceRDD
(Spark Structured Streaming) is requested for preferred locations
Implementations¶
ContinuousMemoryStreamInputPartition
FilePartition
KafkaBatchInputPartition
KafkaContinuousInputPartition
MemoryStreamInputPartition
RateStreamContinuousInputPartition
RateStreamMicroBatchInputPartition
TextSocketContinuousInputPartition
TextSocketInputPartition