SimpleTableProvider¶
SimpleTableProvider
is an extension of the TableProvider abstraction for table providers that do not support custom table schema and partitioning.
Contract¶
Creating Table¶
getTable(
options: CaseInsensitiveStringMap): Table
Creates a Table
Used for getOrLoadTable
Implementations¶
- ConsoleSinkProvider (Spark Structured Streaming)
- KafkaSourceProvider
- MemoryStreamTableProvider (Spark Structured Streaming)
- NoopDataSource
- RateStreamProvider (Spark Structured Streaming)
- TextSocketSourceProvider (Spark Structured Streaming)
Inferring Schema¶
inferSchema(
options: CaseInsensitiveStringMap): StructType
inferSchema
creates a table and requests it for the schema.
inferSchema
is part of the TableProvider abstraction.
Creating Table¶
getTable(
schema: StructType,
partitioning: Array[Transform],
properties: util.Map[String, String]): Table
getTable
creates a table (with the given properties
).
getTable
asserts that there are no Transform
s in given partitioning
.
getTable
is part of the TableProvider abstraction.
Looking Up Table Once¶
getOrLoadTable(
options: CaseInsensitiveStringMap): Table
getOrLoadTable
creates a table and caches it for later use (in an internal variable).