SimpleTableProvider¶

SimpleTableProvider is an extension of the TableProvider abstraction for table providers that do not support custom table schema and partitioning.

Contract¶

Creating Table¶

getTable(
  options: CaseInsensitiveStringMap): Table

Creates a Table

Used for getOrLoadTable

Implementations¶

• ConsoleSinkProvider (Spark Structured Streaming)
• KafkaSourceProvider
• MemoryStreamTableProvider (Spark Structured Streaming)
• NoopDataSource
• RateStreamProvider (Spark Structured Streaming)
• TextSocketSourceProvider (Spark Structured Streaming)

Inferring Schema¶

inferSchema(
  options: CaseInsensitiveStringMap): StructType

inferSchema creates a table and requests it for the schema.

inferSchema is part of the TableProvider abstraction.

Creating Table¶

getTable(
  schema: StructType,
  partitioning: Array[Transform],
  properties: util.Map[String, String]): Table

getTable creates a table (with the given properties).

getTable asserts that there are no Transforms in given partitioning.

getTable is part of the TableProvider abstraction.

Looking Up Table Once¶

getOrLoadTable(
  options: CaseInsensitiveStringMap): Table

getOrLoadTable creates a table and caches it for later use (in an internal variable).