PipelineUpdateContextImpl¶

PipelineUpdateContextImpl is a PipelineUpdateContext.

Creating Instance¶

PipelineUpdateContextImpl takes the following to be created:

• DataflowGraph
• PipelineEvent Callback (PipelineEvent => Unit)
• TableFilter of the tables to be refreshed (default: AllTables)
• TableFilter of the tables to be refreshed (default: NoTables)
• Storage root

While being created, PipelineUpdateContextImpl validates the storage root.

PipelineUpdateContextImpl is created when:

• PipelinesHandler is requested to run a pipeline

Validate Storage Root¶

validateStorageRoot(
  storageRoot: String): Unit

validateStorageRoot asserts that the given storageRoot meets the following requirements:

1. It is an absolute path
2. The schema is defined

Otherwise, validateStorageRoot reports a SparkException:

Pipeline storage root must be an absolute path with a URI scheme (e.g., file://, s3a://, hdfs://).
Got: `[storage_root]`.