PipelineUpdateContextImpl¶

PipelineUpdateContextImpl is a PipelineUpdateContext.

Creating Instance¶

PipelineUpdateContextImpl takes the following to be created:

• DataflowGraph
• PipelineEvent Callback (PipelineEvent => Unit)
• TableFilter of the tables to be refreshed (default: AllTables)
• TableFilter of the tables to be refreshed (default: NoTables)
• Pipeline Storage Location

While being created, PipelineUpdateContextImpl validates the storage root.

PipelineUpdateContextImpl is created when:

• PipelinesHandler is requested to run a pipeline

Pipeline Storage Location¶

PipelineUpdateContextImpl is given a storage location when created.

The storage root is the storage of the StartRun pipeline command (when PipelinesHandler is requested to run a pipeline update).

This storage root is immediately validated.

Validate Storage Root¶

validateStorageRoot(
  storageRoot: String): Unit

validateStorageRoot asserts that the given storageRoot meets the following requirements:

1. It is an absolute path
2. The URI schema is defined

Otherwise, validateStorageRoot reports a SparkException:

Pipeline storage root must be an absolute path with a URI scheme (e.g., file://, s3a://, hdfs://).
Got: `[storage_root]`.