Pipeline

Pipeline is a directed acyclic graph (DAG) of PTransforms and PCollections.

Once constructed, a Pipeline can be run using a PipelineRunner.

Creating Pipeline

Pipeline create() (1)
Pipeline create(
  PipelineOptions options)
1 Uses PipelineOptionsFactory to create a PipelineOptions

Pipeline.create utility creates a new Pipeline (using the provided PipelineOptions or requests PipelineOptionsFactory to create one).

create prints out the following DEBUG message to the logs:

Creating [pipeline]

Running Pipeline

PipelineResult run() (1)
PipelineResult run(
  PipelineOptions options)
1 Uses the default options

run prints out the following DEBUG message to the logs:

Running [this] via [runner]

run validates the options and requests the PipelineRunner to run this pipeline.

Validating Pipeline Options

void validate(
  PipelineOptions options)

validate…​FIXME

Creating Instance

Pipeline takes the following to be created:

Pipeline cannot be created directly and can only be created using Pipeline.create utility.