Beam Runner for Google Cloud Dataflow

Beam applications can be executed on Google Cloud Dataflow using DataflowRunner.

Dependencies

DataflowRunner is included in a separate Beam library that is required to be specified explicitly as a project dependency.

build.sbt
libraryDependencies ++= Seq(
  "org.apache.beam" % "beam-runners-google-cloud-dataflow-java" % "2.21.0",
  "com.google.guava" % "guava" % "29.0-jre")

Demo

import org.apache.beam.sdk.options.PipelineOptionsFactory
import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions
val options = PipelineOptionsFactory
  .create()
  .as(classOf[DataflowPipelineOptions])
options.setAppName("Beam on Dataflow Test")
options.setProject("ProjectID")
options.setTempLocation("gs://bucket/dataflow-beam-runner")

import org.apache.beam.runners.dataflow.DataflowRunner
DataflowRunner.fromOptions(options)