SparkStructuredStreamingPipelineOptions

SparkStructuredStreamingPipelineOptions is a PipelineOptions of Spark configurations (e.g. masterURL) for SparkStructuredStreamingRunner.

import org.apache.beam.sdk.options.PipelineOptionsFactory
import org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineOptions
val opts = PipelineOptionsFactory.as(classOf[SparkStructuredStreamingPipelineOptions])

scala> PipelineOptionsFactory.printHelp(System.out, classOf[SparkStructuredStreamingPipelineOptions])
org.apache.beam.runners.spark.SparkCommonPipelineOptions:

  --checkpointDir=<String>
    Default: TmpCheckpointDirFactory
    A checkpoint directory for streaming resilience, ignored in batch. For
    durability, a reliable filesystem such as HDFS/S3/GS is necessary.
  ...