StreamingSymmetricHashJoinHelper Utility¶

Join State Watermark Predicates¶

getStateWatermarkPredicates(
  leftAttributes: Seq[Attribute],
  rightAttributes: Seq[Attribute],
  leftKeys: Seq[Expression],
  rightKeys: Seq[Expression],
  condition: Option[Expression],
  eventTimeWatermark: Option[Long]): JoinStateWatermarkPredicates

getStateWatermarkPredicates creates a JoinStateWatermarkPredicates with the JoinStateWatermarkPredicates for the left and right side of a join (if defined).

getStateWatermarkPredicates finds the index of the first column (attribute) with the watermark delay metadata marker among the given leftKeys first, and if not found, among the given rightKeys. getStateWatermarkPredicates may find no column.

getStateWatermarkPredicates determines the state watermark predicate (a JoinStateWatermarkPredicate) for the leftStateWatermarkPredicate and rightStateWatermarkPredicate sides of a join.

getStateWatermarkPredicates is used when:

• IncrementalExecution is requested for the state preparations rules (while optimizing query plans with StreamingSymmetricHashJoinExec physical operators)

Join State Watermark Predicate (for One Side of Join)¶

getOneSideStateWatermarkPredicate(
  oneSideInputAttributes: Seq[Attribute],
  oneSideJoinKeys: Seq[Expression],
  otherSideInputAttributes: Seq[Attribute]): Option[JoinStateWatermarkPredicate]

Watermark on Join Keys¶

With a watermark defined on one of the join keys (leftKeys or rightKeys of getStateWatermarkPredicates), getOneSideStateWatermarkPredicate creates a JoinStateKeyWatermarkPredicate with a watermark (eviction) expression for the following:

• A BoundReference (Spark SQL) for the key with watermark
• The given eventTimeWatermark (of getStateWatermarkPredicates)

Watermark on Input¶

With a watermark defined on the given oneSideInputAttributes, getOneSideStateWatermarkPredicate creates a JoinStateValueWatermarkPredicate with a watermark (eviction) expression for the following:

• The Attribute among the given oneSideInputAttributes with the watermark delay metadata marker
• getStateValueWatermark

No Watermark¶

getOneSideStateWatermarkPredicate creates no JoinStateWatermarkPredicate (None) when no watermark was found.