Skip to content

StreamingSymmetricHashJoinHelper Utility

Join State Watermark Predicates

  leftAttributes: Seq[Attribute],
  rightAttributes: Seq[Attribute],
  leftKeys: Seq[Expression],
  rightKeys: Seq[Expression],
  condition: Option[Expression],
  eventTimeWatermark: Option[Long]): JoinStateWatermarkPredicates

getStateWatermarkPredicates creates a JoinStateWatermarkPredicates with the JoinStateWatermarkPredicates for the left and right side of a join (if defined).

getStateWatermarkPredicates finds the index of the first column (attribute) with the watermark delay metadata marker among the given leftKeys first, and if not found, among the given rightKeys. getStateWatermarkPredicates may find no column.

getStateWatermarkPredicates determines the state watermark predicate (a JoinStateWatermarkPredicate) for the leftStateWatermarkPredicate and rightStateWatermarkPredicate sides of a join.

JoinStateWatermarkPredicate oneSideInputAttributes oneSideJoinKeys otherSideInputAttributes
leftStateWatermarkPredicate leftAttributes leftKeys rightAttributes
rightStateWatermarkPredicate rightAttributes rightKeys leftAttributes

getStateWatermarkPredicates is used when:

Join State Watermark Predicate (for One Side of Join)

  oneSideInputAttributes: Seq[Attribute],
  oneSideJoinKeys: Seq[Expression],
  otherSideInputAttributes: Seq[Attribute]): Option[JoinStateWatermarkPredicate]

Watermark on Join Keys

With a watermark defined on one of the join keys (leftKeys or rightKeys of getStateWatermarkPredicates), getOneSideStateWatermarkPredicate creates a JoinStateKeyWatermarkPredicate with a watermark (eviction) expression for the following:

Watermark on Input

With a watermark defined on the given oneSideInputAttributes, getOneSideStateWatermarkPredicate creates a JoinStateValueWatermarkPredicate with a watermark (eviction) expression for the following:

No Watermark

getOneSideStateWatermarkPredicate creates no JoinStateWatermarkPredicate (None) when no watermark was found.