ElementTrackingStore¶

ElementTrackingStore is a KVStore that tracks the number of entities (elements) of specific types in a store and triggers actions once they reach a threshold.

Creating Instance¶

ElementTrackingStore takes the following to be created:

• KVStore
• SparkConf

ElementTrackingStore is created when:

• AppStatusStore is requested to createLiveStore
• FsHistoryProvider is requested to rebuildAppStore

Writing Value to Store¶

write(
  value: Any): Unit

write is part of the KVStore abstraction.

write requests the KVStore to write the value

Writing Value to Store and Checking Triggers¶

write(
  value: Any,
  checkTriggers: Boolean): WriteQueueResult

write writes the value.

write...FIXME

write is used when:

• LiveEntity is requested to write
• StreamingQueryStatusListener (Spark Structured Streaming) is requested to onQueryStarted and onQueryTerminated

Creating View of Specific Entities¶

view[T](
  klass: Class[T]): KVStoreView[T]

view is part of the KVStore abstraction.

view requests the KVStore for a view of klass entities.

Registering Trigger¶

addTrigger(
  klass: Class[_],
  threshold: Long)(
  action: Long => Unit): Unit

addTrigger...FIXME

addTrigger is used when:

• AppStatusListener is created
• HiveThriftServer2Listener (Spark Thrift Server) is created
• SQLAppStatusListener (Spark SQL) is created
• StreamingQueryStatusListener (Spark Structured Streaming) is created