Skip to content

DelegatingLogStore

DelegatingLogStore is the default LogStore.

Creating Instance

DelegatingLogStore takes the following to be created:

DelegatingLogStore is created when:

Default LogStore

defaultLogStore: LogStore

DelegatingLogStore creates a LogStore (lazily) that is used when requested to schemeBasedLogStore.

Lazy Value

defaultLogStore is a Scala lazy value to guarantee that the code to initialize it is executed once only (when accessed for the first time) and the computed value never changes afterwards.

LogStore by Scheme Lookup Table

schemeToLogStoreMap: Map[String, LogStore]

DelegatingLogStore uses an internal registry of LogStores by scheme for looking them up once created.

Looking Up LogStore Delegate by Path

getDelegate(
  path: Path): LogStore

getDelegate is a mere alias of schemeBasedLogStore.

schemeBasedLogStore

schemeBasedLogStore(
  path: Path): LogStore

schemeBasedLogStore takes the scheme component (of the URI) of the given path.

If undefined, schemeBasedLogStore gives the defaultLogStore.

For a scheme defined, schemeBasedLogStore looks it up in the schemeToLogStoreMap registry and returns it when found.

Otherwise, schemeBasedLogStore creates a LogStore based on the following (in the order):

  1. Scheme-specific configuration key to look up the class name of the LogStore in the SparkConf
  2. Default LogStore class name for the scheme
  3. Uses the defaultLogStore

schemeBasedLogStore registers the LogStore in the schemeToLogStoreMap registry for future lookups.

schemeBasedLogStore prints out the following INFO message to the logs:

LogStore [className] is used for scheme [scheme]

Default LogStore (Class Name) for Scheme

getDefaultLogStoreClassName(
  scheme: String): Option[String]

getDefaultLogStoreClassName returns the class name of the LogStore for a given scheme or None (undefined).

Schemes Class Name
s3, s3a, s3n S3SingleDriverLogStore
abfs, abfss, adl, wasb, wasbs AzureLogStore

Creating LogStore

createLogStore(
  className: String): LogStore

createLogStore creates a LogStore for the given class name.

createLogStore is used when: