Plugin Framework¶

Plugin Framework is an API for registering custom extensions (plugins) to be executed on the driver and executors.

Plugin Framework uses separate PluginContainers for the driver and executors, and spark.plugins configuration property for SparkPlugins to be registered.

Plugin Framework was introduced in Spark 2.4.4 (with an API for executors) with further changes in Spark 3.0.0 (to cover the driver).

Resources¶

• Advanced Instrumentation in the official documentation of Apache Spark
• Commit for SPARK-29397
• Spark Plugin Framework in 3.0 - Part 1: Introduction by Madhukara Phatak
• Spark Memory Monitor by squito
• SparkPlugins by Luca Canali (CERN)