Skip to content

ExecutorPluginContainer

ExecutorPluginContainer is a PluginContainer for Executors.

Creating Instance

ExecutorPluginContainer takes the following to be created:

ExecutorPluginContainer is created when:

ExecutorPlugins

ExecutorPluginContainer initializes executorPlugins internal registry of ExecutorPlugins when created.

Initialization

executorPlugins finds all the configuration properties with spark.plugins.internal.conf. prefix (in the SparkConf) for extra configuration of every ExecutorPlugin of the given SparkPlugins.

For every SparkPlugin (in the given SparkPlugins) that defines an ExecutorPlugin, executorPlugins creates a PluginContextImpl, requests the ExecutorPlugin to init (with the PluginContextImpl and the extra configuration) and the PluginContextImpl to registerMetrics.

In the end, executorPlugins prints out the following INFO message to the logs (for every ExecutorPlugin):

Initialized executor component for plugin [name].

Logging

Enable ALL logging level for org.apache.spark.internal.plugin.ExecutorPluginContainer logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.internal.plugin.ExecutorPluginContainer=ALL

Refer to Logging.