Skip to content

== Executors Tab

Executors tab in spark-webui.md[web UI] shows...FIXME

.Executors Tab in web UI (local mode) image::spark-webui-executors.png[align="center"]

The Executors tab is available under /executors URL, i.e. http://localhost:4040/executors.

Internally, the Executors tab is represented by spark-webui-ExecutorsTab.md[ExecutorsTab].

[NOTE]

What's interesting in how Storage Memory is displayed in the Executors tab is that the default UnifiedMemoryManager memory:UnifiedMemoryManager.md#getMaxMemory[calculates the maximum memory] in a way that is different from what the page displays (using the custom JavaScript formatBytes function in utils.js).

[source, scala]

// local mode with spark.driver.memory 2g // ./bin/spark-shell --conf spark.driver.memory=2g // UnifiedMemoryManager reports 912MB // You can see it after enabling INFO messages for BlockManagerMasterEndpoint

INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.8:54503 with 912.3 MB RAM, BlockManagerId(driver, 192.168.1.8, 54503, None)

// custom JavaScript formatBytes function (from utils.js) reports...956.6MB // See https://github.com/apache/spark/blob/master/core/src/main/resources/org/apache/spark/ui/static/utils.js#L40-L48 def formatBytes(bytes: Double) = { val k = 1000 val i = math.floor(math.log(bytes) / math.log(k)) val maxMemoryWebUI = bytes / math.pow(k, i) f"$maxMemoryWebUI%1.1f" } scala> println(formatBytes(maxMemory)) 956.6


====

=== [[getExecInfo]] getExecInfo Method

[source, scala]

getExecInfo( listener: ExecutorsListener, statusId: Int, isActive: Boolean): ExecutorSummary


getExecInfo creates a ExecutorSummary.

CAUTION: FIXME

NOTE: getExecInfo is used when...FIXME

=== [[settings]] Settings

==== [[spark_ui_threadDumpsEnabled]] spark.ui.threadDumpsEnabled

spark.ui.threadDumpsEnabled (default: true) is to enable (true) or disable (false) <>.


Last update: 2020-10-06