Skip to content

spark.python Configuration Properties

daemon.module

spark.python.daemon.module

The Python module to run the daemon to execute Python workers

Default: pyspark.daemon

Used when:

use.daemon

spark.python.use.daemon

Because forking processes from Java is expensive, PySpark prefers launching a single Python daemon (spark.python.daemon.module) to fork new workers for tasks. This daemon currently only works on UNIX-based systems now because it uses signals for child management, so we can also fall back to launching workers (spark.python.worker.module) directly.

Default: true (unless PySpark runs on Windows)

Used when:

worker.module

spark.python.worker.module

The Python module to run a Python worker

Default: pyspark.worker

Used when: