spark.python Configuration Properties¶
daemon.module¶
spark.python.daemon.module
The Python module to run the daemon to execute Python workers
Default: pyspark.daemon
Used when:
PythonWorkerFactory
is created
use.daemon¶
spark.python.use.daemon
Because forking processes from Java is expensive, PySpark prefers launching a single Python daemon (spark.python.daemon.module) to fork new workers for tasks. This daemon currently only works on UNIX-based systems now because it uses signals for child management, so we can also fall back to launching workers (spark.python.worker.module) directly.
Default: true
(unless PySpark runs on Windows)
Used when:
PythonWorkerFactory
is created
worker.module¶
spark.python.worker.module
The Python module to run a Python worker
Default: pyspark.worker
Used when:
PythonWorkerFactory
is created