daemon.py¶
daemon.py is a Python module in pyspark package.
from pyspark import daemon
Entry Point¶
Top-Level Code Environment
If the module is executed in the top-level code environment (e.g., python -m), its __name__ is set to the string __main__.
Sometimes "top-level code" is called an entry point to the application.
Learn more in the __main__ — Top-level code environment.
When executed in the top-level code environment, daemon.py calls manager function.
manager¶
manager()
manager runs until it is stopped (e.g., CTRL-C).
manager creates a new process group (os.setpgid(0, 0)).
manager creates a listening socket on the loopback interface (possibly using IPv6 based on SPARK_PREFER_IPV6 environment variable).
manager reads SPARK_REUSE_WORKER environment variable (reuse).
manager launches a worker process (in a child process using os.fork()).
Launching Worker Process¶
worker(
sock: socket,
authenticated: Bool) -> Optional[int]
Note
worker is called by a worker process after theos.fork().
worker runs a worker.