Skip to content is a Python module in pyspark package.

from pyspark import daemon

Entry Point

Top-Level Code Environment

If the module is executed in the top-level code environment (e.g., python -m), its __name__ is set to the string __main__.

Sometimes "top-level code" is called an entry point to the application.

Learn more in the __main__ — Top-level code environment.

When executed in the top-level code environment, calls manager function.



manager runs until it is stopped (e.g., CTRL-C).

manager creates a new process group (os.setpgid(0, 0)).

manager creates a listening socket on the loopback interface (possibly using IPv6 based on SPARK_PREFER_IPV6 environment variable).

manager reads SPARK_REUSE_WORKER environment variable (reuse).

manager launches a worker process (in a child process using os.fork()).

Launching Worker Process

  sock: socket,
  authenticated: Bool) -> Optional[int]


worker is called by a worker process after theos.fork().

worker runs a worker.