launch_gateway( conf=None, popen_kwargs=None)
launch_gateway reads PYSPARK_GATEWAY_PORT and PYSPARK_GATEWAY_SECRET environment variables if defined and assumes that the child Java gateway process has already been started (e.g. PythonGatewayServer).
launch_gateway builds the command to start
- Appends all the configuration properties (from the input
PYSPARK_SUBMIT_ARGSenvironment variable if defined or assumes
launch_gateway sets up
_PYSPARK_DRIVER_CONN_INFO_PATH environment variable to point at an unique temporary file.
launch_gateway configures a pipe to stdin for the corresponding Java gateway process to use to monitor the Python process.
bin/spark-submit command and waits for a connection info file to be created at
launch_gateway reads the port and the secret from the file once available.
launch_gateway connects to the gateway using py4j's
JavaGateway based on PYSPARK_PIN_THREAD environment variable.
launch_gateway imports Spark packages and classes (using py4j):
launch_gateway is used when:
SparkContextis requested to _ensure_initialized