PythonGatewayServer is the Python runner for
pyspark shell (Apache Spark).
main( args: Array[String]): Unit
main requests the
Py4JServer for the listening port (boundPort) and prints out the following DEBUG message to the logs:
Started PythonGatewayServer on port [boundPort]
main pauses (blocks) until the Python driver finishes (by reading from the system input that blocks until input data is available, the end of the stream is detected, or an exception is thrown).
In the end, once the Python driver finishes,
main prints out the following DEBUG message to the logs:
Exiting due to broken pipe from Python driver
main prints out the following ERROR message to the logs and exists when the listening port is
[server] failed to bind; exiting
_PYSPARK_DRIVER_CONN_INFO_PATH environment variable for the path of a connection info file for communication between this and the Python processes.
ALL logging level for
org.apache.spark.api.python.PythonGatewayServer logger to see what happens inside.
Add the following line to
Refer to Logging.