Environment Variables¶
PySpark uses environment variables to configure execution environment.
PYSPARK_DRIVER_PYTHON¶
The Python Executable in PySpark Connect unless PYSPARK_PYTHON is defined
Default: python3
PYSPARK_GATEWAY_PORT¶
PYSPARK_GATEWAY_SECRET¶
PYSPARK_PIN_THREAD¶
Enables pinned thread mode to synchronize PVM threads with JVM threads based on Py4J's ClientServer (true) or GatewayServer (false)
Default: false
Used when:
- launch_gateway is executed
- Py4JServer is created (and initializes the server)
PYSPARK_PYTHON¶
The Python Executable
Default: python3