Skip to content


SparkContext Initialization

Creating Instance

SparkContext takes the following to be created:

  • Master URL (default: None)
  • Application Name (default: None)
  • Spark Home (default: None)
  • Py Files (default: None)
  • Environment (default: None)
  • Batch Size (default: 0)
  • PickleSerializer
  • SparkConf (default: None)
  • Gateway (default: None)
  • Corresponding SparkContext on JVM (default: None)
  • BasicProfiler

While being created, SparkContext _ensure_initialized (with the gateway and the conf) followed by _do_init.


from pyspark import SparkContext


SparkContext defines _gateway property for a JavaGateway that is given or launched when _ensure_initialized.


SparkContext defines _jvm property for a JVMView (py4j) to access to the Java Virtual Machine of the JavaGateway.


  cls, instance=None, gateway=None, conf=None)

_ensure_initialized is a @classmethod.

_ensure_initialized takes the given gateway or launch_gateway.


_ensure_initialized is used when:


  self, master, appName, sparkHome,
  pyFiles, environment, batchSize, serializer,
  conf, jsc, profiler_cls)


Last update: 2021-03-04