Spark Connect ¶

Apache Spark 3.4 introduces Spark Connect module for a client-server interface for Apache Spark for remote connectivity to Spark clusters (using the DataFrame API and unresolved logical plans as the protocol based on gRPC Java).

The Spark Connect server can be started using sbin/start-connect-server.sh shell script.

$ ./sbin/start-connect-server.sh
starting org.apache.spark.sql.connect.service.SparkConnectServer, logging to...

$ tail -1 logs/spark-jacek-org.apache.spark.sql.connect.service.SparkConnectServer-1-Jaceks-Mac-mini.local.out
... Spark Connect server started.