Demo: Debugging Delta Lake Using IntelliJ IDEA¶

Import Delta Lake's sources to IntelliJ IDEA.

Configure a new Remote debug configuration in IntelliJ IDEA (e.g. Run > Debug > Edit Configurations...) and simply give it a name and save.

Tip

Use Option+Ctrl+D to access Debug menu on mac OS.

Remote JVM Configuration

Run spark-shell as follows to enable remote JVM for debugging.

export SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005

spark-shell \
  --packages io.delta:delta-core_2.12:3.2.1 \
  --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
  --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \
  --conf spark.databricks.delta.snapshotPartitions=1