Demo: Debugging Delta Lake Using IntelliJ IDEA

Import Delta Lake’s sources to IntelliJ IDEA.

Configure a new Remote debug configuration in IntelliJ IDEA (e.g. Run > Debug > Edit Configurations…​) and simply give it a name and save.

Use Option+Ctrl+D to access Debug menu.
demo remote jvm
Figure 1. Remote JVM Configuration

Run spark-shell as follows to enable remote JVM for debugging.

$ export SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
$ spark-shell \
  --packages io.delta:delta-core_2.12:0.5.0 \
  --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
  --conf spark.databricks.delta.snapshotPartitions=1