Spark Performance Deep Dive on Databricks: Shuffle Tuning, Skew Handling, and Z-Ordering with Delta Lake + Unity Catalog
The article argues that most Spark performance issues on Databricks are not solved by scaling the cluster but are caused by shuffle and skew, and no amount of extra nodes fixes a badly partitioned join. It builds a realistic pipeline from the ground up: order events joined against a small dimension table, aggregated, and written to Delta Lake. The post works through how Spark's shuffle actually behaves during a wide transformation, diagnosing and fixing data skew with salting and adaptive query execution (AQE), and laying out the resulting Delta table with Z-Ordering so downstream queries skip irrelevant files. It also covers governing access to the whole pipeline with Unity Catalog, which provides centralized access control and lineage instead of per-workspace table grants. The pipeline shape is a batch job reading raw events, joining against a dimension table, aggregating, and writing to a governed Delta table. The article includes a setup.sql script to be run in a Databricks SQL or notebook cell.
Understanding shuffle and skew tuning is critical for efficient Spark jobs on Databricks.