aidp-spark-optimization
Use when a Spark job/notebook is slow, missing an SLA, spilling, OOMing, generating too many small files, or shuffling/skewing heavily; when reviewing Spark/PySpark code or a Spark UI for performance; or before running a large Spark workload. Covers open-source Apache Spark 3.5.0 (+ Delta Lake) tuning -- partitions, shuffle, joins, skew, file layout, memory, codegen, caching, AQE, compression, and configuration.
Pinned to revision 4e8964e24a5a, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/aidp-spark-optimization/SKILL.md
- skills/aidp-spark-optimization/LICENSE
- skills/aidp-spark-optimization/references/01-partitioning.md
- skills/aidp-spark-optimization/references/02-joins.md
- skills/aidp-spark-optimization/references/03-file-layout-io.md
- skills/aidp-spark-optimization/references/04-memory-and-spill.md
- skills/aidp-spark-optimization/references/05-codegen.md
- skills/aidp-spark-optimization/references/06-caching-materialization.md
- skills/aidp-spark-optimization/references/07-aqe.md
- skills/aidp-spark-optimization/references/08-delta-lake.md
- skills/aidp-spark-optimization/references/09-oracle-database.md
- skills/aidp-spark-optimization/references/aidp-notes.md
- skills/aidp-spark-optimization/references/case-studies.md
- skills/aidp-spark-optimization/references/cluster-sizing.md
- skills/aidp-spark-optimization/references/config-matrix.md
- skills/aidp-spark-optimization/references/diagnosis.md
- skills/aidp-spark-optimization/references/quick-reference.md
Every link opens the file at its source, pinned to the revision this page describes.