gcp-spark
Authors, optimizes, and runs PySpark and Spark code on Google Cloud Managed Spark (Dataproc Serverless, clusters, and Spark Connect). Use when: - Authoring or running Spark/PySpark notebooks in any kernel (Local Kernel with spark-remote templates, Remote Spark / Remote Kernel, ManagedSparkSession, Spark Connect). - Querying, transforming, or writing data with PySpark or Spark SQL, including Apache / BigLake Iceberg, BigQuery, GCS, Spanner, Cloud SQL, AlloyDB, Amazon S3, or Azure Blob Storage / ADLS Gen2. - Building PySpark ETL pipelines, machine learning workloads, or batch jobs. - Managing Dataproc clusters, serverless batches, or interactive sessions. - Optimizing PySpark or Spark SQL queries for performance, memory, and OOM prevention. Do NOT use when: - Auditing multi-region fleet inventory, DCU/cost consumption, idle/zombie clusters, or workload observability (use @skill:gcp-spark-observability). - Troubleshooting failed Spark jobs or Dataproc batches (use @skill:gcp-spark-troubleshooting).
- Version
- v35
- License
- Apache-2.0
Pinned to revision 716f429da267, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/gcp-spark/SKILL.md
- skills/gcp-spark/references/aws_s3.md
- skills/gcp-spark/references/azure_storage.md
- skills/gcp-spark/references/biglake_iceberg.md
- skills/gcp-spark/references/gcloud_dataproc.md
- skills/gcp-spark/references/jdbc_cloudsql.md
- skills/gcp-spark/references/ml_tasks.md
- skills/gcp-spark/references/read_write_data.md
- skills/gcp-spark/references/schema_direct_inspection.md
- skills/gcp-spark/references/spanner.md
- skills/gcp-spark/references/spark_optimizations.md
- skills/gcp-spark/references/spark_refactoring_guide.md
Every link opens the file at its source, pinned to the revision this page describes.