gcp-spark
Develops and executes Spark code on Dataproc Clusters and Serverless. Reads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner. Debugs execution failures. Use when: - Writing Spark ETL pipelines on GCP. - Training or running inference with ML models with spark on GCP. - Managing Spark clusters, jobs, batches, and interactive sessions. Don't use when: - Writing generic Python scripts that don't use Spark. - Performing simple SQL queries that can be done directly in BigQuery.
- Version
- v2
- License
- Apache-2.0
Pinned to revision db28b191f5db, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/gcp-spark/SKILL.md
- skills/gcp-spark/references/gcloud_dataproc.md
- skills/gcp-spark/references/ml_tasks.md
- skills/gcp-spark/references/read_write_data.md
- skills/gcp-spark/references/schema_direct_inspection.md
- skills/gcp-spark/references/spark_optimizations.md
Every link opens the file at its source, pinned to the revision this page describes.