{"id":"spark-optimization","name":"spark-optimization","summary":"Apache Sparkジョブをパーティション、キャッシュ、シャッフル最適化、メモリチューニングで最適化します。","body":"# Apache Spark Optimization\n\nProduction patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.\n\n## When to Use This Skill\n\n- Optimizing slow Spark jobs\n- Tuning memory and executor configuration\n- Implementing efficient partitioning strategies\n- Debugging Spark performance issues\n- Scaling Spark pipelines for large datasets\n- Reducing shuffle and data skew\n\n## Core Concepts\n\n### 1. Spark Execution Model\n\n```\nDriver Program\n    ↓\nJob (triggered by action)\n    ↓\nStages (separated by shuffles)\n    ↓\nTasks (one per partition)\n```\n\n### 2. Key Performance Factors\n\n| Factor            | Impact                | Solution                      |\n| ----------------- | --------------------- | ----------------------------- |\n| **Shuffle**       | Network I/O, disk I/O | Minimize wide transformations |\n| **Data Skew**     | Uneven task duration  | Salting, broadcast joins      |\n| **Serialization** | CPU overhead          | Use Kryo, columnar formats    |\n| **Memory**        | GC pressure, spills   | Tune executor memory          |\n| **Partitions**    | Parallelism           | Right-size partitions         |\n\n## Quick Start\n\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql import functions as F\n\n# Create optimized Spark session\nspark = (SparkSession.builder\n    .appName(\"OptimizedJob\")\n    .config(\"spark.sql.adaptive.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\n    .config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    .getOrCreate())\n\n# Read with optimized settings\ndf = (spark.read\n    .format(\"parquet\")\n    .option(\"mergeSchema\", \"false\")\n    .load(\"s3://bucket/data/\"))\n\n# Efficient transformations\nresult = (df\n    .filter(F.col(\"date\") >= \"2024-01-01\")\n    .select(\"id\", \"amount\", \"category\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\").alias(\"total\")))\n\nresult.write.mode(\"overwrite\").parquet(\"s3://bucket/output/\")\n```\n\n## Detailed patterns and worked examples\n\nDetailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.\n\n## Best Practices\n\n### Do's\n\n- **Enable AQE** - Adaptive query execution handles many issues\n- **Use Parquet/Delta** - Columnar formats with compression\n- **Broadcast small tables** - Avoid shuffle for small joins\n- **Monitor Spark UI** - Check for skew, spills, GC\n- **Right-size partitions** - 128MB - 256MB per partition\n\n### Don'ts\n\n- **Don't collect large data** - Keep data distributed\n- **Don't use UDFs unnecessarily** - Use built-in functions\n- **Don't over-cache** - Memory is limited\n- **Don't ignore data skew** - It dominates job time\n- **Don't use `.count()` for existence** - Use `.take(1)` or `.isEmpty()`","author":"@wshobson","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/wshobson/agents/tree/main/plugins/data-engineering/skills/spark-optimization","license":"MIT","category":"coding","lang":"en","tokens":682,"stars":0,"calls30d":2,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/details.md","size":9844,"sha256":"1ea455732366d868059dfba807169336df1db75acf4e7f558aee3b41b7f8d210"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":[]}}