spark-optimization

SKILL.md

Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.

| Shuffle | Network I/O, disk I/O | Minimize wide transformations | | Data Skew | Uneven task duration | Salting, broadcast joins | | Serialization | CPU overhead | Use Kryo, columnar formats | | Memory | GC pressure, spills | Tune executor memory | | Partitions | Parallelism | Right-size partitions |

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines. Source: wshobson/agents.

Open your terminal or command line tool (Terminal, iTerm, Windows Terminal, etc.) Copy and run this command: npx skills add https://github.com/wshobson/agents --skill spark-optimization Once installed, the skill will be automatically configured in your AI coding environment and ready to use in Claude Code or Cursor

View raw