spark-optimization
Optimize Apache Spark performance with partitioning, memory management, and efficient join strategies.
Install
npx skills add https://github.com/wshobson/agents --skill spark-optimizationStats
| Total installs | 3,397 |
| Weekly installs | 3.4K |
| GitHub stars | 32.4K |
| First seen | Jan 20, 2026 |
| Source | @wshobson/agents |
Summary
- Covers partitioning strategies, broadcast joins, bucketed joins, and skew handling with salting techniques to minimize shuffle overhead
- Includes caching and persistence patterns with storage level selection, checkpointing for complex lineages, and memory configuration breakdown
- Provides data format optimization for Parquet and Delta Lake, column pruning, predicate pushdown, and Z-ordering for multi-dimensional queries
- Features monitoring and debugging patterns including query plan analysis, stage metrics tracking, and partition skew detection
- Adaptive Query Execution (AQE) configuration and production-ready settings for executor memory, parallelism, serialization, and compression
Tags
Related skills
| Skill | Installs | vs spark-optimization |
|---|---|---|
| okx-dex-market | 3,451 | +54 |
| market-sizing-analysis | 3,673 | +276 |
| crawl | 3,548 | +151 |
| azure-pricing | 3,339 | -58 |
| backtesting-trading-strategies | 2,672 | -725 |
FAQ
- How many installs does spark-optimization have?
- spark-optimization has 3,397 total installs and 3.4K installs this week.
- Where is spark-optimization hosted?
- spark-optimization is published by @wshobson/agents at https://github.com/wshobson/agents.
- How many GitHub stars does spark-optimization have?
- spark-optimization has 32.4K GitHub stars.
- When was spark-optimization first indexed?
- OrangeBot.AI first indexed spark-optimization on Jan 20, 2026.
- How do I install spark-optimization?
- Run: npx skills add https://github.com/wshobson/agents --skill spark-optimization