spark-optimization

Name: spark-optimization
Author: wshobson

8.2kinstalls4.1kforks

How to install spark-optimization?

wshobson/agents/spark-optimization

$npx -y skills add wshobson/agents --skill spark-optimization

Installs into the current project.

Summary

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

Files · 1

View on GitHub

SKILL.md

1	# Apache Spark Optimization
2
3	Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
4
5	## When to Use This Skill
6
7	- Optimizing slow Spark jobs
8	- Tuning memory and executor configuration
9	- Implementing efficient partitioning strategies
10	- Debugging Spark performance issues
11	- Scaling Spark pipelines for large datasets
12	- Reducing shuffle and data skew
13
14	## Core Concepts
15
16	### 1. Spark Execution Model
17
18	```
19	Driver Program
20	↓
21	Job (triggered by action)
22	↓
23	Stages (separated by shuffles)
24	↓
25	Tasks (one per partition)
26	```
27
28	### 2. Key Performance Factors
29
30	\| Factor \| Impact \| Solution \|
31	\| ----------------- \| --------------------- \| ----------------------------- \|
32	\| Shuffle \| Network I/O, disk I/O \| Minimize wide transformations \|
33	\| Data Skew \| Uneven task duration \| Salting, broadcast joins \|
34	\| Serialization \| CPU overhead \| Use Kryo, columnar formats \|
35	\| Memory \| GC pressure, spills \| Tune executor memory \|
36	\| Partitions \| Parallelism \| Right-size partitions \|
37
38	## Quick Start
39
40	```python
41	from pyspark.sql import SparkSession
42	from pyspark.sql import functions as F
43
44	# Create optimized Spark session
45	spark = (SparkSession.builder
46	.appName("OptimizedJob")
47	.config("spark.sql.adaptive.enabled", "true")
48	.config("spark.sql.adaptive.coalescePartitions.enabled", "true")
49	.config("spark.sql.adaptive.skewJoin.enabled", "true")
50	.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
51	.config("spark.sql.shuffle.partitions", "200")
52	.getOrCreate())
53
54	# Read with optimized settings
55	df = (spark.read
56	.format("parquet")
57	.option("mergeSchema", "false")
58	.load("s3://bucket/data/"))
59
60	# Efficient transformations
61	result = (df
62	.filter(F.col("date") >= "2024-01-01")
63	.select("id", "amount", "category")
64	.groupBy("category")
65	.agg(F.sum("amount").alias("total")))
66
67	result.write.mode("overwrite").parquet("s3://bucket/output/")
68	```
69
70	## Detailed patterns and worked examples
71
72	Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.
73
74	## Best Practices
75
76	### Do's
77
78	- Enable AQE - Adaptive query execution handles many issues
79	- Use Parquet/Delta - Columnar formats with compression
80	- Broadcast small tables - Avoid shuffle for small joins
81	- Monitor Spark UI - Check for skew, spills, GC
82	- Right-size partitions - 128MB - 256MB per partition
83
84	### Don'ts
85
86	- Don't collect large data - Keep data distributed
87	- Don't use UDFs unnecessarily - Use built-in functions
88	- Don't over-cache - Memory is limited
89	- Don't ignore data skew - It dominates job time
90	- Don't use `.count()` for existence - Use `.take(1)` or `.isEmpty()`

People who liked this also viewed

6 picks

Type

spark-optimization

8.2kinstalls4.1kforks

View on GitHub

How to install spark-optimization?

wshobson/agents/spark-optimization

$npx -y skills add wshobson/agents --skill spark-optimization

Installs into the current project.

Summary

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

Files · 1

View on GitHub

SKILL.md

1	# Apache Spark Optimization
2
3	Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
4
5	## When to Use This Skill
6
7	- Optimizing slow Spark jobs
8	- Tuning memory and executor configuration
9	- Implementing efficient partitioning strategies
10	- Debugging Spark performance issues
11	- Scaling Spark pipelines for large datasets
12	- Reducing shuffle and data skew
13
14	## Core Concepts
15
16	### 1. Spark Execution Model
17
18	```
19	Driver Program
20	↓
21	Job (triggered by action)
22	↓
23	Stages (separated by shuffles)
24	↓
25	Tasks (one per partition)
26	```
27
28	### 2. Key Performance Factors
29
30	\| Factor \| Impact \| Solution \|
31	\| ----------------- \| --------------------- \| ----------------------------- \|
32	\| Shuffle \| Network I/O, disk I/O \| Minimize wide transformations \|
33	\| Data Skew \| Uneven task duration \| Salting, broadcast joins \|
34	\| Serialization \| CPU overhead \| Use Kryo, columnar formats \|
35	\| Memory \| GC pressure, spills \| Tune executor memory \|
36	\| Partitions \| Parallelism \| Right-size partitions \|
37
38	## Quick Start
39
40	```python
41	from pyspark.sql import SparkSession
42	from pyspark.sql import functions as F
43
44	# Create optimized Spark session
45	spark = (SparkSession.builder
46	.appName("OptimizedJob")
47	.config("spark.sql.adaptive.enabled", "true")
48	.config("spark.sql.adaptive.coalescePartitions.enabled", "true")
49	.config("spark.sql.adaptive.skewJoin.enabled", "true")
50	.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
51	.config("spark.sql.shuffle.partitions", "200")
52	.getOrCreate())
53
54	# Read with optimized settings
55	df = (spark.read
56	.format("parquet")
57	.option("mergeSchema", "false")
58	.load("s3://bucket/data/"))
59
60	# Efficient transformations
61	result = (df
62	.filter(F.col("date") >= "2024-01-01")
63	.select("id", "amount", "category")
64	.groupBy("category")
65	.agg(F.sum("amount").alias("total")))
66
67	result.write.mode("overwrite").parquet("s3://bucket/output/")
68	```
69
70	## Detailed patterns and worked examples
71
72	Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.
73
74	## Best Practices
75
76	### Do's
77
78	- Enable AQE - Adaptive query execution handles many issues
79	- Use Parquet/Delta - Columnar formats with compression
80	- Broadcast small tables - Avoid shuffle for small joins
81	- Monitor Spark UI - Check for skew, spills, GC
82	- Right-size partitions - 128MB - 256MB per partition
83
84	### Don'ts
85
86	- Don't collect large data - Keep data distributed
87	- Don't use UDFs unnecessarily - Use built-in functions
88	- Don't over-cache - Memory is limited
89	- Don't ignore data skew - It dominates job time
90	- Don't use `.count()` for existence - Use `.take(1)` or `.isEmpty()`

People who liked this also viewed

6 picks

Type