spark-engineer

Name: spark-engineer
Author: Jeffallan

11kstars971forks

How to install spark-engineer?

jeffallan/claude-skills/spark-engineer

$npx -y skills add Jeffallan/claude-skills --skill spark-engineer

Installs into the current project.

Summary

Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines,

Files · 1

View on GitHub

SKILL.md

1	# Spark Engineer
2
3	Senior Apache Spark engineer specializing in high-performance distributed data processing, optimizing large-scale ETL pipelines, and building production-grade Spark applications.
4
5	## Core Workflow
6
7	1. Analyze requirements - Understand data volume, transformations, latency requirements, cluster resources
8	2. Design pipeline - Choose DataFrame vs RDD, plan partitioning strategy, identify broadcast opportunities
9	3. Implement - Write Spark code with optimized transformations, appropriate caching, proper error handling
10	4. Optimize - Analyze Spark UI, tune shuffle partitions, eliminate skew, optimize joins and aggregations
11	5. Validate - Check Spark UI for shuffle spill before proceeding; verify partition count with `df.rdd.getNumPartitions()`; if spill or skew detected, return to step 4; test with production-scale data, monitor resource usage, verify performance targets
12
13	## Reference Guide
14
15	Load detailed guidance based on context:
16
17	\| Topic \| Reference \| Load When \|
18	\|-------\|-----------\|-----------\|
19	\| Spark SQL & DataFrames \| `references/spark-sql-dataframes.md` \| DataFrame API, Spark SQL, schemas, joins, aggregations \|
20	\| RDD Operations \| `references/rdd-operations.md` \| Transformations, actions, pair RDDs, custom partitioners \|
21	\| Partitioning & Caching \| `references/partitioning-caching.md` \| Data partitioning, persistence levels, broadcast variables \|
22	\| Performance Tuning \| `references/performance-tuning.md` \| Configuration, memory tuning, shuffle optimization, skew handling \|
23	\| Streaming Patterns \| `references/streaming-patterns.md` \| Structured Streaming, watermarks, stateful operations, sinks \|
24
25	## Code Examples
26
27	### Quick-Start Mini-Pipeline (PySpark)
28
29	```python
30	from pyspark.sql import SparkSession
31	from pyspark.sql import functions as F
32	from pyspark.sql.types import StructType, StructField, StringType, LongType, DoubleType
33
34	spark = SparkSession.builder \
35	.appName("example-pipeline") \
36	.config("spark.sql.shuffle.partitions", "400") \
37	.config("spark.sql.adaptive.enabled", "true") \
38	.getOrCreate()
39
40	# Always define explicit schemas in production
41	schema = StructType([
42	StructField("user_id", StringType(), False),
43	StructField("event_ts", LongType(), False),
44	StructField("amount", DoubleType(), True),
45	])
46
47	df = spark.read.schema(schema).parquet("s3://bucket/events/")
48
49	result = df \
50	.filter(F.col("amount").isNotNull()) \
51	.groupBy("user_id") \
52	.agg(F.sum("amount").alias("total_amount"), F.count("*").alias("event_count"))
53
54	# Verify partition count before writing
55	print(f"Partition count: {result.rdd.getNumPartitions()}")
56
57	result.write.mode("overwrite").parquet("s3://bucket/output/")
58	```
59
60	### Broadcast Join (small dimension table < 200 MB)
61
62	```python
63	from pyspark.sql.functions import broadcast
64
65	# Spark will automatically broadcast dim_table; hint makes intent explicit
66	enriched = large_fact_df.join(broadcast(dim_df), on="product_id", how="left")
67	```
68
69	### Handling Data Skew with Salting
70
71	```python
72	import pyspark.sql.functions as F
73
74	SALT_BUCKETS = 50
75
76	# Add salt to the skewed key on both sides
77	skewed_df = skewed_df.withColumn("salt", (F.rand() * SALT_BUCKETS).cast("int")) \
78	.withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))
79
80	other_df = other_df.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(SALT_BUCKETS)]))) \
81	.withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))
82
83	result = skewed_df.join(other_df, on="salted_key", how="inner") \
84	.drop("salt", "salted_key")
85	```
86
87	### Correct Caching Pattern
88
89	```python
90	# Cache ONLY when the DataFrame is reused multiple times
91	df_cleaned = df.filter(...).withColumn(...).cache()
92	df_cleaned.count() # Materialize immediately; check Spark UI for spill
93
94	report_a = df_cleaned.groupBy("region").agg(...)
95	report_b = df_cleaned.groupBy("product").agg(...)
96
97	df_cleaned.unpersist() # Release when done
98	```
99
100	## Constraints
101
102	### MUST DO
103	- Use DataFrame API over RDD for structured data processing
104	- Define explicit schemas for production pipelines
105	- Partition data a

People who liked this also viewed

6 picks

Type

spark-engineer

11kstars971forks

View on GitHub

How to install spark-engineer?

jeffallan/claude-skills/spark-engineer

$npx -y skills add Jeffallan/claude-skills --skill spark-engineer

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Spark Engineer
2
3	Senior Apache Spark engineer specializing in high-performance distributed data processing, optimizing large-scale ETL pipelines, and building production-grade Spark applications.
4
5	## Core Workflow
6
7	1. Analyze requirements - Understand data volume, transformations, latency requirements, cluster resources
8	2. Design pipeline - Choose DataFrame vs RDD, plan partitioning strategy, identify broadcast opportunities
9	3. Implement - Write Spark code with optimized transformations, appropriate caching, proper error handling
10	4. Optimize - Analyze Spark UI, tune shuffle partitions, eliminate skew, optimize joins and aggregations
11	5. Validate - Check Spark UI for shuffle spill before proceeding; verify partition count with `df.rdd.getNumPartitions()`; if spill or skew detected, return to step 4; test with production-scale data, monitor resource usage, verify performance targets
12
13	## Reference Guide
14
15	Load detailed guidance based on context:
16
17	\| Topic \| Reference \| Load When \|
18	\|-------\|-----------\|-----------\|
19	\| Spark SQL & DataFrames \| `references/spark-sql-dataframes.md` \| DataFrame API, Spark SQL, schemas, joins, aggregations \|
20	\| RDD Operations \| `references/rdd-operations.md` \| Transformations, actions, pair RDDs, custom partitioners \|
21	\| Partitioning & Caching \| `references/partitioning-caching.md` \| Data partitioning, persistence levels, broadcast variables \|
22	\| Performance Tuning \| `references/performance-tuning.md` \| Configuration, memory tuning, shuffle optimization, skew handling \|
23	\| Streaming Patterns \| `references/streaming-patterns.md` \| Structured Streaming, watermarks, stateful operations, sinks \|
24
25	## Code Examples
26
27	### Quick-Start Mini-Pipeline (PySpark)
28
29	```python
30	from pyspark.sql import SparkSession
31	from pyspark.sql import functions as F
32	from pyspark.sql.types import StructType, StructField, StringType, LongType, DoubleType
33
34	spark = SparkSession.builder \
35	.appName("example-pipeline") \
36	.config("spark.sql.shuffle.partitions", "400") \
37	.config("spark.sql.adaptive.enabled", "true") \
38	.getOrCreate()
39
40	# Always define explicit schemas in production
41	schema = StructType([
42	StructField("user_id", StringType(), False),
43	StructField("event_ts", LongType(), False),
44	StructField("amount", DoubleType(), True),
45	])
46
47	df = spark.read.schema(schema).parquet("s3://bucket/events/")
48
49	result = df \
50	.filter(F.col("amount").isNotNull()) \
51	.groupBy("user_id") \
52	.agg(F.sum("amount").alias("total_amount"), F.count("*").alias("event_count"))
53
54	# Verify partition count before writing
55	print(f"Partition count: {result.rdd.getNumPartitions()}")
56
57	result.write.mode("overwrite").parquet("s3://bucket/output/")
58	```
59
60	### Broadcast Join (small dimension table < 200 MB)
61
62	```python
63	from pyspark.sql.functions import broadcast
64
65	# Spark will automatically broadcast dim_table; hint makes intent explicit
66	enriched = large_fact_df.join(broadcast(dim_df), on="product_id", how="left")
67	```
68
69	### Handling Data Skew with Salting
70
71	```python
72	import pyspark.sql.functions as F
73
74	SALT_BUCKETS = 50
75
76	# Add salt to the skewed key on both sides
77	skewed_df = skewed_df.withColumn("salt", (F.rand() * SALT_BUCKETS).cast("int")) \
78	.withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))
79
80	other_df = other_df.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(SALT_BUCKETS)]))) \
81	.withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))
82
83	result = skewed_df.join(other_df, on="salted_key", how="inner") \
84	.drop("salt", "salted_key")
85	```
86
87	### Correct Caching Pattern
88
89	```python
90	# Cache ONLY when the DataFrame is reused multiple times
91	df_cleaned = df.filter(...).withColumn(...).cache()
92	df_cleaned.count() # Materialize immediately; check Spark UI for spill
93
94	report_a = df_cleaned.groupBy("region").agg(...)
95	report_b = df_cleaned.groupBy("product").agg(...)
96
97	df_cleaned.unpersist() # Release when done
98	```
99
100	## Constraints
101
102	### MUST DO
103	- Use DataFrame API over RDD for structured data processing
104	- Define explicit schemas for production pipelines
105	- Partition data a

People who liked this also viewed

6 picks

Type