fabric-pyspark-perf-remediate

Name: fabric-pyspark-perf-remediate
Author: PatrickGallucci

16stars1forks

How to install fabric-pyspark-perf-remediate?

patrickgallucci/fabric-skills/fabric-pyspark-perf-remediate

$npx -y skills add PatrickGallucci/fabric-skills --skill fabric-pyspark-perf-remediate

Installs into the current project.

Summary

Diagnose and resolve Apache Spark performance issues in Microsoft Fabric notebooks and Spark Job Definitions. Use when PySpark jobs are slow, notebooks take too long, Spark stages are skewed, shuffles are excessive, out-of-memory errors occur, Delta Lake writes are slow, or Fabri

Files · 1

View on GitHub

SKILL.md

1	# Microsoft Fabric PySpark Performance remediate
2
3	Systematic guide for diagnosing and resolving Apache Spark performance problems in Microsoft Fabric Data Engineering workloads, including notebooks, Spark Job Definitions, and pipeline activities.
4
5	## When to Use This Skill
6
7	Activate when encountering any of these scenarios:
8
9	- PySpark notebook cells take unexpectedly long to execute
10	- Spark Job Definitions exceed expected duration or fail with timeouts
11	- Out-of-memory (OOM) errors on driver or executors
12	- Excessive shuffle read/write in Spark UI stage details
13	- Data skew causing individual tasks to run much longer than peers
14	- Delta Lake table writes are slow or produce many small files
15	- Fabric capacity utilization is high or jobs are queued/throttled
16	- Need to choose between resource profiles (readHeavy vs writeHeavy)
17	- Deciding whether to enable autotune, native execution engine, or Optimized Write
18	- Interpreting Spark UI metrics (stages, tasks, storage, SQL plan)
19
20	## Prerequisites
21
22	- Access to a Microsoft Fabric workspace with Data Engineering/Science experience
23	- Fabric capacity (F2 or higher) with Spark compute enabled
24	- Familiarity with PySpark DataFrames and Spark SQL
25	- Access to Spark UI via the Monitoring Hub or notebook session details
26
27	## Quick Diagnostic Workflow
28
29	Follow this triage sequence to identify the root cause:
30
31	1. Check capacity status - Is the Fabric capacity throttled or overloaded? See Monitoring Hub for queued jobs and CU utilization.
32	2. Identify the slow stage - Open Spark UI, find the stage with the longest duration, and check task-level metrics.
33	3. Classify the bottleneck - Use the decision matrix below to categorize the issue.
34	4. Apply targeted fix - Follow the relevant reference guide for your bottleneck type.
35	5. Validate improvement - Re-run the job and compare Spark UI metrics before and after.
36
37	## Bottleneck Decision Matrix
38
39	Use these indicators to classify your performance issue:
40
41	Shuffle Bottleneck: Shuffle read/write bytes are large (>1 GB per stage), many tasks in the stage, high GC time. Fix with broadcast joins, reduced shuffle partitions, or pre-partitioned data. See [shuffle-and-join-optimization.md](./references/shuffle-and-join-optimization.md).
42
43	Data Skew: One or few tasks take 10x longer than median, input size per task is highly uneven, some tasks process GBs while others process MBs. Fix with salting, repartitioning, or adaptive query execution. See [data-skew-resolution.md](./references/data-skew-resolution.md).
44
45	Memory Pressure / OOM: Executor lost errors, "Container killed by YARN for exceeding memory limits", excessive GC time (>10% of task time), disk spill. Fix with memory configuration, caching strategy, or partition sizing. See [memory-and-spill-tuning.md](./references/memory-and-spill-tuning.md).
46
47	Small Files / Delta Compaction: Write operations produce thousands of small Parquet files, reads are slow due to file listing overhead, table maintenance never run. Fix with OPTIMIZE, Optimized Write, or streaming trigger intervals. See [delta-table-optimization.md](./references/delta-table-optimization.md).
48
49	Capacity / Compute Sizing: Jobs queued for long periods, capacity utilization consistently above 80%, throttling messages in monitoring. Fix with pool sizing, resource profiles, or autoscale billing. See [fabric-compute-tuning.md](./references/fabric-compute-tuning.md).
50
51	## Key Spark Configuration Quick Reference
52
53	### Session-Level Overrides (Notebook Cell 1)
54
55	```python
56	# Shuffle partition count - default is 200, tune to data volume
57	spark.conf.set("spark.sql.shuffle.partitions", "auto")
58
59	# Broadcast join threshold - default 10MB, increase for medium tables
60	spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "50m")
61
62	# Max partition bytes - default 128MB, increase for fewer larger partitions
63	spark.conf.set("spark.sql.files.maxPartitionBytes", "256m")
64
65	# Enable Optimized Write for Delta
66	spark.conf.set("spark.microsoft.delta.optimizeWrite.enabled", "true")
67
68	# Enable autotune (preview, Runtime 1.1/1.2 only)
69	spark.conf.set("s

People who liked this also viewed

6 picks

Type

fabric-pyspark-perf-remediate

16stars1forks

View on GitHub

How to install fabric-pyspark-perf-remediate?

patrickgallucci/fabric-skills/fabric-pyspark-perf-remediate

$npx -y skills add PatrickGallucci/fabric-skills --skill fabric-pyspark-perf-remediate

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Microsoft Fabric PySpark Performance remediate
2
3	Systematic guide for diagnosing and resolving Apache Spark performance problems in Microsoft Fabric Data Engineering workloads, including notebooks, Spark Job Definitions, and pipeline activities.
4
5	## When to Use This Skill
6
7	Activate when encountering any of these scenarios:
8
9	- PySpark notebook cells take unexpectedly long to execute
10	- Spark Job Definitions exceed expected duration or fail with timeouts
11	- Out-of-memory (OOM) errors on driver or executors
12	- Excessive shuffle read/write in Spark UI stage details
13	- Data skew causing individual tasks to run much longer than peers
14	- Delta Lake table writes are slow or produce many small files
15	- Fabric capacity utilization is high or jobs are queued/throttled
16	- Need to choose between resource profiles (readHeavy vs writeHeavy)
17	- Deciding whether to enable autotune, native execution engine, or Optimized Write
18	- Interpreting Spark UI metrics (stages, tasks, storage, SQL plan)
19
20	## Prerequisites
21
22	- Access to a Microsoft Fabric workspace with Data Engineering/Science experience
23	- Fabric capacity (F2 or higher) with Spark compute enabled
24	- Familiarity with PySpark DataFrames and Spark SQL
25	- Access to Spark UI via the Monitoring Hub or notebook session details
26
27	## Quick Diagnostic Workflow
28
29	Follow this triage sequence to identify the root cause:
30
31	1. Check capacity status - Is the Fabric capacity throttled or overloaded? See Monitoring Hub for queued jobs and CU utilization.
32	2. Identify the slow stage - Open Spark UI, find the stage with the longest duration, and check task-level metrics.
33	3. Classify the bottleneck - Use the decision matrix below to categorize the issue.
34	4. Apply targeted fix - Follow the relevant reference guide for your bottleneck type.
35	5. Validate improvement - Re-run the job and compare Spark UI metrics before and after.
36
37	## Bottleneck Decision Matrix
38
39	Use these indicators to classify your performance issue:
40
41	Shuffle Bottleneck: Shuffle read/write bytes are large (>1 GB per stage), many tasks in the stage, high GC time. Fix with broadcast joins, reduced shuffle partitions, or pre-partitioned data. See [shuffle-and-join-optimization.md](./references/shuffle-and-join-optimization.md).
42
43	Data Skew: One or few tasks take 10x longer than median, input size per task is highly uneven, some tasks process GBs while others process MBs. Fix with salting, repartitioning, or adaptive query execution. See [data-skew-resolution.md](./references/data-skew-resolution.md).
44
45	Memory Pressure / OOM: Executor lost errors, "Container killed by YARN for exceeding memory limits", excessive GC time (>10% of task time), disk spill. Fix with memory configuration, caching strategy, or partition sizing. See [memory-and-spill-tuning.md](./references/memory-and-spill-tuning.md).
46
47	Small Files / Delta Compaction: Write operations produce thousands of small Parquet files, reads are slow due to file listing overhead, table maintenance never run. Fix with OPTIMIZE, Optimized Write, or streaming trigger intervals. See [delta-table-optimization.md](./references/delta-table-optimization.md).
48
49	Capacity / Compute Sizing: Jobs queued for long periods, capacity utilization consistently above 80%, throttling messages in monitoring. Fix with pool sizing, resource profiles, or autoscale billing. See [fabric-compute-tuning.md](./references/fabric-compute-tuning.md).
50
51	## Key Spark Configuration Quick Reference
52
53	### Session-Level Overrides (Notebook Cell 1)
54
55	```python
56	# Shuffle partition count - default is 200, tune to data volume
57	spark.conf.set("spark.sql.shuffle.partitions", "auto")
58
59	# Broadcast join threshold - default 10MB, increase for medium tables
60	spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "50m")
61
62	# Max partition bytes - default 128MB, increase for fewer larger partitions
63	spark.conf.set("spark.sql.files.maxPartitionBytes", "256m")
64
65	# Enable Optimized Write for Delta
66	spark.conf.set("spark.microsoft.delta.optimizeWrite.enabled", "true")
67
68	# Enable autotune (preview, Runtime 1.1/1.2 only)
69	spark.conf.set("s

People who liked this also viewed

6 picks

Type