fabric-spark-perf-remediate

Name: fabric-spark-perf-remediate
Author: PatrickGallucci

16stars1forks

How to install fabric-spark-perf-remediate?

patrickgallucci/fabric-skills/fabric-spark-perf-remediate

$npx -y skills add PatrickGallucci/fabric-skills --skill fabric-spark-perf-remediate

Installs into the current project.

Summary

Diagnose and resolve Apache Spark performance issues in Microsoft Fabric. Use when asked to troubleshoot slow Spark notebooks, optimize Spark SQL queries, fix data skew or shuffle bottlenecks, tune spark.sql.shuffle.partitions or autoBroadcastJoinThreshold, configure resource pro

Files · 1

View on GitHub

SKILL.md

1	# Microsoft Fabric Apache Spark Performance remediate
2
3	Systematic workflows for diagnosing, analyzing, and resolving Apache Spark performance problems in Microsoft Fabric Data Engineering and Data Science workloads.
4
5	## When to Use This Skill
6
7	Activate when encountering any of the following scenarios:
8
9	- Spark notebooks or jobs running slower than expected
10	- Capacity throttling errors (HTTP 430 / TooManyRequestsForCapacity)
11	- Data skew detected by Spark Advisor in notebook cells
12	- Excessive shuffle read/write in Spark UI stages
13	- Small files accumulation in Delta Lake tables
14	- Streaming ingestion throughput degradation
15	- Need to select or tune a Fabric Spark resource profile
16	- VOrder vs. Optimized Write decision-making
17	- Autotune configuration and validation
18	- Right-sizing Spark pools, node counts, or Fabric capacity SKUs
19
20	## Prerequisites
21
22	- Access to a Microsoft Fabric workspace with Data Engineering enabled
23	- Contributor or higher role on the workspace
24	- Familiarity with PySpark or Spark SQL
25	- PowerShell 7+ (for diagnostic scripts)
26	- Fabric REST API access token (for API-based diagnostics)
27
28	## Quick Diagnosis Decision Tree
29
30	Start here when a Spark job is slow:
31
32	1. Is the job queued or throttled? Check Monitoring Hub for HTTP 430.
33	- Yes → See [Capacity and Concurrency Tuning](./references/spark-configuration-tuning.md#capacity-and-concurrency)
34	- No → Continue
35
36	2. Did the Spark Advisor flag warnings? Check notebook cell indicators.
37	- Data Skew detected → See [Data Skew Resolution](./references/spark-configuration-tuning.md#data-skew-resolution)
38	- No warnings → Continue
39
40	3. Is a single stage disproportionately slow? Open Spark UI → Stages tab.
41	- Yes, shuffle stage → See [Shuffle Optimization](./references/spark-configuration-tuning.md#shuffle-optimization)
42	- Yes, scan stage → See [File Scan Optimization](./references/delta-lake-optimization.md#file-scan-optimization)
43	- No → Continue
44
45	4. Are executors underutilized? Check Resources tab in monitoring detail.
46	- High idle cores → See [Pool and Executor Sizing](./references/spark-configuration-tuning.md#pool-and-executor-sizing)
47	- All cores busy → See [Partitioning Strategy](./references/spark-configuration-tuning.md#partitioning-strategy)
48
49	5. Is the issue write-related? Check write duration in Spark UI.
50	- Yes → See [Delta Write Optimization](./references/delta-lake-optimization.md#delta-write-optimization)
51	- No → See [General Spark SQL Tuning](./references/spark-configuration-tuning.md#spark-sql-tuning)
52
53	## Core Spark Configuration Quick Reference
54
55	These are the three settings Fabric Autotune manages automatically. If autotune is disabled, tune them manually:
56
57	\| Setting \| Default \| Purpose \| Tuning Guidance \|
58	\|---------\|---------\|---------\|-----------------\|
59	\| `spark.sql.shuffle.partitions` \| 200 \| Partition count during joins/aggregations \| Set to 2-3x total executor cores for your pool \|
60	\| `spark.sql.autoBroadcastJoinThreshold` \| 10 MB \| Max table size for broadcast joins \| Increase to 100-256 MB for star-schema joins \|
61	\| `spark.sql.files.maxPartitionBytes` \| 128 MB \| Max bytes per file-read partition \| Increase for large sequential scans, decrease for high parallelism \|
62
63	## Resource Profiles Quick Reference
64
65	Fabric provides predefined profiles that bundle optimized Spark settings:
66
67	\| Profile \| Best For \| VOrder \| Key Characteristics \|
68	\|---------\|----------\|--------\|---------------------\|
69	\| `writeHeavy` \| ETL, batch ingestion, streaming \| Disabled \| Default for new workspaces; optimized write throughput \|
70	\| `readHeavyForSpark` \| Interactive Spark queries, analytics \| Enabled \| Optimized read paths for Spark workloads \|
71	\| `readHeavyForPBI` \| Power BI dashboards, DW queries \| Enabled \| Optimized for DirectLake and cross-engine reads \|
72
73	Apply a profile at the environment level or override per-session:
74
75	```python
76	# Per-session override example
77	spark.conf.set("spark.fabric.resource.profile", "readHeavyForSpark")
78	```
79
80	## Autotune Quick Start
81
82	Enable autotune to let Fabric automatically optimize shuffle partitions, broadcast thresholds, and partition bytes:
83
84	```python
85	# Enable in a notebook session
86	spark.conf.set("spark.ms.autotune.enabled", "true")
87
88	# Or set in Environment > Spark Pr

People who liked this also viewed

6 picks

Type

fabric-spark-perf-remediate

16stars1forks

View on GitHub

How to install fabric-spark-perf-remediate?

patrickgallucci/fabric-skills/fabric-spark-perf-remediate

$npx -y skills add PatrickGallucci/fabric-skills --skill fabric-spark-perf-remediate

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Microsoft Fabric Apache Spark Performance remediate
2
3	Systematic workflows for diagnosing, analyzing, and resolving Apache Spark performance problems in Microsoft Fabric Data Engineering and Data Science workloads.
4
5	## When to Use This Skill
6
7	Activate when encountering any of the following scenarios:
8
9	- Spark notebooks or jobs running slower than expected
10	- Capacity throttling errors (HTTP 430 / TooManyRequestsForCapacity)
11	- Data skew detected by Spark Advisor in notebook cells
12	- Excessive shuffle read/write in Spark UI stages
13	- Small files accumulation in Delta Lake tables
14	- Streaming ingestion throughput degradation
15	- Need to select or tune a Fabric Spark resource profile
16	- VOrder vs. Optimized Write decision-making
17	- Autotune configuration and validation
18	- Right-sizing Spark pools, node counts, or Fabric capacity SKUs
19
20	## Prerequisites
21
22	- Access to a Microsoft Fabric workspace with Data Engineering enabled
23	- Contributor or higher role on the workspace
24	- Familiarity with PySpark or Spark SQL
25	- PowerShell 7+ (for diagnostic scripts)
26	- Fabric REST API access token (for API-based diagnostics)
27
28	## Quick Diagnosis Decision Tree
29
30	Start here when a Spark job is slow:
31
32	1. Is the job queued or throttled? Check Monitoring Hub for HTTP 430.
33	- Yes → See [Capacity and Concurrency Tuning](./references/spark-configuration-tuning.md#capacity-and-concurrency)
34	- No → Continue
35
36	2. Did the Spark Advisor flag warnings? Check notebook cell indicators.
37	- Data Skew detected → See [Data Skew Resolution](./references/spark-configuration-tuning.md#data-skew-resolution)
38	- No warnings → Continue
39
40	3. Is a single stage disproportionately slow? Open Spark UI → Stages tab.
41	- Yes, shuffle stage → See [Shuffle Optimization](./references/spark-configuration-tuning.md#shuffle-optimization)
42	- Yes, scan stage → See [File Scan Optimization](./references/delta-lake-optimization.md#file-scan-optimization)
43	- No → Continue
44
45	4. Are executors underutilized? Check Resources tab in monitoring detail.
46	- High idle cores → See [Pool and Executor Sizing](./references/spark-configuration-tuning.md#pool-and-executor-sizing)
47	- All cores busy → See [Partitioning Strategy](./references/spark-configuration-tuning.md#partitioning-strategy)
48
49	5. Is the issue write-related? Check write duration in Spark UI.
50	- Yes → See [Delta Write Optimization](./references/delta-lake-optimization.md#delta-write-optimization)
51	- No → See [General Spark SQL Tuning](./references/spark-configuration-tuning.md#spark-sql-tuning)
52
53	## Core Spark Configuration Quick Reference
54
55	These are the three settings Fabric Autotune manages automatically. If autotune is disabled, tune them manually:
56
57	\| Setting \| Default \| Purpose \| Tuning Guidance \|
58	\|---------\|---------\|---------\|-----------------\|
59	\| `spark.sql.shuffle.partitions` \| 200 \| Partition count during joins/aggregations \| Set to 2-3x total executor cores for your pool \|
60	\| `spark.sql.autoBroadcastJoinThreshold` \| 10 MB \| Max table size for broadcast joins \| Increase to 100-256 MB for star-schema joins \|
61	\| `spark.sql.files.maxPartitionBytes` \| 128 MB \| Max bytes per file-read partition \| Increase for large sequential scans, decrease for high parallelism \|
62
63	## Resource Profiles Quick Reference
64
65	Fabric provides predefined profiles that bundle optimized Spark settings:
66
67	\| Profile \| Best For \| VOrder \| Key Characteristics \|
68	\|---------\|----------\|--------\|---------------------\|
69	\| `writeHeavy` \| ETL, batch ingestion, streaming \| Disabled \| Default for new workspaces; optimized write throughput \|
70	\| `readHeavyForSpark` \| Interactive Spark queries, analytics \| Enabled \| Optimized read paths for Spark workloads \|
71	\| `readHeavyForPBI` \| Power BI dashboards, DW queries \| Enabled \| Optimized for DirectLake and cross-engine reads \|
72
73	Apply a profile at the environment level or override per-session:
74
75	```python
76	# Per-session override example
77	spark.conf.set("spark.fabric.resource.profile", "readHeavyForSpark")
78	```
79
80	## Autotune Quick Start
81
82	Enable autotune to let Fabric automatically optimize shuffle partitions, broadcast thresholds, and partition bytes:
83
84	```python
85	# Enable in a notebook session
86	spark.conf.set("spark.ms.autotune.enabled", "true")
87
88	# Or set in Environment > Spark Pr

People who liked this also viewed

6 picks

Type