fabric-notebook-perf-remediate

Name: fabric-notebook-perf-remediate
Author: PatrickGallucci

16stars1forks

How to install fabric-notebook-perf-remediate?

patrickgallucci/fabric-skills/fabric-notebook-perf-remediate

$npx -y skills add PatrickGallucci/fabric-skills --skill fabric-notebook-perf-remediate

Installs into the current project.

Summary

Diagnose and resolve performance issues in Microsoft Fabric notebooks running Apache Spark. Use when notebooks are slow, Spark jobs are timing out, sessions hit HTTP 430 throttling errors, data skew causes straggling tasks, shuffle operations are inefficient, Delta Lake tables ne

Files · 1

View on GitHub

SKILL.md

1	# Microsoft Fabric Notebook Performance remediate
2
3	Systematic toolkit for diagnosing, analyzing, and resolving performance bottlenecks in Microsoft Fabric notebooks powered by Apache Spark.
4
5	## When to Use This Skill
6
7	- Fabric notebook cells are running slowly or timing out
8	- Spark jobs are being throttled with HTTP 430 errors
9	- Capacity Metrics app shows high CU consumption
10	- Data skew is causing unbalanced task execution
11	- Shuffle operations are consuming excessive resources
12	- Delta Lake tables have degraded read/write performance
13	- OOM (Out of Memory) errors during notebook execution
14	- Spark Advisor shows warnings or errors in cell output
15	- Session startup is slow or sessions expire unexpectedly
16	- Pipeline-triggered notebooks are queued for extended periods
17
18	## Prerequisites
19
20	- Workspace Admin or Contributor role in the target Fabric workspace
21	- Access to the Fabric Monitoring Hub for your capacity
22	- Fabric Capacity Metrics app installed (for capacity-level analysis)
23	- Familiarity with PySpark or Spark SQL syntax
24
25	## remediate Decision Tree
26
27	Identify your symptom and follow the corresponding workflow.
28
29	\| Symptom \| Root Cause Category \| Action \|
30	\|---------\|-------------------\|--------\|
31	\| Notebook cell runs for minutes on small data \| Spark session config or query plan \| See [Spark Session Tuning](./references/spark-session-tuning.md) \|
32	\| HTTP 430 error on job submission \| Capacity exhausted, concurrency limit \| See [Capacity and Throttling](#capacity-and-throttling) \|
33	\| One task takes 10x longer than others \| Data skew \| See [Data Skew Diagnosis](./references/data-skew-diagnosis.md) \|
34	\| Write operations are slow \| V-Order overhead or small file problem \| See [Delta Table Optimization](./references/delta-table-optimization.md) \|
35	\| OOM / executor lost errors \| Memory pressure, partition sizing \| See [Memory and Partition Tuning](#memory-and-partition-tuning) \|
36	\| Session expired / timed out \| Idle timeout settings \| See [Common Errors](./references/common-errors.md) \|
37	\| Notebook queued, never starts \| Queue limits for SKU \| See [Capacity and Throttling](#capacity-and-throttling) \|
38
39	## Quick Wins Checklist
40
41	Apply these optimizations first — they resolve the majority of performance issues.
42
43	1. Enable Native Execution Engine (NEE) — delivers 2x–5x improvement:
44
45	```python
46	spark.conf.set("spark.native.enabled", "true")
47	```
48
49	2. Enable Autotune for adaptive configuration:
50
51	```python
52	spark.conf.set("spark.ms.autotune.enabled", "true")
53	```
54
55	3. Right-size shuffle partitions — default 200 is often wrong:
56
57	```python
58	# For datasets under 1 GB
59	spark.conf.set("spark.sql.shuffle.partitions", "20")
60	# For datasets 1-10 GB
61	spark.conf.set("spark.sql.shuffle.partitions", "100")
62	# For datasets over 10 GB, leave default or increase
63	```
64
65	4. Enable Adaptive Query Execution (AQE):
66
67	```python
68	spark.conf.set("spark.sql.adaptive.enabled", "true")
69	spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
70	spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
71	```
72
73	5. Use DataFrame APIs instead of RDDs — enables Catalyst optimizer and Tungsten engine.
74
75	6. Break complex query chains into staged intermediate writes to reduce Catalyst plan complexity. See [Common Errors](./references/common-errors.md#excessive-query-complexity).
76
77	Run the [notebook health check script](./scripts/notebook-health-check.py) to audit your current session configuration.
78
79	## Capacity and Throttling
80
81	Each Fabric SKU maps to a fixed number of Spark VCores (1 CU = 2 Spark VCores). When all VCores are consumed, new jobs receive HTTP 430 errors.
82
83	\| SKU \| Spark VCores \| Queue Limit \|
84	\|-----\|-------------\|-------------\|
85	\| F2 \| 4 \| 4 \|
86	\| F8 \| 16 \| 8 \|
87	\| F64 / P1 \| 128 \| 64 \|
88	\| F128 / P2 \| 256 \| 128 \|
89	\| F256 / P3 \| 512 \| 256 \|
90
91	Resolution steps:
92
93	1. Open the Monitoring Hub and cancel idle or unnecessary Spark sessions.
94	2. Stop sessions you are not actively using — default idle timeout is 20 minutes.
95	3. Reduce executor count in custom Spark pools to free VCores for parallel jobs.
96	4. Enable Autoscale Billing for Spark for bursty workloads — jobs use dedicated serverless resources instead of consuming capacity CUs.
97	5. For pipeline-triggered notebooks, leverage job queueing (FIFO). Queue expiry is 24 hours.
98
99	> Queueing is not supported for interactive notebook jobs or Fabric trial capa

People who liked this also viewed

6 picks

Type

fabric-notebook-perf-remediate

16stars1forks

View on GitHub

How to install fabric-notebook-perf-remediate?

patrickgallucci/fabric-skills/fabric-notebook-perf-remediate

$npx -y skills add PatrickGallucci/fabric-skills --skill fabric-notebook-perf-remediate

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Microsoft Fabric Notebook Performance remediate
2
3	Systematic toolkit for diagnosing, analyzing, and resolving performance bottlenecks in Microsoft Fabric notebooks powered by Apache Spark.
4
5	## When to Use This Skill
6
7	- Fabric notebook cells are running slowly or timing out
8	- Spark jobs are being throttled with HTTP 430 errors
9	- Capacity Metrics app shows high CU consumption
10	- Data skew is causing unbalanced task execution
11	- Shuffle operations are consuming excessive resources
12	- Delta Lake tables have degraded read/write performance
13	- OOM (Out of Memory) errors during notebook execution
14	- Spark Advisor shows warnings or errors in cell output
15	- Session startup is slow or sessions expire unexpectedly
16	- Pipeline-triggered notebooks are queued for extended periods
17
18	## Prerequisites
19
20	- Workspace Admin or Contributor role in the target Fabric workspace
21	- Access to the Fabric Monitoring Hub for your capacity
22	- Fabric Capacity Metrics app installed (for capacity-level analysis)
23	- Familiarity with PySpark or Spark SQL syntax
24
25	## remediate Decision Tree
26
27	Identify your symptom and follow the corresponding workflow.
28
29	\| Symptom \| Root Cause Category \| Action \|
30	\|---------\|-------------------\|--------\|
31	\| Notebook cell runs for minutes on small data \| Spark session config or query plan \| See [Spark Session Tuning](./references/spark-session-tuning.md) \|
32	\| HTTP 430 error on job submission \| Capacity exhausted, concurrency limit \| See [Capacity and Throttling](#capacity-and-throttling) \|
33	\| One task takes 10x longer than others \| Data skew \| See [Data Skew Diagnosis](./references/data-skew-diagnosis.md) \|
34	\| Write operations are slow \| V-Order overhead or small file problem \| See [Delta Table Optimization](./references/delta-table-optimization.md) \|
35	\| OOM / executor lost errors \| Memory pressure, partition sizing \| See [Memory and Partition Tuning](#memory-and-partition-tuning) \|
36	\| Session expired / timed out \| Idle timeout settings \| See [Common Errors](./references/common-errors.md) \|
37	\| Notebook queued, never starts \| Queue limits for SKU \| See [Capacity and Throttling](#capacity-and-throttling) \|
38
39	## Quick Wins Checklist
40
41	Apply these optimizations first — they resolve the majority of performance issues.
42
43	1. Enable Native Execution Engine (NEE) — delivers 2x–5x improvement:
44
45	```python
46	spark.conf.set("spark.native.enabled", "true")
47	```
48
49	2. Enable Autotune for adaptive configuration:
50
51	```python
52	spark.conf.set("spark.ms.autotune.enabled", "true")
53	```
54
55	3. Right-size shuffle partitions — default 200 is often wrong:
56
57	```python
58	# For datasets under 1 GB
59	spark.conf.set("spark.sql.shuffle.partitions", "20")
60	# For datasets 1-10 GB
61	spark.conf.set("spark.sql.shuffle.partitions", "100")
62	# For datasets over 10 GB, leave default or increase
63	```
64
65	4. Enable Adaptive Query Execution (AQE):
66
67	```python
68	spark.conf.set("spark.sql.adaptive.enabled", "true")
69	spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
70	spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
71	```
72
73	5. Use DataFrame APIs instead of RDDs — enables Catalyst optimizer and Tungsten engine.
74
75	6. Break complex query chains into staged intermediate writes to reduce Catalyst plan complexity. See [Common Errors](./references/common-errors.md#excessive-query-complexity).
76
77	Run the [notebook health check script](./scripts/notebook-health-check.py) to audit your current session configuration.
78
79	## Capacity and Throttling
80
81	Each Fabric SKU maps to a fixed number of Spark VCores (1 CU = 2 Spark VCores). When all VCores are consumed, new jobs receive HTTP 430 errors.
82
83	\| SKU \| Spark VCores \| Queue Limit \|
84	\|-----\|-------------\|-------------\|
85	\| F2 \| 4 \| 4 \|
86	\| F8 \| 16 \| 8 \|
87	\| F64 / P1 \| 128 \| 64 \|
88	\| F128 / P2 \| 256 \| 128 \|
89	\| F256 / P3 \| 512 \| 256 \|
90
91	Resolution steps:
92
93	1. Open the Monitoring Hub and cancel idle or unnecessary Spark sessions.
94	2. Stop sessions you are not actively using — default idle timeout is 20 minutes.
95	3. Reduce executor count in custom Spark pools to free VCores for parallel jobs.
96	4. Enable Autoscale Billing for Spark for bursty workloads — jobs use dedicated serverless resources instead of consuming capacity CUs.
97	5. For pipeline-triggered notebooks, leverage job queueing (FIFO). Queue expiry is 24 hours.
98
99	> Queueing is not supported for interactive notebook jobs or Fabric trial capa

People who liked this also viewed

6 picks

Type