fabric-spark-compute-perf-remediate

Name: fabric-spark-compute-perf-remediate
Author: PatrickGallucci

16stars1forks

How to install fabric-spark-compute-perf-remediate?

patrickgallucci/fabric-skills/fabric-spark-compute-perf-remediate

$npx -y skills add PatrickGallucci/fabric-skills --skill fabric-spark-compute-perf-remediate

Installs into the current project.

Summary

Diagnose and resolve performance issues in Microsoft Fabric Delta Lake and Spark workloads. Use when remediate slow Spark notebooks, long-running Spark jobs, Delta table query degradation, small file problems, VOrder configuration, resource profile tuning, autotune configuration,

Files · 1

View on GitHub

SKILL.md

1	# Fabric Delta Lake & Spark Performance remediate
2
3	Systematic remediate toolkit for diagnosing and resolving performance issues across
4	Microsoft Fabric Spark compute, Delta Lake tables, and Lakehouse workloads.
5
6	## When to Use This Skill
7
8	- Spark notebooks or jobs running slower than expected
9	- Delta table queries degrading over time (small file problem)
10	- Capacity throttling errors (HTTP 430 / TooManyRequestsForCapacity)
11	- Session startup taking minutes instead of seconds
12	- Out-of-memory errors on executors or drivers
13	- Streaming ingestion falling behind or producing small files
14	- VOrder or resource profile misconfiguration
15	- Deciding between read-heavy vs. write-heavy workload profiles
16	- Planning table maintenance (OPTIMIZE, VACUUM, Z-Order)
17	- Tuning autoscaling, dynamic executor allocation, or autotune
18
19	## Prerequisites
20
21	- Access to a Microsoft Fabric workspace with Data Engineering enabled
22	- Contributor or higher role on the workspace
23	- Fabric capacity (F2+) or Trial capacity
24	- PowerShell 7+ for diagnostic scripts
25	- PySpark for notebook-based diagnostics
26
27	## Triage: Identify the Symptom Category
28
29	Start by matching the observed symptom to one of these categories, then follow the linked
30	reference for the detailed workflow.
31
32	\| Symptom \| Category \| Reference \|
33	\| ---------------------------------------- \| --------------------- \| ---------------------------------------------------------- \|
34	\| Notebook takes minutes to start \| Session Startup \| [session-startup.md](./references/session-startup.md) \|
35	\| Queries slow, getting worse over time \| Delta Table Health \| [delta-table-health.md](./references/delta-table-health.md) \|
36	\| HTTP 430 or "TooManyRequestsForCapacity" \| Capacity Throttling \| [capacity-throttling.md](./references/capacity-throttling.md) \|
37	\| OOM errors, executor lost, task failed \| Memory & Compute \| [memory-compute.md](./references/memory-compute.md) \|
38	\| Streaming lag, checkpoint delays \| Streaming Performance \| [streaming-perf.md](./references/streaming-perf.md) \|
39	\| Power BI reports slow against Lakehouse \| Read Optimization \| [read-optimization.md](./references/read-optimization.md) \|
40	\| Spark job slow but no errors \| General Tuning \| [general-tuning.md](./references/general-tuning.md) \|
41
42	## Quick Diagnostic Checklist
43
44	Run through these checks before diving into detailed remediate.
45
46	### 1. Check Capacity and Concurrency
47
48	Verify your Fabric capacity SKU and current utilization. Each SKU has a queue limit
49	for concurrent Spark jobs:
50
51	\| SKU \| Spark VCores \| Queue Limit \|
52	\| ----- \| ------------ \| ----------- \|
53	\| F2 \| 4 \| 4 \|
54	\| F8 \| 16 \| 8 \|
55	\| F16 \| 32 \| 16 \|
56	\| F32 \| 64 \| 32 \|
57	\| F64 \| 128 \| 64 \|
58	\| F128 \| 256 \| 128 \|
59	\| Trial \| 128 \| No queueing \|
60
61	If hitting queue limits, cancel idle jobs via the Monitoring Hub, scale the SKU, or
62	stagger job schedules.
63
64	### 2. Check Resource Profile
65
66	New Fabric workspaces default to the `writeHeavy` profile. Verify the active profile
67	matches your workload:
68
69	\| Profile \| Best For \| VOrder Default \|
70	\| ----------------- \| -------------------------------- \| -------------- \|
71	\| writeHeavy \| ETL, ingestion, batch writes \| Disabled \|
72	\| readHeavyForSpark \| Interactive Spark queries, EDA \| Enabled \|
73	\| readHeavyForPBI \| Power BI Direct Lake, dashboards \| Enabled \|
74
75	Mismatch is one of the most common performance issues. Read-heavy workloads on
76	the writeHeavy profile miss VOrder optimization entirely.
77
78	### 3. Check Delta Table Health
79
80	Run the [diagnostic script](./scripts/Diagnose-DeltaTableHealth.ps1) or execute in a
81	notebook:
82
83	```python
84	# Quick Delta table health check
85	from delta.tables import DeltaTable
86
87	dt = DeltaTable.forName(spark, "your_table_name")
88	history = dt.history()
89	history.select("version", "operation", "operationMetrics").show(20, truncate=False)
90	```
91
92	Look for: high version counts without OPTIMIZE, many small ADD operations, and
93	missing VACUUM runs.
94
95	### 4. Check Environment Configura

People who liked this also viewed

6 picks

Type

fabric-spark-compute-perf-remediate

16stars1forks

View on GitHub

How to install fabric-spark-compute-perf-remediate?

patrickgallucci/fabric-skills/fabric-spark-compute-perf-remediate

$npx -y skills add PatrickGallucci/fabric-skills --skill fabric-spark-compute-perf-remediate

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Fabric Delta Lake & Spark Performance remediate
2
3	Systematic remediate toolkit for diagnosing and resolving performance issues across
4	Microsoft Fabric Spark compute, Delta Lake tables, and Lakehouse workloads.
5
6	## When to Use This Skill
7
8	- Spark notebooks or jobs running slower than expected
9	- Delta table queries degrading over time (small file problem)
10	- Capacity throttling errors (HTTP 430 / TooManyRequestsForCapacity)
11	- Session startup taking minutes instead of seconds
12	- Out-of-memory errors on executors or drivers
13	- Streaming ingestion falling behind or producing small files
14	- VOrder or resource profile misconfiguration
15	- Deciding between read-heavy vs. write-heavy workload profiles
16	- Planning table maintenance (OPTIMIZE, VACUUM, Z-Order)
17	- Tuning autoscaling, dynamic executor allocation, or autotune
18
19	## Prerequisites
20
21	- Access to a Microsoft Fabric workspace with Data Engineering enabled
22	- Contributor or higher role on the workspace
23	- Fabric capacity (F2+) or Trial capacity
24	- PowerShell 7+ for diagnostic scripts
25	- PySpark for notebook-based diagnostics
26
27	## Triage: Identify the Symptom Category
28
29	Start by matching the observed symptom to one of these categories, then follow the linked
30	reference for the detailed workflow.
31
32	\| Symptom \| Category \| Reference \|
33	\| ---------------------------------------- \| --------------------- \| ---------------------------------------------------------- \|
34	\| Notebook takes minutes to start \| Session Startup \| [session-startup.md](./references/session-startup.md) \|
35	\| Queries slow, getting worse over time \| Delta Table Health \| [delta-table-health.md](./references/delta-table-health.md) \|
36	\| HTTP 430 or "TooManyRequestsForCapacity" \| Capacity Throttling \| [capacity-throttling.md](./references/capacity-throttling.md) \|
37	\| OOM errors, executor lost, task failed \| Memory & Compute \| [memory-compute.md](./references/memory-compute.md) \|
38	\| Streaming lag, checkpoint delays \| Streaming Performance \| [streaming-perf.md](./references/streaming-perf.md) \|
39	\| Power BI reports slow against Lakehouse \| Read Optimization \| [read-optimization.md](./references/read-optimization.md) \|
40	\| Spark job slow but no errors \| General Tuning \| [general-tuning.md](./references/general-tuning.md) \|
41
42	## Quick Diagnostic Checklist
43
44	Run through these checks before diving into detailed remediate.
45
46	### 1. Check Capacity and Concurrency
47
48	Verify your Fabric capacity SKU and current utilization. Each SKU has a queue limit
49	for concurrent Spark jobs:
50
51	\| SKU \| Spark VCores \| Queue Limit \|
52	\| ----- \| ------------ \| ----------- \|
53	\| F2 \| 4 \| 4 \|
54	\| F8 \| 16 \| 8 \|
55	\| F16 \| 32 \| 16 \|
56	\| F32 \| 64 \| 32 \|
57	\| F64 \| 128 \| 64 \|
58	\| F128 \| 256 \| 128 \|
59	\| Trial \| 128 \| No queueing \|
60
61	If hitting queue limits, cancel idle jobs via the Monitoring Hub, scale the SKU, or
62	stagger job schedules.
63
64	### 2. Check Resource Profile
65
66	New Fabric workspaces default to the `writeHeavy` profile. Verify the active profile
67	matches your workload:
68
69	\| Profile \| Best For \| VOrder Default \|
70	\| ----------------- \| -------------------------------- \| -------------- \|
71	\| writeHeavy \| ETL, ingestion, batch writes \| Disabled \|
72	\| readHeavyForSpark \| Interactive Spark queries, EDA \| Enabled \|
73	\| readHeavyForPBI \| Power BI Direct Lake, dashboards \| Enabled \|
74
75	Mismatch is one of the most common performance issues. Read-heavy workloads on
76	the writeHeavy profile miss VOrder optimization entirely.
77
78	### 3. Check Delta Table Health
79
80	Run the [diagnostic script](./scripts/Diagnose-DeltaTableHealth.ps1) or execute in a
81	notebook:
82
83	```python
84	# Quick Delta table health check
85	from delta.tables import DeltaTable
86
87	dt = DeltaTable.forName(spark, "your_table_name")
88	history = dt.history()
89	history.select("version", "operation", "operationMetrics").show(20, truncate=False)
90	```
91
92	Look for: high version counts without OPTIMIZE, many small ADD operations, and
93	missing VACUUM runs.
94
95	### 4. Check Environment Configura

People who liked this also viewed

6 picks

Type