fabric-delta-spark-perf

Name: fabric-delta-spark-perf
Author: PatrickGallucci

16stars1forks

How to install fabric-delta-spark-perf?

patrickgallucci/fabric-skills/fabric-delta-spark-perf

$npx -y skills add PatrickGallucci/fabric-skills --skill fabric-delta-spark-perf

Installs into the current project.

Summary

Troubleshoot and optimize Delta Lake and Apache Spark performance in Microsoft Fabric. Use when diagnosing slow Spark jobs, small file problems, data skew, shuffle bottlenecks, out-of-memory errors, V-Order tuning, OPTIMIZE/VACUUM operations, partition strategy, resource profile

Files · 1

View on GitHub

SKILL.md

1	# Microsoft Fabric Delta Lake Spark Performance remediate
2
3	Systematic workflows for diagnosing and resolving Apache Spark and Delta Lake performance issues in Microsoft Fabric Lakehouse environments.
4
5	## When to Use This Skill
6
7	Activate when the user mentions any of the following:
8
9	- Spark job is slow, taking too long, or timing out
10	- Small file problem, too many small files, file fragmentation
11	- Data skew, straggler tasks, unbalanced partitions
12	- Out of memory (OOM) errors on driver or executor
13	- Shuffle spill, excessive shuffle read/write
14	- OPTIMIZE, VACUUM, bin-compaction, or table maintenance
15	- V-Order, Z-Order, or Parquet optimization
16	- Resource profiles: writeHeavy, readHeavyForSpark, readHeavyForPBI
17	- Autotune, Adaptive Query Execution (AQE), broadcast join thresholds
18	- Native Execution Engine configuration
19	- Streaming performance, microbatch tuning, checkpoint issues
20	- Spark pool sizing, autoscale, dynamic executor allocation
21	- Direct Lake performance tied to Delta table structure
22	- Capacity throttling, TooManyRequestsForCapacity errors
23
24	## Prerequisites
25
26	- Microsoft Fabric workspace with Data Engineering or Data Science experience
27	- Apache Spark notebooks or Spark Job Definitions
28	- Lakehouse with Delta tables
29	- Appropriate Fabric capacity SKU (F2 through F2048)
30
31	## Quick Diagnostic Workflow
32
33	When a user reports slow Spark performance, follow this triage sequence:
34
35	### Step 1: Identify the Symptom Category
36
37	\| Symptom \| Likely Root Cause \| Jump To \|
38	\|---------\|-------------------\|---------\|
39	\| Job runs much longer than expected \| Data skew or small files \| Step 2 \|
40	\| OOM error on driver \| `collect()`, `toPandas()`, or large broadcast \| [diagnostic-checklist.md](./references/diagnostic-checklist.md#oom-errors) \|
41	\| OOM error on executor \| Wide joins, large shuffles, insufficient memory \| [diagnostic-checklist.md](./references/diagnostic-checklist.md#oom-errors) \|
42	\| Many tasks, most finish fast, few stragglers \| Data skew \| [diagnostic-checklist.md](./references/diagnostic-checklist.md#data-skew) \|
43	\| High shuffle read/write in Spark UI \| Missing broadcast join or too many partitions \| [diagnostic-checklist.md](./references/diagnostic-checklist.md#shuffle-bottlenecks) \|
44	\| Query reads thousands of small files \| Small file problem, needs OPTIMIZE \| Step 3 \|
45	\| Capacity throttled (HTTP 430) \| Too many concurrent jobs for SKU \| [spark-configurations.md](./references/spark-configurations.md#capacity-limits) \|
46	\| Streaming lag increasing \| Microbatch interval or partition mismatch \| [diagnostic-checklist.md](./references/diagnostic-checklist.md#streaming-performance) \|
47
48	### Step 2: Check for Data Skew
49
50	Run the diagnostic script to detect skew in a target table:
51
52	```python
53	# Quick skew detection
54	df = spark.read.format("delta").table("your_table")
55	df.groupBy("partition_column") \
56	.count() \
57	.orderBy(F.desc("count")) \
58	.show(20)
59	```
60
61	If the top partition has 10x+ more rows than the median, apply skew mitigation. See [diagnostic-checklist.md](./references/diagnostic-checklist.md#data-skew) for remediation steps.
62
63	### Step 3: Check File Health
64
65	```python
66	# Check file count and sizes for a Delta table
67	from delta.tables import DeltaTable
68	dt = DeltaTable.forName(spark, "your_table")
69	detail = spark.sql("DESCRIBE DETAIL your_table")
70	detail.select("numFiles", "sizeInBytes").show()
71
72	# Check for small files (< 32MB)
73	files_df = spark.sql("DESCRIBE DETAIL your_table")
74	```
75
76	If file count is high relative to data volume (e.g., >1000 files for <10GB), run OPTIMIZE. See [table-maintenance-guide.md](./references/table-maintenance-guide.md#optimize-command).
77
78	### Step 4: Verify Spark Configuration
79
80	Check that the environment has an appropriate resource profile:
81
82	```python
83	# Check current resource profile
84	print(spark.conf.get("spark.fabric.resourceProfile", "not set"))
85
86	# Check key write/read settings
87	configs = [
88	"spark.sql.parquet.vorder.default",
89	"spark.databricks.delta.optimizeWrite.enabled",
90	"spark.databricks.delta.optimizeWrite.binSize",
91	"spark.sql.shuffle.partitions",
92	"spark.sql.autoBroadcastJoinThreshold",
93	"spark.sql.files.maxPartitionBytes",
94	"spark.databricks.optimizer.adaptive.enabled"
95	]
96	for c in configs:
97	try:
98	print(f"{c} = {spark.conf.get(c)}")
99	except:
100	print(f"{c} = (default)")
101	```
102
103	See [spark-configurations.md](./references/spark-configurations.md) for recommended values per workload

People who liked this also viewed

6 picks

Type

fabric-delta-spark-perf

16stars1forks

View on GitHub

How to install fabric-delta-spark-perf?

patrickgallucci/fabric-skills/fabric-delta-spark-perf

$npx -y skills add PatrickGallucci/fabric-skills --skill fabric-delta-spark-perf

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Microsoft Fabric Delta Lake Spark Performance remediate
2
3	Systematic workflows for diagnosing and resolving Apache Spark and Delta Lake performance issues in Microsoft Fabric Lakehouse environments.
4
5	## When to Use This Skill
6
7	Activate when the user mentions any of the following:
8
9	- Spark job is slow, taking too long, or timing out
10	- Small file problem, too many small files, file fragmentation
11	- Data skew, straggler tasks, unbalanced partitions
12	- Out of memory (OOM) errors on driver or executor
13	- Shuffle spill, excessive shuffle read/write
14	- OPTIMIZE, VACUUM, bin-compaction, or table maintenance
15	- V-Order, Z-Order, or Parquet optimization
16	- Resource profiles: writeHeavy, readHeavyForSpark, readHeavyForPBI
17	- Autotune, Adaptive Query Execution (AQE), broadcast join thresholds
18	- Native Execution Engine configuration
19	- Streaming performance, microbatch tuning, checkpoint issues
20	- Spark pool sizing, autoscale, dynamic executor allocation
21	- Direct Lake performance tied to Delta table structure
22	- Capacity throttling, TooManyRequestsForCapacity errors
23
24	## Prerequisites
25
26	- Microsoft Fabric workspace with Data Engineering or Data Science experience
27	- Apache Spark notebooks or Spark Job Definitions
28	- Lakehouse with Delta tables
29	- Appropriate Fabric capacity SKU (F2 through F2048)
30
31	## Quick Diagnostic Workflow
32
33	When a user reports slow Spark performance, follow this triage sequence:
34
35	### Step 1: Identify the Symptom Category
36
37	\| Symptom \| Likely Root Cause \| Jump To \|
38	\|---------\|-------------------\|---------\|
39	\| Job runs much longer than expected \| Data skew or small files \| Step 2 \|
40	\| OOM error on driver \| `collect()`, `toPandas()`, or large broadcast \| [diagnostic-checklist.md](./references/diagnostic-checklist.md#oom-errors) \|
41	\| OOM error on executor \| Wide joins, large shuffles, insufficient memory \| [diagnostic-checklist.md](./references/diagnostic-checklist.md#oom-errors) \|
42	\| Many tasks, most finish fast, few stragglers \| Data skew \| [diagnostic-checklist.md](./references/diagnostic-checklist.md#data-skew) \|
43	\| High shuffle read/write in Spark UI \| Missing broadcast join or too many partitions \| [diagnostic-checklist.md](./references/diagnostic-checklist.md#shuffle-bottlenecks) \|
44	\| Query reads thousands of small files \| Small file problem, needs OPTIMIZE \| Step 3 \|
45	\| Capacity throttled (HTTP 430) \| Too many concurrent jobs for SKU \| [spark-configurations.md](./references/spark-configurations.md#capacity-limits) \|
46	\| Streaming lag increasing \| Microbatch interval or partition mismatch \| [diagnostic-checklist.md](./references/diagnostic-checklist.md#streaming-performance) \|
47
48	### Step 2: Check for Data Skew
49
50	Run the diagnostic script to detect skew in a target table:
51
52	```python
53	# Quick skew detection
54	df = spark.read.format("delta").table("your_table")
55	df.groupBy("partition_column") \
56	.count() \
57	.orderBy(F.desc("count")) \
58	.show(20)
59	```
60
61	If the top partition has 10x+ more rows than the median, apply skew mitigation. See [diagnostic-checklist.md](./references/diagnostic-checklist.md#data-skew) for remediation steps.
62
63	### Step 3: Check File Health
64
65	```python
66	# Check file count and sizes for a Delta table
67	from delta.tables import DeltaTable
68	dt = DeltaTable.forName(spark, "your_table")
69	detail = spark.sql("DESCRIBE DETAIL your_table")
70	detail.select("numFiles", "sizeInBytes").show()
71
72	# Check for small files (< 32MB)
73	files_df = spark.sql("DESCRIBE DETAIL your_table")
74	```
75
76	If file count is high relative to data volume (e.g., >1000 files for <10GB), run OPTIMIZE. See [table-maintenance-guide.md](./references/table-maintenance-guide.md#optimize-command).
77
78	### Step 4: Verify Spark Configuration
79
80	Check that the environment has an appropriate resource profile:
81
82	```python
83	# Check current resource profile
84	print(spark.conf.get("spark.fabric.resourceProfile", "not set"))
85
86	# Check key write/read settings
87	configs = [
88	"spark.sql.parquet.vorder.default",
89	"spark.databricks.delta.optimizeWrite.enabled",
90	"spark.databricks.delta.optimizeWrite.binSize",
91	"spark.sql.shuffle.partitions",
92	"spark.sql.autoBroadcastJoinThreshold",
93	"spark.sql.files.maxPartitionBytes",
94	"spark.databricks.optimizer.adaptive.enabled"
95	]
96	for c in configs:
97	try:
98	print(f"{c} = {spark.conf.get(c)}")
99	except:
100	print(f"{c} = (default)")
101	```
102
103	See [spark-configurations.md](./references/spark-configurations.md) for recommended values per workload

People who liked this also viewed

6 picks

Type