fabric-data-factory-perf-remediate

Name: fabric-data-factory-perf-remediate
Author: PatrickGallucci

16stars1forks

How to install fabric-data-factory-perf-remediate?

patrickgallucci/fabric-skills/fabric-data-factory-perf-remediate

$npx -y skills add PatrickGallucci/fabric-skills --skill fabric-data-factory-perf-remediate

Installs into the current project.

Summary

Diagnose and resolve Microsoft Fabric Data Factory pipeline performance issues. Use when pipelines are slow, copy activities timeout, dataflows stall, activities are stuck, throughput is low, capacity is throttled, or jobs queue indefinitely. Covers copy activity tuning (parallel

Files · 1

View on GitHub

SKILL.md

1	# Microsoft Fabric Data Factory Performance remediate
2
3	Systematic approach to diagnosing and resolving performance issues in Microsoft Fabric Data Factory pipelines, copy activities, and dataflows.
4
5	## When to Use This Skill
6
7	- Pipeline execution takes longer than expected
8	- Copy activities are slow or appear stuck
9	- Activities show "Not Started" status for extended periods
10	- Capacity throttling errors (HTTP 430, TooManyRequestsForCapacity)
11	- Throughput is lower than expected for copy operations
12	- Dataflow Gen2 refresh is slow or timing out
13	- Pipeline monitoring shows performance degradation over time
14	- Need to optimize parallelism, DIU, or partitioning settings
15
16	## Prerequisites
17
18	- Access to Microsoft Fabric workspace with Contributor or higher role
19	- Familiarity with the Fabric Monitoring Hub
20	- Understanding of Fabric capacity SKUs and their limits
21	- PowerShell 7+ for running diagnostic scripts
22
23	## Diagnostic Workflow
24
25	### Step 1: Identify the Bottleneck Category
26
27	Determine which category your issue falls into:
28
29	\| Category \| Symptoms \| Start Here \|
30	\| ------------------- \| ------------------------------------------- \| ----------------------------------------------------------------------------------------- \|
31	\| Copy Activity Slow \| Low throughput, long transfer duration \| [copy-activity-tuning.md](./references/copy-activity-tuning.md) \|
32	\| Pipeline Stuck \| Activity shows In Progress with no movement \| [pipeline-stuck-resolution.md](./references/pipeline-stuck-resolution.md) \|
33	\| Capacity Throttling \| HTTP 430 errors, jobs queued \| [capacity-throttling-guide.md](./references/capacity-throttling-guide.md) \|
34	\| Dataflow Slow \| Dataflow Gen2 refresh takes too long \| [dataflow-optimization.md](./references/dataflow-optimization.md) \|
35	\| Spark Job Queue \| Jobs stuck in "Not Started" status \| [capacity-throttling-guide.md](./references/capacity-throttling-guide.md#spark-job-queueing) \|
36
37	### Step 2: Collect Diagnostics
38
39	Run the [diagnostic script](./scripts/Get-FabricPipelineDiagnostics.ps1) to gather baseline metrics:
40
41	```powershell
42	./scripts/Get-FabricPipelineDiagnostics.ps1 -WorkspaceId "<guid>" -PipelineName "MyPipeline"
43	```
44
45	Or manually collect from the Monitoring Hub:
46
47	1. Open Fabric portal and navigate to Monitoring Hub
48	2. Filter by pipeline name and time range
49	3. Select the run details (glasses icon) for the slow run
50	4. Capture the Duration Breakdown for copy activities
51	5. Note the queue time, transfer time, and pre/post-copy script duration
52
53	### Step 3: Apply Targeted Fixes
54
55	Based on the bottleneck category, apply the appropriate optimization from the reference guides.
56
57	## Quick Fixes for Common Issues
58
59	### Copy Activity Running Slowly
60
61	1. Set Intelligent Throughput Optimization to `Maximum` (or custom 4-256)
62	2. Configure Degree of Copy Parallelism based on source type
63	3. Enable Partition Option for SQL sources (Dynamic Range or Physical)
64	4. Pre-calculate partition upper/lower bounds to avoid overhead
65	5. Enable Staging when sink is Fabric Warehouse
66
67	### Pipeline Activity Stuck
68
69	1. Cancel the stuck activity and retry
70	2. Check source/sink connectivity and credentials
71	3. Verify Fabric capacity is not in throttled state
72	4. Review if payload exceeds 896 KB limit
73	5. Check for connection timeout or network interruption
74
75	### Capacity Throttling (HTTP 430)
76
77	1. Check current Spark concurrency against SKU limits
78	2. Cancel unnecessary active Spark jobs via Monitoring Hub
79	3. Consider upgrading to a larger capacity SKU
80	4. Distribute pipeline trigger times to avoid burst load
81	5. Use job queueing for non-interactive Spark workloads
82
83	### Dataflow Gen2 Performance
84
85	1. Reduce data volume with query folding and filters
86	2. Avoid unnecessary data type conversions
87	3. Minimize the number of transformation steps
88	4. Use staging for large datasets
89	5. Check for connector-specific throttling
90
91	## Capacity SKU Quick Reference
92
93	\| SKU \| Max S

People who liked this also viewed

6 picks

Type

fabric-data-factory-perf-remediate

16stars1forks

View on GitHub

How to install fabric-data-factory-perf-remediate?

patrickgallucci/fabric-skills/fabric-data-factory-perf-remediate

$npx -y skills add PatrickGallucci/fabric-skills --skill fabric-data-factory-perf-remediate

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Microsoft Fabric Data Factory Performance remediate
2
3	Systematic approach to diagnosing and resolving performance issues in Microsoft Fabric Data Factory pipelines, copy activities, and dataflows.
4
5	## When to Use This Skill
6
7	- Pipeline execution takes longer than expected
8	- Copy activities are slow or appear stuck
9	- Activities show "Not Started" status for extended periods
10	- Capacity throttling errors (HTTP 430, TooManyRequestsForCapacity)
11	- Throughput is lower than expected for copy operations
12	- Dataflow Gen2 refresh is slow or timing out
13	- Pipeline monitoring shows performance degradation over time
14	- Need to optimize parallelism, DIU, or partitioning settings
15
16	## Prerequisites
17
18	- Access to Microsoft Fabric workspace with Contributor or higher role
19	- Familiarity with the Fabric Monitoring Hub
20	- Understanding of Fabric capacity SKUs and their limits
21	- PowerShell 7+ for running diagnostic scripts
22
23	## Diagnostic Workflow
24
25	### Step 1: Identify the Bottleneck Category
26
27	Determine which category your issue falls into:
28
29	\| Category \| Symptoms \| Start Here \|
30	\| ------------------- \| ------------------------------------------- \| ----------------------------------------------------------------------------------------- \|
31	\| Copy Activity Slow \| Low throughput, long transfer duration \| [copy-activity-tuning.md](./references/copy-activity-tuning.md) \|
32	\| Pipeline Stuck \| Activity shows In Progress with no movement \| [pipeline-stuck-resolution.md](./references/pipeline-stuck-resolution.md) \|
33	\| Capacity Throttling \| HTTP 430 errors, jobs queued \| [capacity-throttling-guide.md](./references/capacity-throttling-guide.md) \|
34	\| Dataflow Slow \| Dataflow Gen2 refresh takes too long \| [dataflow-optimization.md](./references/dataflow-optimization.md) \|
35	\| Spark Job Queue \| Jobs stuck in "Not Started" status \| [capacity-throttling-guide.md](./references/capacity-throttling-guide.md#spark-job-queueing) \|
36
37	### Step 2: Collect Diagnostics
38
39	Run the [diagnostic script](./scripts/Get-FabricPipelineDiagnostics.ps1) to gather baseline metrics:
40
41	```powershell
42	./scripts/Get-FabricPipelineDiagnostics.ps1 -WorkspaceId "<guid>" -PipelineName "MyPipeline"
43	```
44
45	Or manually collect from the Monitoring Hub:
46
47	1. Open Fabric portal and navigate to Monitoring Hub
48	2. Filter by pipeline name and time range
49	3. Select the run details (glasses icon) for the slow run
50	4. Capture the Duration Breakdown for copy activities
51	5. Note the queue time, transfer time, and pre/post-copy script duration
52
53	### Step 3: Apply Targeted Fixes
54
55	Based on the bottleneck category, apply the appropriate optimization from the reference guides.
56
57	## Quick Fixes for Common Issues
58
59	### Copy Activity Running Slowly
60
61	1. Set Intelligent Throughput Optimization to `Maximum` (or custom 4-256)
62	2. Configure Degree of Copy Parallelism based on source type
63	3. Enable Partition Option for SQL sources (Dynamic Range or Physical)
64	4. Pre-calculate partition upper/lower bounds to avoid overhead
65	5. Enable Staging when sink is Fabric Warehouse
66
67	### Pipeline Activity Stuck
68
69	1. Cancel the stuck activity and retry
70	2. Check source/sink connectivity and credentials
71	3. Verify Fabric capacity is not in throttled state
72	4. Review if payload exceeds 896 KB limit
73	5. Check for connection timeout or network interruption
74
75	### Capacity Throttling (HTTP 430)
76
77	1. Check current Spark concurrency against SKU limits
78	2. Cancel unnecessary active Spark jobs via Monitoring Hub
79	3. Consider upgrading to a larger capacity SKU
80	4. Distribute pipeline trigger times to avoid burst load
81	5. Use job queueing for non-interactive Spark workloads
82
83	### Dataflow Gen2 Performance
84
85	1. Reduce data volume with query folding and filters
86	2. Avoid unnecessary data type conversions
87	3. Minimize the number of transformation steps
88	4. Use staging for large datasets
89	5. Check for connector-specific throttling
90
91	## Capacity SKU Quick Reference
92
93	\| SKU \| Max S

People who liked this also viewed

6 picks

Type