fabric-dataflows-perf-remediate

Name: fabric-dataflows-perf-remediate
Author: PatrickGallucci

16stars1forks

How to install fabric-dataflows-perf-remediate?

patrickgallucci/fabric-skills/fabric-dataflows-perf-remediate

$npx -y skills add PatrickGallucci/fabric-skills --skill fabric-dataflows-perf-remediate

Installs into the current project.

Summary

Diagnose and resolve Microsoft Fabric Dataflow Gen2 performance issues including slow refresh times, Fast Copy optimization, query folding failures, staging bottlenecks, gateway latency, incremental refresh tuning, capacity throttling, and data destination write performance. Use

Files · 1

View on GitHub

SKILL.md

1	# Dataflows Gen2 Performance Troubleshooting
2
3	Systematic diagnostic workflows for identifying and resolving performance issues in Microsoft Fabric Dataflow Gen2 workloads covering data movement, transformation execution, staging configuration, and destination write optimization.
4
5	## When to Use This Skill
6
7	- Dataflow Gen2 refresh takes longer than expected
8	- Fast Copy is not being utilized or is failing
9	- Query folding indicators show red or yellow steps
10	- Staging-to-destination data movement is a bottleneck
11	- Gateway-connected dataflows have high latency
12	- Incremental refresh is slower than full refresh
13	- Capacity throttling affects dataflow execution
14	- Refresh failures with permission or connector errors
15	- Need to automate dataflow monitoring and health checks
16	- Migrating from Dataflow Gen1 and seeing performance regressions
17
18	## Prerequisites
19
20	- Microsoft Fabric workspace with Data Factory enabled
21	- Contributor or higher role on the workspace
22	- PowerShell 7+ with Az.Accounts module for automation scripts
23	- Access to Fabric Monitoring Hub for refresh history analysis
24	- Fabric Capacity Metrics app access for CU consumption review
25
26	---
27
28	## Quick Diagnosis: Symptom-to-Solution Map
29
30	\| Symptom \| Likely Cause \| Jump To \|
31	\|---------\|-------------\|---------\|
32	\| Refresh takes 2x+ longer than Gen1 \| Delta Parquet output overhead or staging misconfiguration \| [Consideration 5 in Performance Guide](./references/performance-optimization-guide.md#gen1-vs-gen2-runtime) \|
33	\| Fast Copy not activating \| Unsupported connector or non-foldable transforms \| [Fast Copy Diagnostics](#fast-copy-diagnostics) \|
34	\| "Insufficient permissions for staging artifacts" \| Creator token expired (90+ days) \| [Common Errors](#common-refresh-errors) \|
35	\| Slow design-time previews \| Large dataset loaded in editor \| [Design-Time Optimization](#design-time-optimization) \|
36	\| Staging-to-Lakehouse write is slow \| Extra data hop through staging Warehouse \| [Staging Strategy](#staging-strategy-decision-tree) \|
37	\| Gateway dataflows are slow \| All transforms running on gateway host \| [Gateway Optimization](#gateway-performance) \|
38	\| Incremental refresh slower than full \| Too many small buckets creating overhead \| [Incremental Refresh Tuning](#incremental-refresh-tuning) \|
39	\| HTTP 430 / capacity throttled \| Concurrent Spark jobs exhausting CUs \| [Capacity Management](#capacity-and-throttling) \|
40
41	---
42
43	## Diagnostic Workflow
44
45	### Step 1: Check Refresh History
46
47	1. Open the Fabric workspace and locate the dataflow
48	2. Select the ellipsis (...) > Recent runs
49	3. Review the refresh history for status, duration, and type
50	4. Select a specific refresh Start time to drill into details
51	5. Examine the Tables section for per-entity timing
52	6. Examine the Activities section for destination write timing
53	7. Download detailed logs (bottom-left button) for deep analysis
54	8. Download CSV of refresh runs for trend analysis
55
56	Key metrics to capture: total duration, per-table duration, bytes read/written, rows read/written, engine type (Mashup vs CopyActivity vs SQL DW).
57
58	### Step 2: Identify the Bottleneck Component
59
60	Dataflow Gen2 has three performance-critical components:
61
62	```
63	[Data Source] --> [Dataflow Engine] --> [Data Destination]
64	\| \| \|
65	Connectors Mashup / Fast Copy / Lakehouse /
66	+ Gateway SQL DW Compute Warehouse /
67	SQL Database
68	```
69
70	- Data Source: Connector speed, network latency, gateway throughput
71	- Dataflow Engine: Query folding, staging, Fast Copy, Mashup engine
72	- Data Destination: Write performance, Delta Parquet conversion
73
74	### Step 3: Apply Targeted Optimization
75
76	Based on the bottleneck identified, follow the relevant section below.
77
78	---
79
80	## Fast Copy Diagnostics
81
82	Fast Copy provides up to 9x faster ingestion for supported scenarios.
83
84	### Supported Connectors
85
86	ADLS Gen2, Azure Blob Storage, Azure SQL DB, Lakehouse, PostgreSQL, On-premises SQL Server, Warehouse, Oracle, Snowflake, SQL Database in Fabric.
87
88	### Fast Copy Indicators
89
90	\| Indicator \| Meaning \| Action \|
91	\|-----------\|---------\|--------\|
92	\| Green \| Step supported by Fast Copy \| No action needed \|
93	\| Yellow \| Some steps may support Fast C

People who liked this also viewed

6 picks

Type

fabric-dataflows-perf-remediate

16stars1forks

View on GitHub

How to install fabric-dataflows-perf-remediate?

patrickgallucci/fabric-skills/fabric-dataflows-perf-remediate

$npx -y skills add PatrickGallucci/fabric-skills --skill fabric-dataflows-perf-remediate

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Dataflows Gen2 Performance Troubleshooting
2
3	Systematic diagnostic workflows for identifying and resolving performance issues in Microsoft Fabric Dataflow Gen2 workloads covering data movement, transformation execution, staging configuration, and destination write optimization.
4
5	## When to Use This Skill
6
7	- Dataflow Gen2 refresh takes longer than expected
8	- Fast Copy is not being utilized or is failing
9	- Query folding indicators show red or yellow steps
10	- Staging-to-destination data movement is a bottleneck
11	- Gateway-connected dataflows have high latency
12	- Incremental refresh is slower than full refresh
13	- Capacity throttling affects dataflow execution
14	- Refresh failures with permission or connector errors
15	- Need to automate dataflow monitoring and health checks
16	- Migrating from Dataflow Gen1 and seeing performance regressions
17
18	## Prerequisites
19
20	- Microsoft Fabric workspace with Data Factory enabled
21	- Contributor or higher role on the workspace
22	- PowerShell 7+ with Az.Accounts module for automation scripts
23	- Access to Fabric Monitoring Hub for refresh history analysis
24	- Fabric Capacity Metrics app access for CU consumption review
25
26	---
27
28	## Quick Diagnosis: Symptom-to-Solution Map
29
30	\| Symptom \| Likely Cause \| Jump To \|
31	\|---------\|-------------\|---------\|
32	\| Refresh takes 2x+ longer than Gen1 \| Delta Parquet output overhead or staging misconfiguration \| [Consideration 5 in Performance Guide](./references/performance-optimization-guide.md#gen1-vs-gen2-runtime) \|
33	\| Fast Copy not activating \| Unsupported connector or non-foldable transforms \| [Fast Copy Diagnostics](#fast-copy-diagnostics) \|
34	\| "Insufficient permissions for staging artifacts" \| Creator token expired (90+ days) \| [Common Errors](#common-refresh-errors) \|
35	\| Slow design-time previews \| Large dataset loaded in editor \| [Design-Time Optimization](#design-time-optimization) \|
36	\| Staging-to-Lakehouse write is slow \| Extra data hop through staging Warehouse \| [Staging Strategy](#staging-strategy-decision-tree) \|
37	\| Gateway dataflows are slow \| All transforms running on gateway host \| [Gateway Optimization](#gateway-performance) \|
38	\| Incremental refresh slower than full \| Too many small buckets creating overhead \| [Incremental Refresh Tuning](#incremental-refresh-tuning) \|
39	\| HTTP 430 / capacity throttled \| Concurrent Spark jobs exhausting CUs \| [Capacity Management](#capacity-and-throttling) \|
40
41	---
42
43	## Diagnostic Workflow
44
45	### Step 1: Check Refresh History
46
47	1. Open the Fabric workspace and locate the dataflow
48	2. Select the ellipsis (...) > Recent runs
49	3. Review the refresh history for status, duration, and type
50	4. Select a specific refresh Start time to drill into details
51	5. Examine the Tables section for per-entity timing
52	6. Examine the Activities section for destination write timing
53	7. Download detailed logs (bottom-left button) for deep analysis
54	8. Download CSV of refresh runs for trend analysis
55
56	Key metrics to capture: total duration, per-table duration, bytes read/written, rows read/written, engine type (Mashup vs CopyActivity vs SQL DW).
57
58	### Step 2: Identify the Bottleneck Component
59
60	Dataflow Gen2 has three performance-critical components:
61
62	```
63	[Data Source] --> [Dataflow Engine] --> [Data Destination]
64	\| \| \|
65	Connectors Mashup / Fast Copy / Lakehouse /
66	+ Gateway SQL DW Compute Warehouse /
67	SQL Database
68	```
69
70	- Data Source: Connector speed, network latency, gateway throughput
71	- Dataflow Engine: Query folding, staging, Fast Copy, Mashup engine
72	- Data Destination: Write performance, Delta Parquet conversion
73
74	### Step 3: Apply Targeted Optimization
75
76	Based on the bottleneck identified, follow the relevant section below.
77
78	---
79
80	## Fast Copy Diagnostics
81
82	Fast Copy provides up to 9x faster ingestion for supported scenarios.
83
84	### Supported Connectors
85
86	ADLS Gen2, Azure Blob Storage, Azure SQL DB, Lakehouse, PostgreSQL, On-premises SQL Server, Warehouse, Oracle, Snowflake, SQL Database in Fabric.
87
88	### Fast Copy Indicators
89
90	\| Indicator \| Meaning \| Action \|
91	\|-----------\|---------\|--------\|
92	\| Green \| Step supported by Fast Copy \| No action needed \|
93	\| Yellow \| Some steps may support Fast C

People who liked this also viewed

6 picks

Type