fabric-pandas-perf-remediate

Name: fabric-pandas-perf-remediate
Author: PatrickGallucci

16stars1forks

How to install fabric-pandas-perf-remediate?

patrickgallucci/fabric-skills/fabric-pandas-perf-remediate

$npx -y skills add PatrickGallucci/fabric-skills --skill fabric-pandas-perf-remediate

Installs into the current project.

Summary

Troubleshoot and optimize pandas performance in Microsoft Fabric Spark notebooks. Use when diagnosing slow pandas operations, toPandas() out-of-memory errors, pandas API on Spark (pyspark.pandas) bottlenecks, DataFrame conversion failures, collect() memory issues, driver memory e

Files · 1

View on GitHub

SKILL.md

1	# Fabric Pandas Performance Troubleshooting
2
3	Diagnose and resolve pandas-related performance issues in Microsoft Fabric Spark notebooks, including memory exhaustion, slow conversions, and suboptimal pandas API on Spark usage.
4
5	## When to Use This Skill
6
7	- Notebook cells hang or timeout during pandas operations
8	- `toPandas()` fails with OutOfMemoryError or Java heap space errors
9	- `collect()` crashes the driver node
10	- Pandas API on Spark (`pyspark.pandas` / `ps`) runs slower than expected
11	- DataFrame conversion between Spark and pandas causes memory spikes
12	- Notebook kernel restarts unexpectedly during data processing
13	- Large dataset operations exhaust driver memory on Fabric capacity
14	- Need to choose between pandas, Spark DataFrame, or pandas API on Spark
15
16	## Prerequisites
17
18	- Microsoft Fabric workspace with Data Engineering experience
19	- Fabric capacity F2 or higher (F64+ recommended for large datasets)
20	- PySpark notebook with Spark session active
21	- Basic familiarity with pandas and PySpark DataFrames
22
23	## Quick Diagnosis
24
25	### Symptom-to-Solution Map
26
27	\| Symptom \| Likely Cause \| Jump To \|
28	\|---------\|-------------\|---------\|
29	\| `toPandas()` OOM error \| Dataset too large for driver \| [toPandas Optimization](#topandas-optimization) \|
30	\| Kernel restart during pandas op \| Driver memory exhausted \| [Driver Memory Tuning](#driver-memory-tuning) \|
31	\| `pyspark.pandas` slower than native pandas \| Spark overhead on small data \| [Right-Size Your Approach](#right-size-your-approach) \|
32	\| Slow groupby/merge in pandas API on Spark \| Excessive shuffling \| [Shuffle Optimization](#shuffle-optimization) \|
33	\| Cell timeout on DataFrame conversion \| Large collect to driver \| [Incremental Processing](#incremental-processing) \|
34	\| `ArrowInvalid` or conversion errors \| Schema mismatch / nulls \| [Arrow Conversion Fixes](#arrow-conversion-fixes) \|
35	\| High memory but slow pandas operations \| GC pressure / fragmentation \| [Memory Profiling](#memory-profiling) \|
36
37	## Right-Size Your Approach
38
39	Critical Decision: Choose the right DataFrame API for your data size and workload.
40
41	```
42	Dataset Size Decision Tree:
43	─────────────────────────────────────────────────────────
44	< 100 MB → Native pandas (pd.DataFrame)
45	100 MB - 1 GB → pandas API on Spark (ps.DataFrame)
46	> 1 GB → PySpark DataFrame (spark.DataFrame)
47	> 10 GB → PySpark + partitioning + Delta optimization
48
49	Mixed workload? → Process in Spark, convert final aggregation to pandas
50	Visualization? → Aggregate in Spark first, toPandas() on summary only
51	ML feature eng? → Spark for transforms, pandas for final model input
52	```
53
54	### API Comparison
55
56	\| Operation \| Native pandas \| pandas API on Spark \| PySpark DataFrame \|
57	\|-----------\|--------------\|--------------------\|--------------------\|
58	\| Memory model \| Single-node (driver) \| Distributed \| Distributed \|
59	\| Max practical size \| ~2-4 GB \| 10s-100s GB \| TB+ \|
60	\| Startup overhead \| None \| Spark session \| Spark session \|
61	\| groupby speed (small) \| Fast \| Slower (shuffle) \| Slower (shuffle) \|
62	\| groupby speed (large) \| OOM risk \| Fast \| Fast \|
63	\| Interop with Spark \| `.toPandas()` \| `.to_spark()` \| Native \|
64
65	## toPandas Optimization
66
67	### Problem
68	`toPandas()` collects the entire distributed DataFrame to the single driver node. This is the #1 cause of OOM in Fabric notebooks.
69
70	### Solutions (Progressive)
71
72	1. Reduce data BEFORE conversion
73	```python
74	# BAD - converts entire table
75	pdf = spark_df.toPandas()
76
77	# GOOD - filter and select first
78	pdf = (spark_df
79	.filter("date >= '2024-01-01'")
80	.select("customer_id", "revenue", "region")
81	.toPandas())
82	```
83
84	2. Aggregate in Spark, convert summary
85	```python
86	# BAD - convert raw data then aggregate in pandas
87	pdf = spark_df.toPandas()
88	result = pdf.groupby('region')['revenue'].sum()
89
90	# GOOD - aggregate in Spark first
91	summary = spark_df.groupBy("region").agg(F.sum("revenue").alias("total_revenue"))
92	pdf = summary.toPandas() # Only converting small aggregated result
93	```
94
95	3. Enable Apache Arrow for faster conversion
96	```python
97	# Enable Arrow-based columnar transfer (3-100x faster)
98	spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")
99	spark.conf.set("spark.sql.execution.arrow.pyspark.fallback.enabled", "true")
100
101	# Now toPandas() uses Arrow columnar format
102	pdf = spark_df.toPandas()
103	```
104
105	4. Use sampling for exploration
106	```python
107	# Sample before conver

People who liked this also viewed

6 picks

Type

fabric-pandas-perf-remediate

16stars1forks

View on GitHub

How to install fabric-pandas-perf-remediate?

patrickgallucci/fabric-skills/fabric-pandas-perf-remediate

$npx -y skills add PatrickGallucci/fabric-skills --skill fabric-pandas-perf-remediate

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Fabric Pandas Performance Troubleshooting
2
3	Diagnose and resolve pandas-related performance issues in Microsoft Fabric Spark notebooks, including memory exhaustion, slow conversions, and suboptimal pandas API on Spark usage.
4
5	## When to Use This Skill
6
7	- Notebook cells hang or timeout during pandas operations
8	- `toPandas()` fails with OutOfMemoryError or Java heap space errors
9	- `collect()` crashes the driver node
10	- Pandas API on Spark (`pyspark.pandas` / `ps`) runs slower than expected
11	- DataFrame conversion between Spark and pandas causes memory spikes
12	- Notebook kernel restarts unexpectedly during data processing
13	- Large dataset operations exhaust driver memory on Fabric capacity
14	- Need to choose between pandas, Spark DataFrame, or pandas API on Spark
15
16	## Prerequisites
17
18	- Microsoft Fabric workspace with Data Engineering experience
19	- Fabric capacity F2 or higher (F64+ recommended for large datasets)
20	- PySpark notebook with Spark session active
21	- Basic familiarity with pandas and PySpark DataFrames
22
23	## Quick Diagnosis
24
25	### Symptom-to-Solution Map
26
27	\| Symptom \| Likely Cause \| Jump To \|
28	\|---------\|-------------\|---------\|
29	\| `toPandas()` OOM error \| Dataset too large for driver \| [toPandas Optimization](#topandas-optimization) \|
30	\| Kernel restart during pandas op \| Driver memory exhausted \| [Driver Memory Tuning](#driver-memory-tuning) \|
31	\| `pyspark.pandas` slower than native pandas \| Spark overhead on small data \| [Right-Size Your Approach](#right-size-your-approach) \|
32	\| Slow groupby/merge in pandas API on Spark \| Excessive shuffling \| [Shuffle Optimization](#shuffle-optimization) \|
33	\| Cell timeout on DataFrame conversion \| Large collect to driver \| [Incremental Processing](#incremental-processing) \|
34	\| `ArrowInvalid` or conversion errors \| Schema mismatch / nulls \| [Arrow Conversion Fixes](#arrow-conversion-fixes) \|
35	\| High memory but slow pandas operations \| GC pressure / fragmentation \| [Memory Profiling](#memory-profiling) \|
36
37	## Right-Size Your Approach
38
39	Critical Decision: Choose the right DataFrame API for your data size and workload.
40
41	```
42	Dataset Size Decision Tree:
43	─────────────────────────────────────────────────────────
44	< 100 MB → Native pandas (pd.DataFrame)
45	100 MB - 1 GB → pandas API on Spark (ps.DataFrame)
46	> 1 GB → PySpark DataFrame (spark.DataFrame)
47	> 10 GB → PySpark + partitioning + Delta optimization
48
49	Mixed workload? → Process in Spark, convert final aggregation to pandas
50	Visualization? → Aggregate in Spark first, toPandas() on summary only
51	ML feature eng? → Spark for transforms, pandas for final model input
52	```
53
54	### API Comparison
55
56	\| Operation \| Native pandas \| pandas API on Spark \| PySpark DataFrame \|
57	\|-----------\|--------------\|--------------------\|--------------------\|
58	\| Memory model \| Single-node (driver) \| Distributed \| Distributed \|
59	\| Max practical size \| ~2-4 GB \| 10s-100s GB \| TB+ \|
60	\| Startup overhead \| None \| Spark session \| Spark session \|
61	\| groupby speed (small) \| Fast \| Slower (shuffle) \| Slower (shuffle) \|
62	\| groupby speed (large) \| OOM risk \| Fast \| Fast \|
63	\| Interop with Spark \| `.toPandas()` \| `.to_spark()` \| Native \|
64
65	## toPandas Optimization
66
67	### Problem
68	`toPandas()` collects the entire distributed DataFrame to the single driver node. This is the #1 cause of OOM in Fabric notebooks.
69
70	### Solutions (Progressive)
71
72	1. Reduce data BEFORE conversion
73	```python
74	# BAD - converts entire table
75	pdf = spark_df.toPandas()
76
77	# GOOD - filter and select first
78	pdf = (spark_df
79	.filter("date >= '2024-01-01'")
80	.select("customer_id", "revenue", "region")
81	.toPandas())
82	```
83
84	2. Aggregate in Spark, convert summary
85	```python
86	# BAD - convert raw data then aggregate in pandas
87	pdf = spark_df.toPandas()
88	result = pdf.groupby('region')['revenue'].sum()
89
90	# GOOD - aggregate in Spark first
91	summary = spark_df.groupBy("region").agg(F.sum("revenue").alias("total_revenue"))
92	pdf = summary.toPandas() # Only converting small aggregated result
93	```
94
95	3. Enable Apache Arrow for faster conversion
96	```python
97	# Enable Arrow-based columnar transfer (3-100x faster)
98	spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")
99	spark.conf.set("spark.sql.execution.arrow.pyspark.fallback.enabled", "true")
100
101	# Now toPandas() uses Arrow columnar format
102	pdf = spark_df.toPandas()
103	```
104
105	4. Use sampling for exploration
106	```python
107	# Sample before conver

People who liked this also viewed

6 picks

Type