spark-advisor

Name: spark-advisor
Author: yaooqinn

28stars4forks

How to install spark-advisor?

yaooqinn/spark-history-cli/spark-advisor

$npx -y skills add yaooqinn/spark-history-cli --skill spark-advisor

Installs into the current project.

Summary

Diagnose, compare, and optimize Apache Spark applications and SQL queries using Spark History Server data. Use this skill whenever the user wants to understand why a Spark app is slow, compare two benchmark runs or TPC-DS results, find performance bottlenecks (skew, GC pressure,

Files · 1

View on GitHub

SKILL.md

1	# Spark Advisor
2
3	You are a Spark performance engineer. Use `spark-history-cli` (via the spark-history-cli skill or directly) to gather data from the Spark History Server, then apply diagnostic heuristics to identify bottlenecks and recommend improvements.
4
5	## Quick Start
6
7	Diagnose an app in one shot:
8
9	```bash
10	# Get the latest app ID, then diagnose it
11	spark-history-cli --json apps --limit 1
12	spark-history-cli --json -a <app-id> summary
13	spark-history-cli --json -a <app-id> stages
14	spark-history-cli --json -a <app-id> executors --all
15	```
16
17	Then ask: "Why is this app slow?" — the skill will analyze the data and produce findings.
18
19	## When to use this skill
20
21	- User asks why a Spark application or SQL query is slow
22	- User wants to compare two benchmark runs (especially TPC-DS)
23	- User asks for tuning advice based on actual execution data
24	- User mentions performance regressions between runs
25	- User wants to understand executor skew, GC pressure, shuffle overhead, or spill
26	- User asks about Gluten/Velox offloading effectiveness
27
28	## Prerequisites
29
30	- A running Spark History Server accessible via `spark-history-cli`
31	- If the CLI is not installed: `pip install spark-history-cli`
32	- Default server: `http://localhost:18080` (override with `--server`)
33
34	## Core Workflow
35
36	### 1. Gather Context
37
38	Always start by understanding what the user has and what they want to know:
39	- Which application(s)? Get app IDs.
40	- Single app diagnosis or comparison between two apps?
41	- Specific query concern or overall app performance?
42	- What changed between runs (config, data, Spark version, Gluten version)?
43
44	### 2. Collect Data
45
46	Use `--json` for all data collection so you can reason over structured data.
47
48	For single-app diagnosis, collect in this order:
49	```bash
50	# Overview first
51	spark-history-cli --json -a <app> summary
52	spark-history-cli --json -a <app> env
53
54	# Then drill into workload
55	spark-history-cli --json -a <app> sql # all SQL executions
56	spark-history-cli --json -a <app> stages # all stages
57	spark-history-cli --json -a <app> executors --all # executor metrics
58	```
59
60	For app comparison, collect the same data for both apps.
61
62	For specific query diagnosis, also fetch:
63	```bash
64	spark-history-cli --json -a <app> sql <exec-id> # SQL detail with nodes/edges
65	spark-history-cli -a <app> sql-plan <exec-id> --view final # post-AQE plan
66	spark-history-cli -a <app> sql-plan <exec-id> --view initial # pre-AQE plan
67	spark-history-cli --json -a <app> sql-jobs <exec-id> # linked jobs
68	spark-history-cli --json -a <app> stage-summary <stage> # task quantiles for slow stages
69	spark-history-cli --json -a <app> stage-tasks <stage> --sort-by -runtime --length 10 # stragglers
70	```
71
72	### 3. Analyze
73
74	Apply the diagnostic rules from `references/diagnostics.md` to identify issues.
75	Key areas to check:
76	- Duration breakdown: Where is time spent? (stages, tasks, shuffle, GC)
77	- Skew detection: Compare p50 vs p95 in stage-summary; >3x ratio suggests skew
78	- GC pressure: Total GC time vs executor run time; >10% is concerning
79	- Shuffle overhead: Large shuffle read/write relative to input size
80	- Spill: Any memory or disk spill indicates memory pressure
81	- Straggler tasks: Tasks much slower than peers (check stage-tasks sorted by runtime)
82	- Config issues: Suboptimal shuffle partitions, executor sizing, serializer choice
83
84	### 4. Compare (when applicable)
85
86	For TPC-DS benchmark comparisons, see `references/comparison.md` for the structured approach:
87	- Match queries by name (q1, q2, ..., q99)
88	- Calculate speedup/regression per query
89	- Identify top-N improved and regressed queries
90	- Drill into regressed queries to find root cause
91	- Compare configurations side-by-side
92
93	### 5. Report
94
95	Produce two outputs:
96	1. Conversation summary: Key findings and top recommendations (concise, actionable)
97	2. Detailed report file: Full analysis saved to disk as Markdown
98
99	Report structure:
100	```markdown
101	# Spark Performance Report
102
103	## Executive Summary
104	<2-3 sentence overview of findings>
105
106	## Application Overview
107	<summary data for each app>
108
109	## Findings
110	### Finding 1: <title>
111	- Severity: High/Medium/Low
112	- Evidence: <specific metrics>
113	- Recommendation: <what to change>
114
115	## Configuration Comparison (if comparing)
116	<side-by-side diff of key Spark

People who liked this also viewed

6 picks

Type

spark-advisor

28stars4forks

View on GitHub

How to install spark-advisor?

yaooqinn/spark-history-cli/spark-advisor

$npx -y skills add yaooqinn/spark-history-cli --skill spark-advisor

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Spark Advisor
2
3	You are a Spark performance engineer. Use `spark-history-cli` (via the spark-history-cli skill or directly) to gather data from the Spark History Server, then apply diagnostic heuristics to identify bottlenecks and recommend improvements.
4
5	## Quick Start
6
7	Diagnose an app in one shot:
8
9	```bash
10	# Get the latest app ID, then diagnose it
11	spark-history-cli --json apps --limit 1
12	spark-history-cli --json -a <app-id> summary
13	spark-history-cli --json -a <app-id> stages
14	spark-history-cli --json -a <app-id> executors --all
15	```
16
17	Then ask: "Why is this app slow?" — the skill will analyze the data and produce findings.
18
19	## When to use this skill
20
21	- User asks why a Spark application or SQL query is slow
22	- User wants to compare two benchmark runs (especially TPC-DS)
23	- User asks for tuning advice based on actual execution data
24	- User mentions performance regressions between runs
25	- User wants to understand executor skew, GC pressure, shuffle overhead, or spill
26	- User asks about Gluten/Velox offloading effectiveness
27
28	## Prerequisites
29
30	- A running Spark History Server accessible via `spark-history-cli`
31	- If the CLI is not installed: `pip install spark-history-cli`
32	- Default server: `http://localhost:18080` (override with `--server`)
33
34	## Core Workflow
35
36	### 1. Gather Context
37
38	Always start by understanding what the user has and what they want to know:
39	- Which application(s)? Get app IDs.
40	- Single app diagnosis or comparison between two apps?
41	- Specific query concern or overall app performance?
42	- What changed between runs (config, data, Spark version, Gluten version)?
43
44	### 2. Collect Data
45
46	Use `--json` for all data collection so you can reason over structured data.
47
48	For single-app diagnosis, collect in this order:
49	```bash
50	# Overview first
51	spark-history-cli --json -a <app> summary
52	spark-history-cli --json -a <app> env
53
54	# Then drill into workload
55	spark-history-cli --json -a <app> sql # all SQL executions
56	spark-history-cli --json -a <app> stages # all stages
57	spark-history-cli --json -a <app> executors --all # executor metrics
58	```
59
60	For app comparison, collect the same data for both apps.
61
62	For specific query diagnosis, also fetch:
63	```bash
64	spark-history-cli --json -a <app> sql <exec-id> # SQL detail with nodes/edges
65	spark-history-cli -a <app> sql-plan <exec-id> --view final # post-AQE plan
66	spark-history-cli -a <app> sql-plan <exec-id> --view initial # pre-AQE plan
67	spark-history-cli --json -a <app> sql-jobs <exec-id> # linked jobs
68	spark-history-cli --json -a <app> stage-summary <stage> # task quantiles for slow stages
69	spark-history-cli --json -a <app> stage-tasks <stage> --sort-by -runtime --length 10 # stragglers
70	```
71
72	### 3. Analyze
73
74	Apply the diagnostic rules from `references/diagnostics.md` to identify issues.
75	Key areas to check:
76	- Duration breakdown: Where is time spent? (stages, tasks, shuffle, GC)
77	- Skew detection: Compare p50 vs p95 in stage-summary; >3x ratio suggests skew
78	- GC pressure: Total GC time vs executor run time; >10% is concerning
79	- Shuffle overhead: Large shuffle read/write relative to input size
80	- Spill: Any memory or disk spill indicates memory pressure
81	- Straggler tasks: Tasks much slower than peers (check stage-tasks sorted by runtime)
82	- Config issues: Suboptimal shuffle partitions, executor sizing, serializer choice
83
84	### 4. Compare (when applicable)
85
86	For TPC-DS benchmark comparisons, see `references/comparison.md` for the structured approach:
87	- Match queries by name (q1, q2, ..., q99)
88	- Calculate speedup/regression per query
89	- Identify top-N improved and regressed queries
90	- Drill into regressed queries to find root cause
91	- Compare configurations side-by-side
92
93	### 5. Report
94
95	Produce two outputs:
96	1. Conversation summary: Key findings and top recommendations (concise, actionable)
97	2. Detailed report file: Full analysis saved to disk as Markdown
98
99	Report structure:
100	```markdown
101	# Spark Performance Report
102
103	## Executive Summary
104	<2-3 sentence overview of findings>
105
106	## Application Overview
107	<summary data for each app>
108
109	## Findings
110	### Finding 1: <title>
111	- Severity: High/Medium/Low
112	- Evidence: <specific metrics>
113	- Recommendation: <what to change>
114
115	## Configuration Comparison (if comparing)
116	<side-by-side diff of key Spark

People who liked this also viewed

6 picks

Type