senior-prompt-engineer

Name: senior-prompt-engineer
Author: hanamizuki

104stars8forks

How to install senior-prompt-engineer?

hanamizuki/solopreneur/senior-prompt-engineer

$npx -y skills add hanamizuki/solopreneur --skill senior-prompt-engineer

Installs into the current project.

Summary

Use when the user asks to optimize prompts, design prompt templates, evaluate LLM outputs with an eval set, measure RAG retrieval quality, validate agent/tool configurations, analyze token usage, or design structured-output contracts. Covers eval-driven prompt iteration, RAG metr

Files · 1

View on GitHub

SKILL.md

1	# Senior Prompt Engineer
2
3	Eval-driven prompt engineering, RAG quality measurement, and agent workflow validation. Everything here is model-agnostic by design: techniques are framed by what they do, not by which model generation they were observed on, and the tools never hardcode model IDs or pricing — you supply your provider's current rates when you want dollar figures.
4
5	## Operating Rules
6
7	1. Never change a prompt without a baseline. Capture metrics first (`--analyze --output baseline.json`), then compare every iteration against it.
8	2. Eval set before optimization. 10–20 representative cases with expected outputs minimum. If the user has no eval set, build one with them before touching the prompt — optimizing against vibes is the #1 failure mode.
9	3. Prefer platform features over prompt hacks. If the provider offers native structured outputs / JSON schema enforcement, tool-use APIs, or prompt caching, use those instead of "respond ONLY with JSON" incantations. Prompt-level format enforcement is the fallback, not the default.
10	4. Current-generation models need less scaffolding. Don't add chain-of-thought boilerplate, role framing, or few-shot examples reflexively — frontier models often do worse with redundant scaffolding. Add each element only when the eval set shows it helps.
11	5. Cost numbers are always user-supplied. Look up the provider's current per-Mtok pricing and pass it via `--price-per-mtok` (never trust a cached price table — including any you remember).
12
13	## Tools (exact CLIs, all stdlib)
14
15	### 1. Prompt Optimizer — `scripts/prompt_optimizer.py`
16
17	Static analysis: token estimate, clarity/structure scores (0–100), ambiguity + redundancy detection, few-shot example extraction.
18
19	```bash
20	# Full analysis (human-readable report)
21	python3 scripts/prompt_optimizer.py prompt.txt --analyze
22
23	# Save machine-readable baseline for later comparison
24	python3 scripts/prompt_optimizer.py prompt.txt --analyze --json --output baseline.json
25
26	# Token estimate; cost only if you supply your provider's current rate
27	python3 scripts/prompt_optimizer.py prompt.txt --tokens --model claude --price-per-mtok 3.00
28
29	# Whitespace/redundancy-trimmed version
30	python3 scripts/prompt_optimizer.py prompt.txt --optimize --output optimized.txt
31
32	# Extract Input/Output few-shot pairs to JSON
33	python3 scripts/prompt_optimizer.py prompt.txt --extract-examples --output examples.json
34
35	# Compare a revision against the saved baseline
36	python3 scripts/prompt_optimizer.py optimized.txt --analyze --compare baseline.json
37	```
38
39	`--model` accepts any string; only the tokenizer family is inferred (names containing "claude" → 3.5 chars/token, otherwise 4.0). Exit 0 on success, 1 on missing file.
40
41	### 2. RAG Evaluator — `scripts/rag_evaluator.py`
42
43	Measures retrieval and grounding quality from two JSON files (formats printed in `--help`).
44
45	```bash
46	python3 scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json
47	python3 scripts/rag_evaluator.py --contexts ctx.json --questions q.json --k 10 --json
48	python3 scripts/rag_evaluator.py --contexts ctx.json --questions q.json --output report.json --verbose
49	python3 scripts/rag_evaluator.py --contexts ctx.json --questions q.json --compare baseline_report.json
50	```
51
52	Reports context relevance, precision@k, coverage, answer faithfulness, groundedness. Treat relevance < 0.80 as a retrieval problem (chunking/embedding/filtering), not a prompt problem — fix retrieval before rewriting the generation prompt.
53
54	### 3. Agent Orchestrator — `scripts/agent_orchestrator.py`
55
56	Validates agent configs (YAML/JSON): tool wiring, missing required config, loop risk, token estimates.
57
58	```bash
59	python3 scripts/agent_orchestrator.py agent.yaml --validate
60	python3 scripts/agent_orchestrator.py agent.yaml --visualize --format mermaid
61	python3 scripts/agent_orchestrator.py agent.yaml --estimate-cost --runs 100 \
62	--input-price-per-mtok 3.00 --output-price-per-mtok 15.00
63	```
64
65	Without the two price flags, `--estimate-cost` reports token estimates only. The `model:` field in the config is informational — any model name is accepted.
66
67	## Workflows
68
69	### Prompt Optimization (eval-gated)
70
71	1. Baseline: `python3 scripts/prompt_optimizer.py current_prompt.txt --analyze --json --output baseline.json`
72	2. Diagnose from the report: ambiguous verbs ("analyze", "handle"), redundant blocks, missing output contract, token waste.
73	3. Apply one change at a time, in this order of leverage:
74	\| Symptom \| Fix \|
75	\|---------\|-----\|
76	\| Malformed/unparse

People who liked this also viewed

6 picks

Type

senior-prompt-engineer

104stars8forks

View on GitHub

How to install senior-prompt-engineer?

hanamizuki/solopreneur/senior-prompt-engineer

$npx -y skills add hanamizuki/solopreneur --skill senior-prompt-engineer

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Senior Prompt Engineer
2
3	Eval-driven prompt engineering, RAG quality measurement, and agent workflow validation. Everything here is model-agnostic by design: techniques are framed by what they do, not by which model generation they were observed on, and the tools never hardcode model IDs or pricing — you supply your provider's current rates when you want dollar figures.
4
5	## Operating Rules
6
7	1. Never change a prompt without a baseline. Capture metrics first (`--analyze --output baseline.json`), then compare every iteration against it.
8	2. Eval set before optimization. 10–20 representative cases with expected outputs minimum. If the user has no eval set, build one with them before touching the prompt — optimizing against vibes is the #1 failure mode.
9	3. Prefer platform features over prompt hacks. If the provider offers native structured outputs / JSON schema enforcement, tool-use APIs, or prompt caching, use those instead of "respond ONLY with JSON" incantations. Prompt-level format enforcement is the fallback, not the default.
10	4. Current-generation models need less scaffolding. Don't add chain-of-thought boilerplate, role framing, or few-shot examples reflexively — frontier models often do worse with redundant scaffolding. Add each element only when the eval set shows it helps.
11	5. Cost numbers are always user-supplied. Look up the provider's current per-Mtok pricing and pass it via `--price-per-mtok` (never trust a cached price table — including any you remember).
12
13	## Tools (exact CLIs, all stdlib)
14
15	### 1. Prompt Optimizer — `scripts/prompt_optimizer.py`
16
17	Static analysis: token estimate, clarity/structure scores (0–100), ambiguity + redundancy detection, few-shot example extraction.
18
19	```bash
20	# Full analysis (human-readable report)
21	python3 scripts/prompt_optimizer.py prompt.txt --analyze
22
23	# Save machine-readable baseline for later comparison
24	python3 scripts/prompt_optimizer.py prompt.txt --analyze --json --output baseline.json
25
26	# Token estimate; cost only if you supply your provider's current rate
27	python3 scripts/prompt_optimizer.py prompt.txt --tokens --model claude --price-per-mtok 3.00
28
29	# Whitespace/redundancy-trimmed version
30	python3 scripts/prompt_optimizer.py prompt.txt --optimize --output optimized.txt
31
32	# Extract Input/Output few-shot pairs to JSON
33	python3 scripts/prompt_optimizer.py prompt.txt --extract-examples --output examples.json
34
35	# Compare a revision against the saved baseline
36	python3 scripts/prompt_optimizer.py optimized.txt --analyze --compare baseline.json
37	```
38
39	`--model` accepts any string; only the tokenizer family is inferred (names containing "claude" → 3.5 chars/token, otherwise 4.0). Exit 0 on success, 1 on missing file.
40
41	### 2. RAG Evaluator — `scripts/rag_evaluator.py`
42
43	Measures retrieval and grounding quality from two JSON files (formats printed in `--help`).
44
45	```bash
46	python3 scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json
47	python3 scripts/rag_evaluator.py --contexts ctx.json --questions q.json --k 10 --json
48	python3 scripts/rag_evaluator.py --contexts ctx.json --questions q.json --output report.json --verbose
49	python3 scripts/rag_evaluator.py --contexts ctx.json --questions q.json --compare baseline_report.json
50	```
51
52	Reports context relevance, precision@k, coverage, answer faithfulness, groundedness. Treat relevance < 0.80 as a retrieval problem (chunking/embedding/filtering), not a prompt problem — fix retrieval before rewriting the generation prompt.
53
54	### 3. Agent Orchestrator — `scripts/agent_orchestrator.py`
55
56	Validates agent configs (YAML/JSON): tool wiring, missing required config, loop risk, token estimates.
57
58	```bash
59	python3 scripts/agent_orchestrator.py agent.yaml --validate
60	python3 scripts/agent_orchestrator.py agent.yaml --visualize --format mermaid
61	python3 scripts/agent_orchestrator.py agent.yaml --estimate-cost --runs 100 \
62	--input-price-per-mtok 3.00 --output-price-per-mtok 15.00
63	```
64
65	Without the two price flags, `--estimate-cost` reports token estimates only. The `model:` field in the config is informational — any model name is accepted.
66
67	## Workflows
68
69	### Prompt Optimization (eval-gated)
70
71	1. Baseline: `python3 scripts/prompt_optimizer.py current_prompt.txt --analyze --json --output baseline.json`
72	2. Diagnose from the report: ambiguous verbs ("analyze", "handle"), redundant blocks, missing output contract, token waste.
73	3. Apply one change at a time, in this order of leverage:
74	\| Symptom \| Fix \|
75	\|---------\|-----\|
76	\| Malformed/unparse

People who liked this also viewed

6 picks

Type