backtest-expert

Name: backtest-expert
Author: mphinance

80installs3forks

How to install backtest-expert?

mphinance/alpha-skills/backtest-expert

$npx -y skills add mphinance/alpha-skills --skill backtest-expert

Installs into the current project.

Summary

Expert guidance for systematic backtesting of trading strategies. Use when developing, testing, stress-testing, or validating quantitative trading strategies. Covers "beating ideas to death" methodology, parameter robustness testing, slippage modeling, bias prevention, and interp

Files · 1

View on GitHub

SKILL.md

1	# Backtest Expert
2
3	Systematic approach to backtesting trading strategies based on professional methodology that prioritizes robustness over optimistic results.
4
5	## Core Philosophy
6
7	Goal: Find strategies that "break the least", not strategies that "profit the most" on paper.
8
9	Principle: Add friction, stress test assumptions, and see what survives. If a strategy holds up under pessimistic conditions, it's more likely to work in live trading.
10
11	## When to Use This Skill
12
13	Use this skill when:
14	- Developing or validating systematic trading strategies
15	- Evaluating whether a trading idea is robust enough for live implementation
16	- Troubleshooting why a backtest might be misleading
17	- Learning proper backtesting methodology
18	- Avoiding common pitfalls (curve-fitting, look-ahead bias, survivorship bias)
19	- Assessing parameter sensitivity and regime dependence
20	- Setting realistic expectations for slippage and execution costs
21
22	## Prerequisites
23
24	- Python 3.9+ (for evaluation script)
25	- No API keys required
26	- No external data dependencies — metrics are user-provided
27
28	## Workflow
29
30	### 1. State the Hypothesis
31
32	Define the edge in one sentence.
33
34	Example: "Stocks that gap up >3% on earnings and pull back to previous day's close within first hour provide mean-reversion opportunity."
35
36	If you can't articulate the edge clearly, don't proceed to testing.
37
38	### 2. Codify Rules with Zero Discretion
39
40	Define with complete specificity:
41	- Entry: Exact conditions, timing, price type
42	- Exit: Stop loss, profit target, time-based exit
43	- Position sizing: Fixed $$, % of portfolio, volatility-adjusted
44	- Filters: Market cap, volume, sector, volatility conditions
45	- Universe: What instruments are eligible
46
47	Critical: No subjective judgment allowed. Every decision must be rule-based and unambiguous.
48
49	### 3. Run Initial Backtest
50
51	Test over:
52	- Minimum 5 years (preferably 10+)
53	- Multiple market regimes (bull, bear, high/low volatility)
54	- Realistic costs: Commissions + conservative slippage
55
56	Examine initial results for basic viability. If fundamentally broken, iterate on hypothesis.
57
58	### 4. Stress Test the Strategy
59
60	This is where 80% of testing time should be spent.
61
62	Parameter sensitivity:
63	- Test stop loss at 50%, 75%, 100%, 125%, 150% of baseline
64	- Test profit target at 80%, 90%, 100%, 110%, 120% of baseline
65	- Vary entry/exit timing by ±15-30 minutes
66	- Look for "plateaus" of stable performance, not narrow spikes
67
68	Execution friction:
69	- Increase slippage to 1.5-2x typical estimates
70	- Model worst-case fills (buy at ask+1 tick, sell at bid-1 tick)
71	- Add realistic order rejection scenarios
72	- Test with pessimistic commission structures
73
74	Time robustness:
75	- Analyze year-by-year performance
76	- Require positive expectancy in majority of years
77	- Ensure strategy doesn't rely on 1-2 exceptional periods
78	- Test in different market regimes separately
79
80	Sample size:
81	- Absolute minimum: 30 trades
82	- Preferred: 100+ trades
83	- High confidence: 200+ trades
84
85	### 5. Out-of-Sample Validation
86
87	Walk-forward analysis:
88	1. Optimize on training period (e.g., Year 1-3)
89	2. Test on validation period (Year 4)
90	3. Roll forward and repeat
91	4. Compare in-sample vs out-of-sample performance
92
93	Warning signs:
94	- Out-of-sample <50% of in-sample performance
95	- Need frequent parameter re-optimization
96	- Parameters change dramatically between periods
97
98	### 6. Evaluate Results
99
100	Questions to answer:
101	- Does edge survive pessimistic assumptions?
102	- Is performance stable across parameter variations?
103	- Does strategy work in multiple market regimes?
104	- Is sample size sufficient for statistical confidence?
105	- Are results realistic, not "too good to be true"?
106
107	Decision criteria:
108	- ✅ Deploy: Survives all stress tests with acceptable performance
109	- 🔄 Refine: Core logic sound but needs parameter adjustment
110	- ❌ Abandon: Fails stress tests or relies on fragile assumptions
111
112	Use the evaluation script for a structured, quantitative assessment:
113
114	```bash
115	python3 skills/backtest-expert/scripts/evaluate_backtest.py \
116	--total-trades 150 \
117	--win-rate 62 \
118	--avg-win-pct 1.8 \
119	--avg-loss-pct 1.2 \
120	--max-drawdown-pct 15 \
121	--years-tested 8 \
122	--num-parameters 3 \
123	--slippage-tested \
124	--output-dir reports/
125	```
126
127	The script scores across 5 dimensions (Sample Size, Expectancy, Risk Management, Robustness, Execution Realism), detects red flags, and outputs a Deploy/Refine/Abandon verdict.
128
129	## Key Testing Principles
130
131	### Punish the Strategy
132
133	Add friction ever

People who liked this also viewed

6 picks

Type

backtest-expert

80installs3forks

View on GitHub

How to install backtest-expert?

mphinance/alpha-skills/backtest-expert

$npx -y skills add mphinance/alpha-skills --skill backtest-expert

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Backtest Expert
2
3	Systematic approach to backtesting trading strategies based on professional methodology that prioritizes robustness over optimistic results.
4
5	## Core Philosophy
6
7	Goal: Find strategies that "break the least", not strategies that "profit the most" on paper.
8
9	Principle: Add friction, stress test assumptions, and see what survives. If a strategy holds up under pessimistic conditions, it's more likely to work in live trading.
10
11	## When to Use This Skill
12
13	Use this skill when:
14	- Developing or validating systematic trading strategies
15	- Evaluating whether a trading idea is robust enough for live implementation
16	- Troubleshooting why a backtest might be misleading
17	- Learning proper backtesting methodology
18	- Avoiding common pitfalls (curve-fitting, look-ahead bias, survivorship bias)
19	- Assessing parameter sensitivity and regime dependence
20	- Setting realistic expectations for slippage and execution costs
21
22	## Prerequisites
23
24	- Python 3.9+ (for evaluation script)
25	- No API keys required
26	- No external data dependencies — metrics are user-provided
27
28	## Workflow
29
30	### 1. State the Hypothesis
31
32	Define the edge in one sentence.
33
34	Example: "Stocks that gap up >3% on earnings and pull back to previous day's close within first hour provide mean-reversion opportunity."
35
36	If you can't articulate the edge clearly, don't proceed to testing.
37
38	### 2. Codify Rules with Zero Discretion
39
40	Define with complete specificity:
41	- Entry: Exact conditions, timing, price type
42	- Exit: Stop loss, profit target, time-based exit
43	- Position sizing: Fixed $$, % of portfolio, volatility-adjusted
44	- Filters: Market cap, volume, sector, volatility conditions
45	- Universe: What instruments are eligible
46
47	Critical: No subjective judgment allowed. Every decision must be rule-based and unambiguous.
48
49	### 3. Run Initial Backtest
50
51	Test over:
52	- Minimum 5 years (preferably 10+)
53	- Multiple market regimes (bull, bear, high/low volatility)
54	- Realistic costs: Commissions + conservative slippage
55
56	Examine initial results for basic viability. If fundamentally broken, iterate on hypothesis.
57
58	### 4. Stress Test the Strategy
59
60	This is where 80% of testing time should be spent.
61
62	Parameter sensitivity:
63	- Test stop loss at 50%, 75%, 100%, 125%, 150% of baseline
64	- Test profit target at 80%, 90%, 100%, 110%, 120% of baseline
65	- Vary entry/exit timing by ±15-30 minutes
66	- Look for "plateaus" of stable performance, not narrow spikes
67
68	Execution friction:
69	- Increase slippage to 1.5-2x typical estimates
70	- Model worst-case fills (buy at ask+1 tick, sell at bid-1 tick)
71	- Add realistic order rejection scenarios
72	- Test with pessimistic commission structures
73
74	Time robustness:
75	- Analyze year-by-year performance
76	- Require positive expectancy in majority of years
77	- Ensure strategy doesn't rely on 1-2 exceptional periods
78	- Test in different market regimes separately
79
80	Sample size:
81	- Absolute minimum: 30 trades
82	- Preferred: 100+ trades
83	- High confidence: 200+ trades
84
85	### 5. Out-of-Sample Validation
86
87	Walk-forward analysis:
88	1. Optimize on training period (e.g., Year 1-3)
89	2. Test on validation period (Year 4)
90	3. Roll forward and repeat
91	4. Compare in-sample vs out-of-sample performance
92
93	Warning signs:
94	- Out-of-sample <50% of in-sample performance
95	- Need frequent parameter re-optimization
96	- Parameters change dramatically between periods
97
98	### 6. Evaluate Results
99
100	Questions to answer:
101	- Does edge survive pessimistic assumptions?
102	- Is performance stable across parameter variations?
103	- Does strategy work in multiple market regimes?
104	- Is sample size sufficient for statistical confidence?
105	- Are results realistic, not "too good to be true"?
106
107	Decision criteria:
108	- ✅ Deploy: Survives all stress tests with acceptable performance
109	- 🔄 Refine: Core logic sound but needs parameter adjustment
110	- ❌ Abandon: Fails stress tests or relies on fragile assumptions
111
112	Use the evaluation script for a structured, quantitative assessment:
113
114	```bash
115	python3 skills/backtest-expert/scripts/evaluate_backtest.py \
116	--total-trades 150 \
117	--win-rate 62 \
118	--avg-win-pct 1.8 \
119	--avg-loss-pct 1.2 \
120	--max-drawdown-pct 15 \
121	--years-tested 8 \
122	--num-parameters 3 \
123	--slippage-tested \
124	--output-dir reports/
125	```
126
127	The script scores across 5 dimensions (Sample Size, Expectancy, Risk Management, Robustness, Execution Realism), detects red flags, and outputs a Deploy/Refine/Abandon verdict.
128
129	## Key Testing Principles
130
131	### Punish the Strategy
132
133	Add friction ever

People who liked this also viewed

6 picks

Type