factor-benchmark

Name: factor-benchmark
Author: minihellboy

95stars28forks

How to install factor-benchmark?

minihellboy/factorminer/factor-benchmark

$npx -y skills add minihellboy/factorminer --skill factor-benchmark

Installs into the current project.

Summary

Run FactorMiner benchmark workflows — the Table 1 Top-K freeze benchmark, memory and strategy ablations, transaction-cost pressure tests, and the full suite. Use to compare FactorMiner against baselines or to reproduce paper results. Triggers on "benchmark", "ablation", "compare

Files · 1

View on GitHub

SKILL.md

1	# Factor Benchmark
2
3	This skill runs FactorMiner's canonical benchmark surface — the rigorous comparison layer that turns a single run into evidence.
4
5	## Modes
6
7	\| Mode \| What it answers \|
8	\|---\|---\|
9	\| `table1` \| Top-K freeze benchmark across configured universes vs. baselines — the headline reproduction. \|
10	\| `ablation-memory` \| How much does experience memory contribute? \|
11	\| `ablation-strategy` \| Effect of memory policy × dependence metric × backend. \|
12	\| `cost-pressure` \| How does the library hold up under rising transaction costs? \|
13	\| `efficiency` \| Operator- and factor-level runtime/compute cost. \|
14	\| `suite` \| The full benchmark suite in one run. \|
15
16	## Workflow
17
18	### Run a benchmark
19
20	```bash
21	factorminer -o output/bench benchmark table1 --data path/to/market_data.csv
22	```
23
24	```bash
25	factorminer -o output/bench benchmark suite --data path/to/market_data.csv
26	```
27
28	Pass a pre-mined library to benchmark a specific run rather than mining fresh:
29
30	```bash
31	factorminer -o output/bench benchmark table1 \
32	--data market_data.csv \
33	--factor-miner-library output/run1/factor_library.json
34	```
35
36	`efficiency` takes no data — it profiles the engine itself.
37
38	### Read the result
39
40	The CLI prints a per-universe summary (library IC, ICIR, avg \|ρ\|) and writes JSON payloads into the output directory. Fold those JSON files into the research note with `factor-report --benchmark`.
41
42	## Interpreting ablations
43
44	- An ablation that removes a feature and barely moves the metric means that feature is not earning its compute on this dataset — report that plainly.
45	- `cost-pressure` is the honesty check: a library that only wins at zero cost is not a result.
46
47	## Guardrails
48
49	- Benchmark numbers are comparative research evidence, not a performance guarantee.
50	- Use the same dataset and splits across compared runs, or the comparison is meaningless.
51	- Reproduction claims must cite the exact config and run directory.

People who liked this also viewed

6 picks

Type

factor-benchmark

95stars28forks

View on GitHub

How to install factor-benchmark?

minihellboy/factorminer/factor-benchmark

$npx -y skills add minihellboy/factorminer --skill factor-benchmark

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Factor Benchmark
2
3	This skill runs FactorMiner's canonical benchmark surface — the rigorous comparison layer that turns a single run into evidence.
4
5	## Modes
6
7	\| Mode \| What it answers \|
8	\|---\|---\|
9	\| `table1` \| Top-K freeze benchmark across configured universes vs. baselines — the headline reproduction. \|
10	\| `ablation-memory` \| How much does experience memory contribute? \|
11	\| `ablation-strategy` \| Effect of memory policy × dependence metric × backend. \|
12	\| `cost-pressure` \| How does the library hold up under rising transaction costs? \|
13	\| `efficiency` \| Operator- and factor-level runtime/compute cost. \|
14	\| `suite` \| The full benchmark suite in one run. \|
15
16	## Workflow
17
18	### Run a benchmark
19
20	```bash
21	factorminer -o output/bench benchmark table1 --data path/to/market_data.csv
22	```
23
24	```bash
25	factorminer -o output/bench benchmark suite --data path/to/market_data.csv
26	```
27
28	Pass a pre-mined library to benchmark a specific run rather than mining fresh:
29
30	```bash
31	factorminer -o output/bench benchmark table1 \
32	--data market_data.csv \
33	--factor-miner-library output/run1/factor_library.json
34	```
35
36	`efficiency` takes no data — it profiles the engine itself.
37
38	### Read the result
39
40	The CLI prints a per-universe summary (library IC, ICIR, avg \|ρ\|) and writes JSON payloads into the output directory. Fold those JSON files into the research note with `factor-report --benchmark`.
41
42	## Interpreting ablations
43
44	- An ablation that removes a feature and barely moves the metric means that feature is not earning its compute on this dataset — report that plainly.
45	- `cost-pressure` is the honesty check: a library that only wins at zero cost is not a result.
46
47	## Guardrails
48
49	- Benchmark numbers are comparative research evidence, not a performance guarantee.
50	- Use the same dataset and splits across compared runs, or the comparison is meaningless.
51	- Reproduction claims must cite the exact config and run directory.

People who liked this also viewed

6 picks

Type