regression-sweep

Name: regression-sweep
Author: transilienceai

423stars74forks

How to install regression-sweep?

transilienceai/communitytools/regression-sweep

$npx -y skills add transilienceai/communitytools --skill regression-sweep

Installs into the current project.

Summary

Re-validates every previously-confirmed finding against its current target — detects drift (patched, mitigated, re-introduced). Cron-driven weekly sweep across the org's validated finding tree.

Files · 1

View on GitHub

SKILL.md

1	# Regression Sweep
2
3	Walk the entire `validated/*.json` tree, re-fire each finding's `poc.py`, compare output against the recorded `poc_output.txt`, and write a weekly drift report. Mounted onto the cloud-agent task #4.
4
5	## Trigger
6
7	Cron weekly (default Mondays 02:00 UTC). May also be invoked ad-hoc after a major patch deployment.
8
9	## Workflow
10
11	1. Index validated findings. Glob `validated/*.json` and resolve each entry's `FINDING_DIR` (under `findings/finding-NNN/`).
12	2. Per finding:
13	- Re-run `python3 poc.py` with a 60-second timeout.
14	- Capture stdout/stderr into `findings/finding-NNN/evidence/validation/regression-{week}-rerun.txt`.
15	- Diff against `findings/finding-NNN/evidence/validation/poc-rerun-output.txt` (the validator's original re-run output) using a normalized line-set comparison (strip timestamps, request IDs, ephemeral tokens).
16	- Re-check the finding's CVE via `tools/nvd-lookup.py` — has severity changed?
17	3. Classify each finding into one of:
18	- `still_valid` — re-run matches baseline within tolerance, CVSS unchanged.
19	- `drift_severity` — re-run matches, but CVSS shifted ≥1.0 (NVD re-scored).
20	- `newly_invalid` — re-run output diverges, exploit no longer fires. Likely patched.
21	- `newly_revalidated` — finding had been marked `REJECTED` later, but now fires again. Regression.
22	- `inconclusive` — re-run errored (network, target unreachable). Retry next sweep.
23	4. Write report to `artifacts/regression-{YYYYWww}.json` + human-readable `regression-{YYYYWww}.md`. Transitions (`newly_revalidated`, `drift_severity`, `newly_invalid`) appear in the report under explicit headers for analyst review.
24
25	## Output
26
27	```
28	{OUTPUT_DIR}/
29	artifacts/
30	regression-{YYYYWww}.json # machine-readable result
31	regression-{YYYYWww}.md # human summary
32	findings/finding-NNN/evidence/validation/
33	regression-{YYYYWww}-rerun.txt # captured re-run output
34	```
35
36	`regression-{week}.json` schema:
37
38	```json
39	{
40	"week": "2026W19",
41	"swept_at": "2026-05-13T02:00:00Z",
42	"counts": {"still_valid": 47, "drift_severity": 2, "newly_invalid": 5, "newly_revalidated": 1, "inconclusive": 3},
43	"findings": [
44	{"finding_id": "finding-012", "asset": "asset42", "cve": "CVE-2024-12345",
45	"verdict": "newly_invalid", "reason": "PoC output diverged: response now 404",
46	"baseline_cvss": 9.8, "current_cvss": 9.8}
47	]
48	}
49	```
50
51	## Rules
52
53	1. Demonstrate, never disrupt. Re-firing a PoC is observation, not mutation. Every `poc.py` already satisfied the demonstrate-only constraints at validation time (task-03 Safety section); the sweep simply re-executes the same script — which by contract reads a proof signal and exits. If a PoC at re-run attempts a mutating action it should not have contained originally, the sweep aborts that finding with `inconclusive` and emits a stderr WARN — the PoC needs re-validation, not regression scoring.
54	2. Bounded per-PoC time. 60-second timeout per `poc.py`. Timeouts → `inconclusive`, not `newly_invalid`. Avoids false-positive "patched" claims caused by network blips.
55	3. Normalized diff. Strip timestamps (`\d{4}-\d{2}-\d{2}T\d{2}:\d{2}`), request-IDs (`[a-f0-9]{32,}`), and ephemeral session tokens before comparing. Real exploit output is structurally stable.
56	4. No new findings. A regression sweep can flip status of existing findings but cannot create new ones. Newly observed vulns belong to the Validation Run task, not this skill.
57	5. Idempotent. Re-running the same week's sweep overwrites the same `regression-{YYYYWww}.json`. The per-finding `regression-{week}-rerun.txt` is timestamped to preserve history.
58	6. Cap concurrent re-runs. Max 5 parallel PoC re-runs per sweep to avoid hammering production assets.
59
60	## References
61
62	- `reference/diff-normalization.md` — full normalization rules and per-finding output-stability heuristics.

People who liked this also viewed

6 picks

Type

regression-sweep

423stars74forks

View on GitHub

How to install regression-sweep?

transilienceai/communitytools/regression-sweep

$npx -y skills add transilienceai/communitytools --skill regression-sweep

Installs into the current project.

Summary

Re-validates every previously-confirmed finding against its current target — detects drift (patched, mitigated, re-introduced). Cron-driven weekly sweep across the org's validated finding tree.

Files · 1

View on GitHub

SKILL.md

1	# Regression Sweep
2
3	Walk the entire `validated/*.json` tree, re-fire each finding's `poc.py`, compare output against the recorded `poc_output.txt`, and write a weekly drift report. Mounted onto the cloud-agent task #4.
4
5	## Trigger
6
7	Cron weekly (default Mondays 02:00 UTC). May also be invoked ad-hoc after a major patch deployment.
8
9	## Workflow
10
11	1. Index validated findings. Glob `validated/*.json` and resolve each entry's `FINDING_DIR` (under `findings/finding-NNN/`).
12	2. Per finding:
13	- Re-run `python3 poc.py` with a 60-second timeout.
14	- Capture stdout/stderr into `findings/finding-NNN/evidence/validation/regression-{week}-rerun.txt`.
15	- Diff against `findings/finding-NNN/evidence/validation/poc-rerun-output.txt` (the validator's original re-run output) using a normalized line-set comparison (strip timestamps, request IDs, ephemeral tokens).
16	- Re-check the finding's CVE via `tools/nvd-lookup.py` — has severity changed?
17	3. Classify each finding into one of:
18	- `still_valid` — re-run matches baseline within tolerance, CVSS unchanged.
19	- `drift_severity` — re-run matches, but CVSS shifted ≥1.0 (NVD re-scored).
20	- `newly_invalid` — re-run output diverges, exploit no longer fires. Likely patched.
21	- `newly_revalidated` — finding had been marked `REJECTED` later, but now fires again. Regression.
22	- `inconclusive` — re-run errored (network, target unreachable). Retry next sweep.
23	4. Write report to `artifacts/regression-{YYYYWww}.json` + human-readable `regression-{YYYYWww}.md`. Transitions (`newly_revalidated`, `drift_severity`, `newly_invalid`) appear in the report under explicit headers for analyst review.
24
25	## Output
26
27	```
28	{OUTPUT_DIR}/
29	artifacts/
30	regression-{YYYYWww}.json # machine-readable result
31	regression-{YYYYWww}.md # human summary
32	findings/finding-NNN/evidence/validation/
33	regression-{YYYYWww}-rerun.txt # captured re-run output
34	```
35
36	`regression-{week}.json` schema:
37
38	```json
39	{
40	"week": "2026W19",
41	"swept_at": "2026-05-13T02:00:00Z",
42	"counts": {"still_valid": 47, "drift_severity": 2, "newly_invalid": 5, "newly_revalidated": 1, "inconclusive": 3},
43	"findings": [
44	{"finding_id": "finding-012", "asset": "asset42", "cve": "CVE-2024-12345",
45	"verdict": "newly_invalid", "reason": "PoC output diverged: response now 404",
46	"baseline_cvss": 9.8, "current_cvss": 9.8}
47	]
48	}
49	```
50
51	## Rules
52
53	1. Demonstrate, never disrupt. Re-firing a PoC is observation, not mutation. Every `poc.py` already satisfied the demonstrate-only constraints at validation time (task-03 Safety section); the sweep simply re-executes the same script — which by contract reads a proof signal and exits. If a PoC at re-run attempts a mutating action it should not have contained originally, the sweep aborts that finding with `inconclusive` and emits a stderr WARN — the PoC needs re-validation, not regression scoring.
54	2. Bounded per-PoC time. 60-second timeout per `poc.py`. Timeouts → `inconclusive`, not `newly_invalid`. Avoids false-positive "patched" claims caused by network blips.
55	3. Normalized diff. Strip timestamps (`\d{4}-\d{2}-\d{2}T\d{2}:\d{2}`), request-IDs (`[a-f0-9]{32,}`), and ephemeral session tokens before comparing. Real exploit output is structurally stable.
56	4. No new findings. A regression sweep can flip status of existing findings but cannot create new ones. Newly observed vulns belong to the Validation Run task, not this skill.
57	5. Idempotent. Re-running the same week's sweep overwrites the same `regression-{YYYYWww}.json`. The per-finding `regression-{week}-rerun.txt` is timestamped to preserve history.
58	6. Cap concurrent re-runs. Max 5 parallel PoC re-runs per sweep to avoid hammering production assets.
59
60	## References
61
62	- `reference/diff-normalization.md` — full normalization rules and per-finding output-stability heuristics.

People who liked this also viewed

6 picks

Type