$npx -y skills add SafeAI-Lab-X/ClawKeeper --skill openclaw-test-heap-leaksInvestigate pnpm test memory growth, Vitest worker OOMs, and suspicious RSS increases in OpenClaw using the scripts/test-parallel.mjs heap snapshot tooling. Use when Codex needs to reproduce test-lane memory growth, collect repeated .heapsnapshot files, compare snapshots fr
| 1 | # OpenClaw Test Heap Leaks |
| 2 | |
| 3 | Use this skill for test-memory investigations. Do not guess from RSS alone when heap snapshots are available. |
| 4 | |
| 5 | ## Workflow |
| 6 | |
| 7 | 1. Reproduce the failing shape first. |
| 8 | - Match the real entrypoint if possible. For Linux CI-style unit failures, start with: |
| 9 | - `pnpm canvas:a2ui:bundle && OPENCLAW_TEST_MEMORY_TRACE=1 OPENCLAW_TEST_HEAPSNAPSHOT_INTERVAL_MS=60000 OPENCLAW_TEST_HEAPSNAPSHOT_DIR=.tmp/heapsnap OPENCLAW_TEST_WORKERS=2 OPENCLAW_TEST_MAX_OLD_SPACE_SIZE_MB=6144 pnpm test` |
| 10 | - Keep `OPENCLAW_TEST_MEMORY_TRACE=1` enabled so the wrapper prints per-file RSS summaries alongside the snapshots. |
| 11 | - If the report is about a specific shard or worker budget, preserve that shape. |
| 12 | |
| 13 | 2. Wait for repeated snapshots before concluding anything. |
| 14 | - Take at least two intervals from the same lane. |
| 15 | - Compare snapshots from the same PID inside one lane directory such as `.tmp/heapsnap/unit-fast/`. |
| 16 | - Use `scripts/heapsnapshot-delta.mjs` to compare either two files directly or the earliest/latest pair per PID in one lane directory. |
| 17 | |
| 18 | 3. Classify the growth before choosing a fix. |
| 19 | - If growth is dominated by Vite/Vitest transformed source strings, `Module`, `system / Context`, bytecode, descriptor arrays, or property maps, treat it as retained module graph growth in long-lived workers. |
| 20 | - If growth is dominated by app objects, caches, buffers, server handles, timers, mock state, sqlite state, or similar runtime objects, treat it as a likely cleanup or lifecycle leak. |
| 21 | |
| 22 | 4. Fix the right layer. |
| 23 | - For retained transformed-module growth in shared workers: |
| 24 | - Move hotspot files out of `unit-fast` by updating `test/fixtures/test-parallel.behavior.json`. |
| 25 | - Prefer `singletonIsolated` for files that are safe alone but inflate shared worker heaps. |
| 26 | - If the file should already have been peeled out by timings but is absent from `test/fixtures/test-timings.unit.json`, call that out explicitly. Missing timings are a scheduling blind spot. |
| 27 | - For real leaks: |
| 28 | - Patch the implicated test or runtime cleanup path. |
| 29 | - Look for missing `afterEach`/`afterAll`, module-reset gaps, retained global state, unreleased DB handles, or listeners/timers that survive the file. |
| 30 | |
| 31 | 5. Verify with the most direct proof. |
| 32 | - Re-run the targeted lane or file with heap snapshots enabled if the suite still finishes in reasonable time. |
| 33 | - If snapshot overhead pushes tests over Vitest timeouts, fall back to the same lane without snapshots and confirm the RSS trend or OOM is reduced. |
| 34 | - For wrapper-only changes, at minimum verify the expected lanes start and the snapshot files are written. |
| 35 | |
| 36 | ## Heuristics |
| 37 | |
| 38 | - Do not call everything a leak. In this repo, large `unit-fast` growth can be a worker-lifetime problem rather than an application object leak. |
| 39 | - `scripts/test-parallel.mjs` and `scripts/test-parallel-memory.mjs` are the primary control points for wrapper diagnostics. |
| 40 | - The lane names printed by `[test-parallel] start ...` and `[test-parallel][mem] summary ...` tell you where to focus. |
| 41 | - When one or two files account for most of the delta and they are missing from timings, reducing impact by isolating them is usually the first pragmatic fix. |
| 42 | - When the same retained object families grow across multiple intervals in the same worker PID, trust the snapshots over intuition. |
| 43 | |
| 44 | ## Snapshot Comparison |
| 45 | |
| 46 | - Direct comparison: |
| 47 | - `node .agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs before.heapsnapshot after.heapsnapshot` |
| 48 | - Auto-select earliest/latest snapshots per PID within one lane: |
| 49 | - `node .agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs --lane-dir .tmp/heapsnap/unit-fast` |
| 50 | - Useful flags: |
| 51 | - `--top 40` |
| 52 | - `--min-kb 32` |
| 53 | - `--pid 16133` |
| 54 | |
| 55 | Read the top positive deltas first. Large positive growth in module-transform artifacts suggests lane isolation; large positive growth in runtime objects suggests a real leak. |
| 56 | |
| 57 | ## Output Expectations |
| 58 | |
| 59 | When using this skill, report: |
| 60 | |
| 61 | - The exact reproduce command. |
| 62 | - Which lane and PID were compared. |
| 63 | - The dominant retained object families from the snapshot delta. |
| 64 | - Whether the issue is a real leak or shared-worker retained module growth. |
| 65 | - The concrete fix or impact-reduction patch. |
| 66 | - What you verified, and what snapshot overhead prevented you from verifying. |