$npx -y skills add mohitmishra786/low-level-dev-skills --skill core-dumpsCore dump analysis skill for production crash triage. Use when loading core files in GDB or LLDB, enabling core dump generation on Linux/macOS, mapping symbols with debuginfo or debuginfod, or extracting backtraces from crashes without re-running the program. Activates on queries
| 1 | # Core Dumps |
| 2 | |
| 3 | ## Purpose |
| 4 | |
| 5 | Guide agents through enabling, collecting, and analysing core dumps for post-mortem crash investigation without rerunning the buggy program. |
| 6 | |
| 7 | ## Triggers |
| 8 | |
| 9 | - "My program crashed in production — how do I analyse the core?" |
| 10 | - "How do I enable core dumps on Linux?" |
| 11 | - "I have a core file but no symbols / source" |
| 12 | - "How do I use debuginfod to get symbols for a core?" |
| 13 | - "coredumpctl show me the crash" |
| 14 | |
| 15 | ## Workflow |
| 16 | |
| 17 | ### 1. Enable core dumps (Linux) |
| 18 | |
| 19 | ```bash |
| 20 | # Per-session (lost on logout) |
| 21 | ulimit -c unlimited |
| 22 | |
| 23 | # Persistent (add to /etc/security/limits.conf) |
| 24 | * soft core unlimited |
| 25 | * hard core unlimited |
| 26 | |
| 27 | # Check current limit |
| 28 | ulimit -c |
| 29 | |
| 30 | # Set core pattern (where and how cores are named) |
| 31 | # Default: 'core' in CWD — often not useful |
| 32 | sudo sysctl -w kernel.core_pattern=/tmp/core-%e-%p-%t |
| 33 | # %e = executable, %p = PID, %t = timestamp |
| 34 | |
| 35 | # Persistent (add to /etc/sysctl.d/99-core.conf) |
| 36 | kernel.core_pattern=/tmp/core-%e-%p-%t |
| 37 | kernel.core_uses_pid=1 |
| 38 | ``` |
| 39 | |
| 40 | ### 2. systemd/coredumpctl (modern Linux) |
| 41 | |
| 42 | If systemd manages core dumps (common on Ubuntu 20+, Fedora, Arch): |
| 43 | |
| 44 | ```bash |
| 45 | # List recent crashes |
| 46 | coredumpctl list |
| 47 | |
| 48 | # Show details of the latest crash |
| 49 | coredumpctl info |
| 50 | |
| 51 | # Load latest crash in GDB |
| 52 | coredumpctl gdb |
| 53 | |
| 54 | # Load specific PID crash |
| 55 | coredumpctl gdb 12345 |
| 56 | |
| 57 | # Export core file |
| 58 | coredumpctl dump -o myapp.core PID |
| 59 | ``` |
| 60 | |
| 61 | Core storage location: `/var/lib/systemd/coredump/`. |
| 62 | |
| 63 | ### 3. Enable core dumps (macOS) |
| 64 | |
| 65 | ```bash |
| 66 | # macOS uses /cores by default (must be root-writable) |
| 67 | ulimit -c unlimited |
| 68 | |
| 69 | # Check |
| 70 | ls /cores/ |
| 71 | |
| 72 | # launchd-launched services: set in plist |
| 73 | # <key>HardResourceLimits</key> |
| 74 | # <dict><key>Core</key><integer>9223372036854775807</integer></dict> |
| 75 | ``` |
| 76 | |
| 77 | ### 4. Analyse a core with GDB |
| 78 | |
| 79 | ```bash |
| 80 | # Load binary and core |
| 81 | gdb ./prog core.12345 |
| 82 | |
| 83 | # If the binary was stripped, provide the unstripped copy |
| 84 | gdb ./prog-with-symbols core.12345 |
| 85 | |
| 86 | # Essential first commands |
| 87 | (gdb) bt # call stack |
| 88 | (gdb) bt full # stack + locals |
| 89 | (gdb) info registers # CPU state at crash |
| 90 | (gdb) frame 2 # jump to interesting frame |
| 91 | (gdb) info locals # local variables in frame |
| 92 | (gdb) print ptr # inspect a pointer |
| 93 | |
| 94 | # All threads (multi-threaded crash) |
| 95 | (gdb) thread apply all bt full |
| 96 | ``` |
| 97 | |
| 98 | ### 5. Analyse a core with LLDB |
| 99 | |
| 100 | ```bash |
| 101 | lldb ./prog -c core.12345 |
| 102 | |
| 103 | # Or |
| 104 | lldb |
| 105 | (lldb) target create ./prog --core core.12345 |
| 106 | |
| 107 | # Commands |
| 108 | (lldb) bt |
| 109 | (lldb) thread backtrace all |
| 110 | (lldb) frame select 2 |
| 111 | (lldb) frame variable |
| 112 | ``` |
| 113 | |
| 114 | ### 6. Missing symbols: debuginfod |
| 115 | |
| 116 | `debuginfod` serves debug symbols from a central server, mapping build IDs to DWARF data. |
| 117 | |
| 118 | ```bash |
| 119 | # Install client (Debian/Ubuntu) |
| 120 | sudo apt install debuginfod |
| 121 | |
| 122 | # Enable (add to ~/.bashrc or /etc/environment) |
| 123 | export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com https://debuginfod.elfutils.org" |
| 124 | |
| 125 | # GDB auto-fetches symbols when DEBUGINFOD_URLS is set |
| 126 | gdb ./prog core |
| 127 | |
| 128 | # Manually query |
| 129 | debuginfod-find debuginfo <build-id> |
| 130 | debuginfod-find source <build-id> /path/to/file.c |
| 131 | ``` |
| 132 | |
| 133 | ### 7. Missing symbols: manual approach |
| 134 | |
| 135 | ```bash |
| 136 | # Check if binary has a build ID |
| 137 | readelf -n ./prog | grep Build |
| 138 | |
| 139 | # Find the correct debug package |
| 140 | # Debian: apt install prog-dbg or prog-dbgsym |
| 141 | # RPM: dnf install prog-debuginfo |
| 142 | |
| 143 | # Point GDB to debug symbols directory |
| 144 | (gdb) set debug-file-directory /usr/lib/debug |
| 145 | |
| 146 | # Or use eu-readelf to dump build ID, then find .debug file |
| 147 | eu-readelf -n ./prog |
| 148 | find /usr/lib/debug -name "*.debug" | xargs eu-readelf -n 2>/dev/null | grep <build-id> |
| 149 | ``` |
| 150 | |
| 151 | ### 8. Strip binaries and keep symbols |
| 152 | |
| 153 | Best practice: build with symbols, strip for distribution, keep an unstripped copy. |
| 154 | |
| 155 | ```bash |
| 156 | # Build |
| 157 | gcc -g -O2 -o prog main.c |
| 158 | |
| 159 | # Separate debug info |
| 160 | objcopy --only-keep-debug prog prog.debug |
| 161 | objcopy --strip-debug prog prog.stripped |
| 162 | |
| 163 | # Add a debuglink so GDB finds the debug file automatically |
| 164 | objcopy --add-gnu-debuglink=prog.debug prog.stripped |
| 165 | |
| 166 | # Deploy prog.stripped; keep prog.debug in a symbols store indexed by build-id |
| 167 | ``` |
| 168 | |
| 169 | ### 9. Quick triage from core without full debug session |
| 170 | |
| 171 | ```bash |
| 172 | # Print backtrace non-interactively |
| 173 | gdb -batch -ex 'bt full' -ex 'thread apply all bt full' ./prog core 2>&1 | tee crash.txt |
| 174 | |
| 175 | # Print registers |
| 176 | gdb -batch -ex 'info registers' ./prog core |
| 177 | |
| 178 | # Check signal that caused crash |
| 179 | gdb -batch -ex 'info signal' ./prog core |
| 180 | ``` |
| 181 | |
| 182 | For a full cheatsheet covering core pattern tokens, coredumpctl, GDB/LLDB commands, debuginfod servers, and strip/symbol workflows, see [references/cheatsheet.md](references/cheatsheet.md). |
| 183 | |
| 184 | ## Related skills |
| 185 | |
| 186 | - Use `skills/debuggers/gdb` for full GDB session details |
| 187 | - Use ` |