$npx -y skills add mohitmishra786/low-level-dev-skills --skill concurrency-debuggingConcurrency debugging skill for diagnosing data races and deadlocks. Use when reading TSan race reports, debugging deadlocks with GDB thread inspection, analyzing lock-order graphs with Helgrind, identifying std::atomic misuse patterns, or reasoning about happens-before in C++ an
| 1 | # Concurrency Debugging |
| 2 | |
| 3 | ## Purpose |
| 4 | |
| 5 | Guide agents through diagnosing and fixing concurrency bugs: reading ThreadSanitizer race reports, using Helgrind for lock-order analysis, detecting deadlocks with GDB thread inspection, identifying common `std::atomic` misuse patterns, and applying happens-before reasoning in C++ and Rust. |
| 6 | |
| 7 | ## Triggers |
| 8 | |
| 9 | - "ThreadSanitizer reported a data race — how do I read the report?" |
| 10 | - "My program deadlocks — how do I debug it?" |
| 11 | - "How do I use Helgrind to find threading bugs?" |
| 12 | - "Am I using std::atomic correctly?" |
| 13 | - "How does happens-before work in C++ memory ordering?" |
| 14 | - "How do I find which threads are deadlocked in GDB?" |
| 15 | |
| 16 | ## Workflow |
| 17 | |
| 18 | ### 1. ThreadSanitizer (TSan) — race detection |
| 19 | |
| 20 | ```bash |
| 21 | # Build with TSan |
| 22 | clang -fsanitize=thread -g -O1 -o prog main.c |
| 23 | # or GCC |
| 24 | gcc -fsanitize=thread -g -O1 -o prog main.c |
| 25 | |
| 26 | # Run (TSan intercepts memory accesses at runtime) |
| 27 | ./prog |
| 28 | |
| 29 | # TSan-specific options |
| 30 | TSAN_OPTIONS="halt_on_error=1:second_deadlock_stack=1" ./prog |
| 31 | ``` |
| 32 | |
| 33 | Reading a TSan report: |
| 34 | |
| 35 | ```text |
| 36 | WARNING: ThreadSanitizer: data race (pid=12345) |
| 37 | Write of size 4 at 0x7f1234 by thread T2: |
| 38 | #0 increment /src/counter.c:8:5 ← access site in T2 |
| 39 | #1 worker_thread /src/counter.c:22:3 |
| 40 | |
| 41 | Previous read of size 4 at 0x7f1234 by thread T1: |
| 42 | #0 read_counter /src/counter.c:3:14 ← conflicting access in T1 |
| 43 | #1 main /src/counter.c:30:5 |
| 44 | |
| 45 | Thread T2 created at: |
| 46 | #0 pthread_create .../tsan_interceptors.cpp |
| 47 | #1 main /src/counter.c:28:3 |
| 48 | |
| 49 | SUMMARY: ThreadSanitizer: data race /src/counter.c:8:5 in increment |
| 50 | ``` |
| 51 | |
| 52 | How to read: |
| 53 | 1. Line 1: type of access (write/read) and address |
| 54 | 2. Stack under "Write of size": the thread that performed the write |
| 55 | 3. Stack under "Previous read/write": the conflicting thread |
| 56 | 4. "Thread T2 created at": where the thread was spawned |
| 57 | 5. Fix: the `increment` and `read_counter` functions access the same address without synchronization |
| 58 | |
| 59 | Common races and fixes: |
| 60 | |
| 61 | | Race pattern | Fix | |
| 62 | |-------------|-----| |
| 63 | | Read/write on global without lock | Add mutex or use `std::atomic` | |
| 64 | | Double-checked locking without `atomic` | Use `std::once_flag` + `std::call_once` | |
| 65 | | `+=` on shared integer | Use `std::atomic<int>::fetch_add()` | |
| 66 | | Container modified while iterated | Lock entire critical section | |
| 67 | | `shared_ptr` ref count race | Already safe (ref count is atomic); but pointed-to object may not be | |
| 68 | |
| 69 | ### 2. Helgrind — lock-order and race detection |
| 70 | |
| 71 | Helgrind uses Valgrind infrastructure to detect lock ordering violations (potential deadlocks) and data races: |
| 72 | |
| 73 | ```bash |
| 74 | # Run with Helgrind |
| 75 | valgrind --tool=helgrind --log-file=helgrind.log ./prog |
| 76 | |
| 77 | # Lock order violation report |
| 78 | ==1234== Thread #3: lock order "0x... M2" after "0x... M1" |
| 79 | ==1234== observed (incorrect) order |
| 80 | ==1234== at pthread_mutex_lock (helgrind/...) |
| 81 | ==1234== by worker2 /src/worker.c:45 ← T3 takes M2 then M1 |
| 82 | ==1234== |
| 83 | ==1234== required order established by acquisition of lock at address 0x... M1 |
| 84 | ==1234== at pthread_mutex_lock |
| 85 | ==1234== by worker1 /src/worker.c:31 ← T1 takes M1 then M2 |
| 86 | ``` |
| 87 | |
| 88 | Lock-order violation = potential deadlock: |
| 89 | - Thread T1 acquires M1, then tries M2 |
| 90 | - Thread T2 acquires M2, then tries M1 |
| 91 | - Both can deadlock if they race |
| 92 | |
| 93 | Fix: enforce a consistent global lock ordering. Always take M1 before M2 everywhere. |
| 94 | |
| 95 | ### 3. Deadlock detection with GDB |
| 96 | |
| 97 | ```bash |
| 98 | # Attach GDB to a deadlocked process |
| 99 | gdb -p $(pgrep prog) |
| 100 | |
| 101 | # Or run under GDB then trigger deadlock |
| 102 | |
| 103 | (gdb) info threads # list all threads and current state |
| 104 | # * 1 Thread 0x... (LWP 1234) "prog" ... in __lll_lock_wait () |
| 105 | # 2 Thread 0x... (LWP 1235) "prog" ... in __lll_lock_wait () |
| 106 | # Threads blocked in __lll_lock_wait = waiting for mutex |
| 107 | |
| 108 | (gdb) thread 1 |
| 109 | (gdb) bt # show which mutex thread 1 is waiting for |
| 110 | |
| 111 | (gdb) thread 2 |
| 112 | (gdb) bt # show which mutex thread 2 holds/waits |
| 113 | |
| 114 | # Find the mutex owner |
| 115 | (gdb) p ((pthread_mutex_t*)0x601090)->__data.__owner # Linux glibc mutex |
| 116 | # prints TID of owning thread |
| 117 | |
| 118 | # Python script to dump all mutex owners (GDB 7+) |
| 119 | python |
| 120 | import gdb |
| 121 | for t in gdb.selected_inferior().threads(): |
| 122 | t.switch() |
| 123 | print(f"Thread {t.num}: {gdb.execute('bt 3', to_string=True)}") |
| 124 | end |
| 125 | ``` |
| 126 | |
| 127 | ### 4. std::atomic misuse patterns |
| 128 | |
| 129 | ```cpp |
| 130 | // WRONG: atomic variable, but non-atomic compound operation |
| 131 | std::atomic<int> counter{0}; |
| 132 | if (counter == 0) counter = 1; // not atomic together! TOCTOU race |
| 133 | |
| 134 | // CORRECT: use compare_exchange |
| 135 | int expected = 0; |
| 136 | counter.compare_exchange_strong(expected, 1); |
| 137 | |
| 138 | / |