numa-programming

Name: numa-programming
Author: mohitmishra786

61installs19forks

How to install numa-programming?

mohitmishra786/low-level-dev-skills/numa-programming

$npx -y skills add mohitmishra786/low-level-dev-skills --skill numa-programming

Installs into the current project.

Summary

NUMA programming skill for multi-socket memory locality. Use when detecting NUMA topology, binding processes with numactl, using libnuma API, building NUMA-aware data structures, or measuring remote access penalties. Activates on queries about numactl, libnuma, NUMA topology, mbi

Files · 1

View on GitHub

SKILL.md

1	# NUMA Programming
2
3	## Purpose
4
5	Guide agents through NUMA-aware programming: topology detection with `numactl` and sysfs, libnuma API (`numa_alloc_onnode`, `mbind`, `set_mempolicy`), process binding with `numactl`, NUMA-aware data structures, remote access diagnosis with `perf stat`, and `lstopo` visualization.
6
7	## When to Use
8
9	- Multi-socket server shows poor scaling despite low CPU utilization
10	- Memory bandwidth saturation on one NUMA node
11	- Binding database or cache process to local memory
12	- Designing per-node freelists or sharded allocators
13	- Measuring remote vs local memory access latency
14	- Tuning HPC, DPDK, or custom allocator for socket locality
15
16	## Workflow
17
18	### 1. Topology detection
19
20	```bash
21	# Hardware topology summary
22	numactl --hardware
23
24	# Detailed topology with distances
25	lstopo --of console
26
27	# sysfs nodes
28	ls /sys/devices/system/node/
29	cat /sys/devices/system/node/node0/meminfo
30	cat /sys/devices/system/node/node0/cpulist
31	```
32
33	Typical output:
34
35	```
36	available: 2 nodes (0-1)
37	node 0 cpus: 0-15
38	node 0 size: 65536 MB
39	node 1 cpus: 16-31
40	node 1 size: 65536 MB
41	node distances:
42	node 0 1
43	0: 10 21
44	1: 21 10
45	```
46
47	Distance 10 = local, higher = remote (cross-socket).
48
49	### 2. Process binding with numactl
50
51	```bash
52	# Bind to node 0 CPUs and memory
53	numactl --cpunodebind=0 --membind=0 ./myapp
54
55	# Interleave memory across all nodes
56	numactl --interleave=all ./myapp
57
58	# Preferred node (fallback if full)
59	numactl --preferred=0 ./myapp
60
61	# Show process NUMA policy
62	numactl --show
63	cat /proc/self/numa_maps
64	```
65
66	### 3. libnuma API
67
68	```c
69	#include <numa.h>
70	#include <numaif.h>
71	#include <stdio.h>
72
73	int main(void) {
74	if (numa_available() < 0) {
75	fprintf(stderr, "NUMA not available\n");
76	return 1;
77	}
78	int node = numa_node_of_cpu(0);
79	printf("CPU 0 on node %d\n", node);
80
81	// Allocate on specific node
82	size_t size = 1024 * 1024 * 1024;
83	void *mem = numa_alloc_onnode(size, 0);
84	if (!mem) return 1;
85
86	// Bind existing memory
87	unsigned long nodemask = 1UL << 0;
88	mbind(mem, size, MPOL_BIND, &nodemask, sizeof(nodemask) * 8, 0);
89
90	numa_free(mem, size);
91	return 0;
92	}
93	```
94
95	```bash
96	gcc -o numa_test numa_test.c -lnuma
97	```
98
99	\| API \| Purpose \|
100	\|-----\|---------\|
101	\| `numa_alloc_onnode` \| Allocate on specific node \|
102	\| `numa_alloc_local` \| Allocate on current CPU's node \|
103	\| `mbind` \| Set policy on existing mapping \|
104	\| `set_mempolicy` \| Default policy for subsequent allocs \|
105	\| `move_pages` \| Migrate pages to target node \|
106
107	### 4. NUMA-aware data structures
108
109	```c
110	// Per-node freelist — avoids cross-node synchronization
111	#define MAX_NODES 8
112	struct per_node_pool {
113	void *free_list[MAX_NODES];
114	int node_count;
115	};
116
117	void pool_alloc_numa(struct per_node_pool p) {
118	int node = numa_node_of_cpu(sched_getcpu());
119	void *blk = p->free_list[node];
120	if (blk) {
121	p->free_list[node] = (void *)blk;
122	return blk;
123	}
124	return numa_alloc_onnode(BLOCK_SIZE, node);
125	}
126	```
127
128	Pin threads to cores on the same node as their pool.
129
130	### 5. Thread affinity alignment
131
132	```bash
133	# Pin thread 0 to CPU 0 (node 0), allocate on node 0
134	numactl --cpunodebind=0 --membind=0 ./worker --id 0
135	numactl --cpunodebind=1 --membind=1 ./worker --id 1
136	```
137
138	```c
139	#include <pthread.h>
140	#include <sched.h>
141
142	cpu_set_t cpuset;
143	CPU_ZERO(&cpuset);
144	CPU_SET(target_cpu, &cpuset);
145	pthread_setaffinity_np(pthread_self(), sizeof(cpuset), &cpuset);
146	```
147
148	### 6. Remote access diagnosis
149
150	```bash
151	# Cache misses often spike with remote memory
152	perf stat -e cache-misses,cache-references,node-load-misses \
153	numactl --cpunodebind=0 --membind=1 ./myapp
154
155	# Compare local vs remote binding
156	perf stat numactl --cpunodebind=0 --membind=0 ./myapp
157	perf stat numactl --cpunodebind=0 --membind=1 ./myapp
158	```
159
160	```bash
161	# NUMA hit/miss stats (if available)
162	perf stat -e node-loads,node-load-misses,node-stores ./myapp
163	```
164
165	### 7. Measuring remote penalty
166
167	```c
168	// Microbenchmark: touch 1GB on local vs remote node
169	clock_t start = clock();
170	for (size_t i = 0; i < size; i += 4096)
171	sum += ((char *)mem)[i];
172	```
173
174	Expect 1.5–3x slowdown for remote access depending on interconnect (QPI/UPI/Infinity Fabric).
175
176	### 8. lstopo visualization
177
178	```bash
179	# Graphical (if X11)
180	lstopo
181
182	# Text with memory/PCI
183	lstopo --of ascii
184
185	# Export for documentation
186	lstopo file.png
187	```
188
189	Shows: NUMA nodes, cores, caches, PCI devices — essential for DPDK NIC placement.
190
191	### 9. Decision tree
192
193	```
194	Poor scaling on multi-socket?
195	├── Check numactl --hardware
196	├── Verify thread and memory on same node
197	├── perf stat node-load-misses
198	├── Remote misses high?
199	│ ├── numactl --membind=local
200	│ └── Per-node data partitioning
201	└── Still slow → memory

People who liked this also viewed

6 picks

Type

numa-programming

61installs19forks

View on GitHub

How to install numa-programming?

mohitmishra786/low-level-dev-skills/numa-programming

$npx -y skills add mohitmishra786/low-level-dev-skills --skill numa-programming

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# NUMA Programming
2
3	## Purpose
4
5	Guide agents through NUMA-aware programming: topology detection with `numactl` and sysfs, libnuma API (`numa_alloc_onnode`, `mbind`, `set_mempolicy`), process binding with `numactl`, NUMA-aware data structures, remote access diagnosis with `perf stat`, and `lstopo` visualization.
6
7	## When to Use
8
9	- Multi-socket server shows poor scaling despite low CPU utilization
10	- Memory bandwidth saturation on one NUMA node
11	- Binding database or cache process to local memory
12	- Designing per-node freelists or sharded allocators
13	- Measuring remote vs local memory access latency
14	- Tuning HPC, DPDK, or custom allocator for socket locality
15
16	## Workflow
17
18	### 1. Topology detection
19
20	```bash
21	# Hardware topology summary
22	numactl --hardware
23
24	# Detailed topology with distances
25	lstopo --of console
26
27	# sysfs nodes
28	ls /sys/devices/system/node/
29	cat /sys/devices/system/node/node0/meminfo
30	cat /sys/devices/system/node/node0/cpulist
31	```
32
33	Typical output:
34
35	```
36	available: 2 nodes (0-1)
37	node 0 cpus: 0-15
38	node 0 size: 65536 MB
39	node 1 cpus: 16-31
40	node 1 size: 65536 MB
41	node distances:
42	node 0 1
43	0: 10 21
44	1: 21 10
45	```
46
47	Distance 10 = local, higher = remote (cross-socket).
48
49	### 2. Process binding with numactl
50
51	```bash
52	# Bind to node 0 CPUs and memory
53	numactl --cpunodebind=0 --membind=0 ./myapp
54
55	# Interleave memory across all nodes
56	numactl --interleave=all ./myapp
57
58	# Preferred node (fallback if full)
59	numactl --preferred=0 ./myapp
60
61	# Show process NUMA policy
62	numactl --show
63	cat /proc/self/numa_maps
64	```
65
66	### 3. libnuma API
67
68	```c
69	#include <numa.h>
70	#include <numaif.h>
71	#include <stdio.h>
72
73	int main(void) {
74	if (numa_available() < 0) {
75	fprintf(stderr, "NUMA not available\n");
76	return 1;
77	}
78	int node = numa_node_of_cpu(0);
79	printf("CPU 0 on node %d\n", node);
80
81	// Allocate on specific node
82	size_t size = 1024 * 1024 * 1024;
83	void *mem = numa_alloc_onnode(size, 0);
84	if (!mem) return 1;
85
86	// Bind existing memory
87	unsigned long nodemask = 1UL << 0;
88	mbind(mem, size, MPOL_BIND, &nodemask, sizeof(nodemask) * 8, 0);
89
90	numa_free(mem, size);
91	return 0;
92	}
93	```
94
95	```bash
96	gcc -o numa_test numa_test.c -lnuma
97	```
98
99	\| API \| Purpose \|
100	\|-----\|---------\|
101	\| `numa_alloc_onnode` \| Allocate on specific node \|
102	\| `numa_alloc_local` \| Allocate on current CPU's node \|
103	\| `mbind` \| Set policy on existing mapping \|
104	\| `set_mempolicy` \| Default policy for subsequent allocs \|
105	\| `move_pages` \| Migrate pages to target node \|
106
107	### 4. NUMA-aware data structures
108
109	```c
110	// Per-node freelist — avoids cross-node synchronization
111	#define MAX_NODES 8
112	struct per_node_pool {
113	void *free_list[MAX_NODES];
114	int node_count;
115	};
116
117	void pool_alloc_numa(struct per_node_pool p) {
118	int node = numa_node_of_cpu(sched_getcpu());
119	void *blk = p->free_list[node];
120	if (blk) {
121	p->free_list[node] = (void *)blk;
122	return blk;
123	}
124	return numa_alloc_onnode(BLOCK_SIZE, node);
125	}
126	```
127
128	Pin threads to cores on the same node as their pool.
129
130	### 5. Thread affinity alignment
131
132	```bash
133	# Pin thread 0 to CPU 0 (node 0), allocate on node 0
134	numactl --cpunodebind=0 --membind=0 ./worker --id 0
135	numactl --cpunodebind=1 --membind=1 ./worker --id 1
136	```
137
138	```c
139	#include <pthread.h>
140	#include <sched.h>
141
142	cpu_set_t cpuset;
143	CPU_ZERO(&cpuset);
144	CPU_SET(target_cpu, &cpuset);
145	pthread_setaffinity_np(pthread_self(), sizeof(cpuset), &cpuset);
146	```
147
148	### 6. Remote access diagnosis
149
150	```bash
151	# Cache misses often spike with remote memory
152	perf stat -e cache-misses,cache-references,node-load-misses \
153	numactl --cpunodebind=0 --membind=1 ./myapp
154
155	# Compare local vs remote binding
156	perf stat numactl --cpunodebind=0 --membind=0 ./myapp
157	perf stat numactl --cpunodebind=0 --membind=1 ./myapp
158	```
159
160	```bash
161	# NUMA hit/miss stats (if available)
162	perf stat -e node-loads,node-load-misses,node-stores ./myapp
163	```
164
165	### 7. Measuring remote penalty
166
167	```c
168	// Microbenchmark: touch 1GB on local vs remote node
169	clock_t start = clock();
170	for (size_t i = 0; i < size; i += 4096)
171	sum += ((char *)mem)[i];
172	```
173
174	Expect 1.5–3x slowdown for remote access depending on interconnect (QPI/UPI/Infinity Fabric).
175
176	### 8. lstopo visualization
177
178	```bash
179	# Graphical (if X11)
180	lstopo
181
182	# Text with memory/PCI
183	lstopo --of ascii
184
185	# Export for documentation
186	lstopo file.png
187	```
188
189	Shows: NUMA nodes, cores, caches, PCI devices — essential for DPDK NIC placement.
190
191	### 9. Decision tree
192
193	```
194	Poor scaling on multi-socket?
195	├── Check numactl --hardware
196	├── Verify thread and memory on same node
197	├── perf stat node-load-misses
198	├── Remote misses high?
199	│ ├── numactl --membind=local
200	│ └── Per-node data partitioning
201	└── Still slow → memory

People who liked this also viewed

6 picks

Type