aoti-debug

Name: aoti-debug
Author: pytorch

2.0kinstalls28kforks

How to install aoti-debug?

pytorch/pytorch/aoti-debug

$npx -y skills add pytorch/pytorch --skill aoti-debug

Installs into the current project.

Summary

Debug AOTInductor (AOTI) errors and crashes. Use when encountering AOTI segfaults, device mismatch errors, constant loading failures, or runtime errors from aot_compile, aot_load, aoti_compile_and_package, or aoti_load_package.

Files · 1

View on GitHub

SKILL.md

1	# AOTI Debugging Guide
2
3	This skill helps diagnose and fix common AOTInductor issues.
4
5	## Error Pattern Routing
6
7	Check the error message and route to the appropriate sub-guide:
8
9	### Triton Index Out of Bounds
10	If the error matches this pattern:
11	```
12	Assertion `index out of bounds: 0 <= tmpN < ksM` failed
13	```
14	→ Follow the guide in `triton-index-out-of-bounds.md`
15
16	### All Other Errors
17	Continue with the sections below.
18
19	---
20
21	## First Step: Always Check Device and Shape Matching
22
23	For ANY AOTI error (segfault, exception, crash, wrong output), ALWAYS check these first:
24
25	1. Compile device == Load device: The model must be loaded on the same device type it was compiled on
26	2. Input devices match: Runtime inputs must be on the same device as the compiled model
27	3. Input shapes match: Runtime input shapes must match the shapes used during compilation (or satisfy dynamic shape constraints)
28
29	```python
30	# During compilation - note the device and shapes
31	model = MyModel().eval() # What device? CPU or .cuda()?
32	inp = torch.randn(2, 10) # What device? What shape?
33	compiled_so = torch._inductor.aot_compile(model, (inp,))
34
35	# During loading - device type MUST match compilation
36	loaded = torch._export.aot_load(compiled_so, "???") # Must match model/input device above
37
38	# During inference - device and shapes MUST match
39	out = loaded(inp.to("???")) # Must match compile device, shape must match
40	```
41
42	If any of these don't match, you will get errors ranging from segfaults to exceptions to wrong outputs.
43
44	## Key Constraint: Device Type Matching
45
46	AOTI requires compile and load to use the same device type.
47
48	- If you compile on CUDA, you must load on CUDA (device index can differ)
49	- If you compile on CPU, you must load on CPU
50	- Cross-device loading (e.g., compile on GPU, load on CPU) is NOT supported
51
52	## Common Error Patterns
53
54	### 1. Device Mismatch Segfault
55
56	Symptom: Segfault, exception, or crash during `aot_load()` or model execution.
57
58	Example error messages:
59	- `The specified pointer resides on host memory and is not registered with any CUDA device`
60	- Crash during constant loading in AOTInductorModelBase
61	- `Expected out tensor to have device cuda:0, but got cpu instead`
62
63	Cause: Compile and load device types don't match (see "First Step" above).
64
65	Solution: Ensure compile and load use the same device type. If compiled on CPU, load on CPU. If compiled on CUDA, load on CUDA.
66
67	### 2. Input Device Mismatch at Runtime
68
69	Symptom: RuntimeError during model execution.
70
71	Cause: Input device doesn't match compile device (see "First Step" above).
72
73	Better Debugging: Run with `AOTI_RUNTIME_CHECK_INPUTS=1` for clearer errors. This flag validates all input properties including device type, dtype, sizes, and strides:
74	```bash
75	AOTI_RUNTIME_CHECK_INPUTS=1 python your_script.py
76	```
77
78	This produces actionable error messages like:
79	```
80	Error: input_handles[0]: unmatched device type, expected: 0(cpu), but got: 1(cuda)
81	```
82
83
84	## Debugging CUDA Illegal Memory Access (IMA) Errors
85
86	If you encounter CUDA illegal memory access errors, follow this systematic approach:
87
88	### Step 1: Sanity Checks
89
90	Before diving deep, try these debugging flags:
91
92	```bash
93	AOTI_RUNTIME_CHECK_INPUTS=1
94	TORCHINDUCTOR_NAN_ASSERTS=1
95	```
96
97	These flags take effect at compilation time (at codegen time):
98
99	- `AOTI_RUNTIME_CHECK_INPUTS=1` checks if inputs satisfy the same guards used during compilation
100	- `TORCHINDUCTOR_NAN_ASSERTS=1` adds codegen before and after each kernel to check for NaN
101
102	### Step 2: Pinpoint the CUDA IMA
103
104	CUDA IMA errors can be non-deterministic. Use these flags to trigger the error deterministically:
105
106	```bash
107	PYTORCH_NO_CUDA_MEMORY_CACHING=1
108	CUDA_LAUNCH_BLOCKING=1
109	```
110
111	These flags take effect at runtime:
112
113	- `PYTORCH_NO_CUDA_MEMORY_CACHING=1` disables PyTorch's Caching Allocator, which allocates bigger buffers than needed immediately. This is usually why CUDA IMA errors are non-deterministic.
114	- `CUDA_LAUNCH_BLOCKING=1` forces kernels to launch one at a time. Without this, you get "CUDA kernel errors might be asynchronously reported" warnings since kernels launch asynchronously.
115
116	### Step 3: Identify Problematic Kernels with Intermediate Value Debugger
117
118	Use the AOTI Intermediate Value Debugger to pinpoint the problematic kernel:
119
120	```bash
121	AOT_INDUCTOR_DEBUG_INTERMEDIATE_VALUE_PRINTER=3
122	```
123
124	This prints kernels one by one at runtime. Together with previous flags, this shows which kernel was launched right before the error.
125
126	To inspect inputs to a specific kernel:
127
128	```bash
129	AOT_INDUCTOR_FILTERED_KERNELS_TO_PRINT="triton_poi_fused_add_ge_logical_and_logical_or_lt_231,_add_position_embeddings_kernel_5" AOT_INDUCTOR_DEBUG_INTERMEDIATE_VALUE_PRINTER=2
130	```
131
132	If i

People who liked this also viewed

6 picks

Type

aoti-debug

2.0kinstalls28kforks

View on GitHub

How to install aoti-debug?

pytorch/pytorch/aoti-debug

$npx -y skills add pytorch/pytorch --skill aoti-debug

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# AOTI Debugging Guide
2
3	This skill helps diagnose and fix common AOTInductor issues.
4
5	## Error Pattern Routing
6
7	Check the error message and route to the appropriate sub-guide:
8
9	### Triton Index Out of Bounds
10	If the error matches this pattern:
11	```
12	Assertion `index out of bounds: 0 <= tmpN < ksM` failed
13	```
14	→ Follow the guide in `triton-index-out-of-bounds.md`
15
16	### All Other Errors
17	Continue with the sections below.
18
19	---
20
21	## First Step: Always Check Device and Shape Matching
22
23	For ANY AOTI error (segfault, exception, crash, wrong output), ALWAYS check these first:
24
25	1. Compile device == Load device: The model must be loaded on the same device type it was compiled on
26	2. Input devices match: Runtime inputs must be on the same device as the compiled model
27	3. Input shapes match: Runtime input shapes must match the shapes used during compilation (or satisfy dynamic shape constraints)
28
29	```python
30	# During compilation - note the device and shapes
31	model = MyModel().eval() # What device? CPU or .cuda()?
32	inp = torch.randn(2, 10) # What device? What shape?
33	compiled_so = torch._inductor.aot_compile(model, (inp,))
34
35	# During loading - device type MUST match compilation
36	loaded = torch._export.aot_load(compiled_so, "???") # Must match model/input device above
37
38	# During inference - device and shapes MUST match
39	out = loaded(inp.to("???")) # Must match compile device, shape must match
40	```
41
42	If any of these don't match, you will get errors ranging from segfaults to exceptions to wrong outputs.
43
44	## Key Constraint: Device Type Matching
45
46	AOTI requires compile and load to use the same device type.
47
48	- If you compile on CUDA, you must load on CUDA (device index can differ)
49	- If you compile on CPU, you must load on CPU
50	- Cross-device loading (e.g., compile on GPU, load on CPU) is NOT supported
51
52	## Common Error Patterns
53
54	### 1. Device Mismatch Segfault
55
56	Symptom: Segfault, exception, or crash during `aot_load()` or model execution.
57
58	Example error messages:
59	- `The specified pointer resides on host memory and is not registered with any CUDA device`
60	- Crash during constant loading in AOTInductorModelBase
61	- `Expected out tensor to have device cuda:0, but got cpu instead`
62
63	Cause: Compile and load device types don't match (see "First Step" above).
64
65	Solution: Ensure compile and load use the same device type. If compiled on CPU, load on CPU. If compiled on CUDA, load on CUDA.
66
67	### 2. Input Device Mismatch at Runtime
68
69	Symptom: RuntimeError during model execution.
70
71	Cause: Input device doesn't match compile device (see "First Step" above).
72
73	Better Debugging: Run with `AOTI_RUNTIME_CHECK_INPUTS=1` for clearer errors. This flag validates all input properties including device type, dtype, sizes, and strides:
74	```bash
75	AOTI_RUNTIME_CHECK_INPUTS=1 python your_script.py
76	```
77
78	This produces actionable error messages like:
79	```
80	Error: input_handles[0]: unmatched device type, expected: 0(cpu), but got: 1(cuda)
81	```
82
83
84	## Debugging CUDA Illegal Memory Access (IMA) Errors
85
86	If you encounter CUDA illegal memory access errors, follow this systematic approach:
87
88	### Step 1: Sanity Checks
89
90	Before diving deep, try these debugging flags:
91
92	```bash
93	AOTI_RUNTIME_CHECK_INPUTS=1
94	TORCHINDUCTOR_NAN_ASSERTS=1
95	```
96
97	These flags take effect at compilation time (at codegen time):
98
99	- `AOTI_RUNTIME_CHECK_INPUTS=1` checks if inputs satisfy the same guards used during compilation
100	- `TORCHINDUCTOR_NAN_ASSERTS=1` adds codegen before and after each kernel to check for NaN
101
102	### Step 2: Pinpoint the CUDA IMA
103
104	CUDA IMA errors can be non-deterministic. Use these flags to trigger the error deterministically:
105
106	```bash
107	PYTORCH_NO_CUDA_MEMORY_CACHING=1
108	CUDA_LAUNCH_BLOCKING=1
109	```
110
111	These flags take effect at runtime:
112
113	- `PYTORCH_NO_CUDA_MEMORY_CACHING=1` disables PyTorch's Caching Allocator, which allocates bigger buffers than needed immediately. This is usually why CUDA IMA errors are non-deterministic.
114	- `CUDA_LAUNCH_BLOCKING=1` forces kernels to launch one at a time. Without this, you get "CUDA kernel errors might be asynchronously reported" warnings since kernels launch asynchronously.
115
116	### Step 3: Identify Problematic Kernels with Intermediate Value Debugger
117
118	Use the AOTI Intermediate Value Debugger to pinpoint the problematic kernel:
119
120	```bash
121	AOT_INDUCTOR_DEBUG_INTERMEDIATE_VALUE_PRINTER=3
122	```
123
124	This prints kernels one by one at runtime. Together with previous flags, this shows which kernel was launched right before the error.
125
126	To inspect inputs to a specific kernel:
127
128	```bash
129	AOT_INDUCTOR_FILTERED_KERNELS_TO_PRINT="triton_poi_fused_add_ge_logical_and_logical_or_lt_231,_add_position_embeddings_kernel_5" AOT_INDUCTOR_DEBUG_INTERMEDIATE_VALUE_PRINTER=2
130	```
131
132	If i

People who liked this also viewed

6 picks

Type