flash-attention

Name: flash-attention
Author: AlexAI-MCP

132stars24forks

How to install flash-attention?

alexai-mcp/hermes-ccc/flash-attention

$npx -y skills add AlexAI-MCP/hermes-CCC --skill flash-attention

Installs into the current project.

Summary

Optimize transformer attention with Flash Attention — 2-4x speedup, 10-20x memory reduction for long sequences on CUDA GPUs.

Files · 1

View on GitHub

SKILL.md

1	# Flash Attention — Fast Memory-Efficient Attention
2
3	Flash Attention provides 2-4x training speedup and 10-20x memory reduction by replacing the standard O(N²) attention with an IO-aware tiling algorithm.
4
5	## When to Use
6
7	- Sequences longer than 512 tokens → always use Flash Attention
8	- GPU OOM during training → Flash Attention is usually the fix
9	- Maximizing training throughput → 2-4x faster than standard attention
10	- H100/A100 with FP8/BF16 → critical for efficiency
11
12	---
13
14	## Option 1: PyTorch Native SDPA (Easiest — PyTorch 2.2+)
15
16	```python
17	import torch
18	import torch.nn.functional as F
19
20	# PyTorch automatically uses Flash Attention if available
21	q = torch.randn(2, 8, 512, 64, device="cuda", dtype=torch.float16)
22	k = torch.randn(2, 8, 512, 64, device="cuda", dtype=torch.float16)
23	v = torch.randn(2, 8, 512, 64, device="cuda", dtype=torch.float16)
24
25	# This automatically dispatches to Flash Attention on compatible hardware
26	output = F.scaled_dot_product_attention(q, k, v, dropout_p=0.0, is_causal=True)
27
28	# Check which kernel is being used
29	with torch.backends.cuda.sdp_kernel(
30	enable_flash=True, enable_math=False, enable_mem_efficient=False
31	):
32	output = F.scaled_dot_product_attention(q, k, v, is_causal=True)
33	```
34
35	---
36
37	## Option 2: flash-attn Library (Maximum Performance)
38
39	```bash
40	# Requires: CUDA toolkit, torch, ninja
41	pip install flash-attn --no-build-isolation
42
43	# If build fails:
44	pip install packaging ninja
45	pip install flash-attn --no-build-isolation --no-cache-dir
46	```
47
48	```python
49	from flash_attn import flash_attn_func, flash_attn_varlen_func
50
51	# Basic usage: q,k,v shape: (batch, seqlen, nheads, headdim)
52	q = torch.randn(2, 512, 8, 64, device="cuda", dtype=torch.float16)
53	k = torch.randn(2, 512, 8, 64, device="cuda", dtype=torch.float16)
54	v = torch.randn(2, 512, 8, 64, device="cuda", dtype=torch.float16)
55
56	output = flash_attn_func(q, k, v, dropout_p=0.0, causal=True)
57	```
58
59	---
60
61	## Option 3: Enable in HuggingFace Transformers
62
63	```python
64	from transformers import AutoModelForCausalLM
65
66	# Automatic Flash Attention 2 (recommended)
67	model = AutoModelForCausalLM.from_pretrained(
68	"meta-llama/Llama-3.1-8B-Instruct",
69	attn_implementation="flash_attention_2",
70	torch_dtype=torch.bfloat16,
71	device_map="auto",
72	)
73
74	# Or: eager (standard), sdpa (PyTorch SDPA)
75	model = AutoModelForCausalLM.from_pretrained(
76	"...",
77	attn_implementation="sdpa", # no install needed
78	)
79	```
80
81	---
82
83	## Hardware Requirements
84
85	\| GPU \| FP16 \| BF16 \| FP8 \|
86	\|-----\|------\|------\|-----\|
87	\| A100 \| ✅ \| ✅ \| ❌ \|
88	\| H100 \| ✅ \| ✅ \| ✅ \|
89	\| A10/A30 \| ✅ \| ✅ \| ❌ \|
90	\| RTX 3090/4090 \| ✅ \| ✅ \| ❌ \|
91	\| V100 \| ✅ \| ❌ \| ❌ \|
92	\| T4 \| ✅ \| ❌ \| ❌ \|
93
94	Requirements:
95	- CUDA 11.6+
96	- PyTorch 2.0+
97	- Ampere GPU or newer (RTX 30 series, A100, H100) for best performance
98
99	---
100
101	## Sliding Window Attention (Long Context)
102
103	```python
104	from flash_attn.flash_attn_interface import flash_attn_varlen_func
105
106	# For sequences longer than GPU memory allows
107	# Use sliding window to limit attention span
108	output = flash_attn_func(
109	q, k, v,
110	causal=True,
111	window_size=(512, 0) # attend to last 512 tokens only
112	)
113	```
114
115	---
116
117	## Memory Savings
118
119	Standard attention: O(N²) memory
120	Flash Attention: O(N) memory (recomputes on backward pass)
121
122	```
123	Sequence length 4096: ~6x memory savings
124	Sequence length 8192: ~15x memory savings
125	Sequence length 32768: ~50x+ memory savings
126	```
127
128	---
129
130	## Benchmarking
131
132	```python
133	import time, torch
134
135	def bench(fn, warmup=3, reps=10):
136	for _ in range(warmup):
137	fn()
138	torch.cuda.synchronize()
139	t0 = time.time()
140	for _ in range(reps):
141	fn()
142	torch.cuda.synchronize()
143	return (time.time() - t0) / reps * 1000 # ms
144
145	q = torch.randn(4, 2048, 16, 64, device="cuda", dtype=torch.float16)
146	k, v = q.clone(), q.clone()
147
148	standard_ms = bench(lambda: F.scaled_dot_product_attention(q, k, v))
149	flash_ms = bench(lambda: flash_attn_func(q, k, v, causal=True))
150	print(f"Standard: {standard_ms:.1f}ms \| Flash: {flash_ms:.1f}ms \| Speedup: {standard_ms/flash_ms:.1f}x")
151	```
152
153	---
154
155	## Common Issues
156
157	Build fails: Ensure CUDA toolkit matches PyTorch CUDA version (`python -c "import torch; print(torch.version.cuda)"`)
158
159	Wrong dtype: Flash Attention requires FP16 or BF16, not FP32
160
161	OOM despite Flash Attention: Use gradient checkpointing additionally: `model.gradient_checkpointing_enable()`
162
163	Not faster on short sequences: Flash Attention shines at >512 tokens; below that overhead can dominate

People who liked this also viewed

6 picks

Type

azure-aiUse for Azure AI: Search, Speech, OpenAI, Document Intelligence. Helps with search, vector/hybrid search, speech-to-text, text-to-speech, transcription, OCR. WHEN: AI Search, query search, vector searSkillsJul 2026472k1.3k
just-scrapeSearch, scrape, crawl, extract structured data, and monitor web pages via the ScrapeGraph AI CLI. Use when the user asks to search the web, scrape a webpage, grab content from a URL, extract JSON fromSkillsJul 2026245k37
firecrawlCLI and Agent Skill for Firecrawl - Add scrape, search, and browsing capabilities to your AI agentsSkillsJul 202694k531
firecrawl-searchWeb search with full page content extraction. Use this skill whenever the user asks to search the web, find articles, research a topic, look something up, find recent news, discover sources, or says "SkillsJul 202676k531
firecrawl-agentAI-powered autonomous data extraction that navigates complex sites and returns structured JSON. Use this skill when the user wants structured data from websites, needs to extract pricing tiers, producSkillsJul 202674k531
firecrawl-mapDiscover and list all URLs on a website, with optional search filtering. Use this skill when the user wants to find a specific page on a large site, list all URLs, see the site structure, find where sSkillsJul 202674k531

flash-attention

132stars24forks

View on GitHub

How to install flash-attention?

alexai-mcp/hermes-ccc/flash-attention

$npx -y skills add AlexAI-MCP/hermes-CCC --skill flash-attention

Installs into the current project.

Summary

Optimize transformer attention with Flash Attention — 2-4x speedup, 10-20x memory reduction for long sequences on CUDA GPUs.

Files · 1

View on GitHub

SKILL.md

1	# Flash Attention — Fast Memory-Efficient Attention
2
3	Flash Attention provides 2-4x training speedup and 10-20x memory reduction by replacing the standard O(N²) attention with an IO-aware tiling algorithm.
4
5	## When to Use
6
7	- Sequences longer than 512 tokens → always use Flash Attention
8	- GPU OOM during training → Flash Attention is usually the fix
9	- Maximizing training throughput → 2-4x faster than standard attention
10	- H100/A100 with FP8/BF16 → critical for efficiency
11
12	---
13
14	## Option 1: PyTorch Native SDPA (Easiest — PyTorch 2.2+)
15
16	```python
17	import torch
18	import torch.nn.functional as F
19
20	# PyTorch automatically uses Flash Attention if available
21	q = torch.randn(2, 8, 512, 64, device="cuda", dtype=torch.float16)
22	k = torch.randn(2, 8, 512, 64, device="cuda", dtype=torch.float16)
23	v = torch.randn(2, 8, 512, 64, device="cuda", dtype=torch.float16)
24
25	# This automatically dispatches to Flash Attention on compatible hardware
26	output = F.scaled_dot_product_attention(q, k, v, dropout_p=0.0, is_causal=True)
27
28	# Check which kernel is being used
29	with torch.backends.cuda.sdp_kernel(
30	enable_flash=True, enable_math=False, enable_mem_efficient=False
31	):
32	output = F.scaled_dot_product_attention(q, k, v, is_causal=True)
33	```
34
35	---
36
37	## Option 2: flash-attn Library (Maximum Performance)
38
39	```bash
40	# Requires: CUDA toolkit, torch, ninja
41	pip install flash-attn --no-build-isolation
42
43	# If build fails:
44	pip install packaging ninja
45	pip install flash-attn --no-build-isolation --no-cache-dir
46	```
47
48	```python
49	from flash_attn import flash_attn_func, flash_attn_varlen_func
50
51	# Basic usage: q,k,v shape: (batch, seqlen, nheads, headdim)
52	q = torch.randn(2, 512, 8, 64, device="cuda", dtype=torch.float16)
53	k = torch.randn(2, 512, 8, 64, device="cuda", dtype=torch.float16)
54	v = torch.randn(2, 512, 8, 64, device="cuda", dtype=torch.float16)
55
56	output = flash_attn_func(q, k, v, dropout_p=0.0, causal=True)
57	```
58
59	---
60
61	## Option 3: Enable in HuggingFace Transformers
62
63	```python
64	from transformers import AutoModelForCausalLM
65
66	# Automatic Flash Attention 2 (recommended)
67	model = AutoModelForCausalLM.from_pretrained(
68	"meta-llama/Llama-3.1-8B-Instruct",
69	attn_implementation="flash_attention_2",
70	torch_dtype=torch.bfloat16,
71	device_map="auto",
72	)
73
74	# Or: eager (standard), sdpa (PyTorch SDPA)
75	model = AutoModelForCausalLM.from_pretrained(
76	"...",
77	attn_implementation="sdpa", # no install needed
78	)
79	```
80
81	---
82
83	## Hardware Requirements
84
85	\| GPU \| FP16 \| BF16 \| FP8 \|
86	\|-----\|------\|------\|-----\|
87	\| A100 \| ✅ \| ✅ \| ❌ \|
88	\| H100 \| ✅ \| ✅ \| ✅ \|
89	\| A10/A30 \| ✅ \| ✅ \| ❌ \|
90	\| RTX 3090/4090 \| ✅ \| ✅ \| ❌ \|
91	\| V100 \| ✅ \| ❌ \| ❌ \|
92	\| T4 \| ✅ \| ❌ \| ❌ \|
93
94	Requirements:
95	- CUDA 11.6+
96	- PyTorch 2.0+
97	- Ampere GPU or newer (RTX 30 series, A100, H100) for best performance
98
99	---
100
101	## Sliding Window Attention (Long Context)
102
103	```python
104	from flash_attn.flash_attn_interface import flash_attn_varlen_func
105
106	# For sequences longer than GPU memory allows
107	# Use sliding window to limit attention span
108	output = flash_attn_func(
109	q, k, v,
110	causal=True,
111	window_size=(512, 0) # attend to last 512 tokens only
112	)
113	```
114
115	---
116
117	## Memory Savings
118
119	Standard attention: O(N²) memory
120	Flash Attention: O(N) memory (recomputes on backward pass)
121
122	```
123	Sequence length 4096: ~6x memory savings
124	Sequence length 8192: ~15x memory savings
125	Sequence length 32768: ~50x+ memory savings
126	```
127
128	---
129
130	## Benchmarking
131
132	```python
133	import time, torch
134
135	def bench(fn, warmup=3, reps=10):
136	for _ in range(warmup):
137	fn()
138	torch.cuda.synchronize()
139	t0 = time.time()
140	for _ in range(reps):
141	fn()
142	torch.cuda.synchronize()
143	return (time.time() - t0) / reps * 1000 # ms
144
145	q = torch.randn(4, 2048, 16, 64, device="cuda", dtype=torch.float16)
146	k, v = q.clone(), q.clone()
147
148	standard_ms = bench(lambda: F.scaled_dot_product_attention(q, k, v))
149	flash_ms = bench(lambda: flash_attn_func(q, k, v, causal=True))
150	print(f"Standard: {standard_ms:.1f}ms \| Flash: {flash_ms:.1f}ms \| Speedup: {standard_ms/flash_ms:.1f}x")
151	```
152
153	---
154
155	## Common Issues
156
157	Build fails: Ensure CUDA toolkit matches PyTorch CUDA version (`python -c "import torch; print(torch.version.cuda)"`)
158
159	Wrong dtype: Flash Attention requires FP16 or BF16, not FP32
160
161	OOM despite Flash Attention: Use gradient checkpointing additionally: `model.gradient_checkpointing_enable()`
162
163	Not faster on short sequences: Flash Attention shines at >512 tokens; below that overhead can dominate

People who liked this also viewed

6 picks

Type