qlora

Name: qlora
Author: itsmostafa

18installs0forks

How to install qlora?

itsmostafa/llm-engineering-skills/qlora

$npx -y skills add itsmostafa/llm-engineering-skills --skill qlora

Installs into the current project.

Summary

Memory-efficient fine-tuning with 4-bit quantization and LoRA adapters. Use when fine-tuning large models (7B+) on consumer GPUs, when VRAM is limited, or when standard LoRA still exceeds memory. Builds on the lora skill.

Files · 1

View on GitHub

SKILL.md

1	# QLoRA: Quantized Low-Rank Adaptation
2
3	QLoRA enables fine-tuning of large language models on consumer GPUs by combining 4-bit quantization with LoRA adapters. A 65B model can be fine-tuned on a single 48GB GPU while matching 16-bit fine-tuning performance.
4
5	> Prerequisites: This skill assumes familiarity with LoRA. See the `lora` skill for LoRA fundamentals (LoraConfig, target_modules, training patterns).
6
7	## Table of Contents
8
9	- [Core Innovations](#core-innovations)
10	- [BitsAndBytesConfig Deep Dive](#bitsandbytesconfig-deep-dive)
11	- [Memory Requirements](#memory-requirements)
12	- [Complete Training Example](#complete-training-example)
13	- [Inference and Merging](#inference-and-merging)
14	- [Troubleshooting](#troubleshooting)
15	- [Best Practices](#best-practices)
16	- [References](#references)
17
18	## Core Innovations
19
20	QLoRA introduces three techniques that reduce memory usage without sacrificing performance:
21
22	### 4-bit NormalFloat (NF4)
23
24	NF4 is an information-theoretically optimal quantization data type for normally distributed weights. Neural network weights are typically normally distributed, making NF4 more efficient than standard 4-bit floats.
25
26	```
27	Storage: 4-bit NF4 (quantized weights)
28	Compute: 16-bit BF16 (dequantized for forward/backward pass)
29	```
30
31	The key insight: weights are stored in 4-bit but dequantized to bf16 for computation. Only the frozen base model is quantized; LoRA adapters remain in full precision.
32
33	NF4 vs FP4:
34
35	\| Quantization \| Description \| Use Case \|
36	\|--------------\|-------------\|----------\|
37	\| `nf4` \| Normalized Float 4-bit, optimal for normal distributions \| Default, recommended \|
38	\| `fp4` \| Standard 4-bit float \| Legacy, rarely needed \|
39
40	### Double Quantization
41
42	Standard quantization requires storing scaling constants (typically fp32) for each quantization block. Double quantization quantizes these constants too:
43
44	```
45	First quantization: weights → 4-bit + fp32 scaling constants
46	Double quantization: scaling constants → 8-bit + fp32 second-level constants
47	```
48
49	This saves approximately 0.37 bits per parameter—significant for billion-parameter models:
50	- 7B model: ~325 MB savings
51	- 70B model: ~3.2 GB savings
52
53	### Paged Optimizers
54
55	During training, gradient checkpointing can cause memory spikes when processing long sequences. Paged optimizers use NVIDIA unified memory to automatically transfer optimizer states between GPU and CPU:
56
57	```
58	Normal training: OOM on memory spike
59	Paged optimizers: GPU ↔ CPU transfer handles spikes gracefully
60	```
61
62	This is handled automatically by bitsandbytes when using 4-bit training.
63
64	## BitsAndBytesConfig Deep Dive
65
66	### All Parameters Explained
67
68	```python
69	from transformers import BitsAndBytesConfig
70	import torch
71
72	bnb_config = BitsAndBytesConfig(
73	# Core 4-bit settings
74	load_in_4bit=True, # Enable 4-bit quantization
75	bnb_4bit_quant_type="nf4", # "nf4" (recommended) or "fp4"
76
77	# Double quantization
78	bnb_4bit_use_double_quant=True, # Quantize the quantization constants
79
80	# Compute precision
81	bnb_4bit_compute_dtype=torch.bfloat16, # Dequantize to this dtype for compute
82
83	# Optional: specific storage type (usually auto-detected)
84	bnb_4bit_quant_storage=torch.uint8, # Storage dtype for quantized weights
85	)
86	```
87
88	### Compute Dtype Selection
89
90	\| Dtype \| Hardware \| Notes \|
91	\|-------\|----------\|-------\|
92	\| `torch.bfloat16` \| Ampere+ (RTX 30xx, A100) \| Recommended, faster \|
93	\| `torch.float16` \| Older GPUs (V100, RTX 20xx) \| Use if bf16 not supported \|
94	\| `torch.float32` \| Any \| Slower, only for debugging \|
95
96	Check bf16 support:
97	```python
98	import torch
99	print(torch.cuda.is_bf16_supported()) # True on Ampere+
100	```
101
102	### Comparison: Quantization Options
103
104	```python
105	# Recommended: NF4 + double quant + bf16
106	optimal_config = BitsAndBytesConfig(
107	load_in_4bit=True,
108	bnb_4bit_quant_type="nf4",
109	bnb_4bit_use_double_quant=True,
110	bnb_4bit_compute_dtype=torch.bfloat16,
111	)
112
113	# Fallback when bf16 is unsupported
114	fp16_config = BitsAndBytesConfig(
115	load_in_4bit=True,
116	bnb_4bit_quant_type="nf4",
117	bnb_4bit_use_double_quant=True,
118	bnb_4bit_compute_dtype=torch.float16, # use when bf16 is unsupported or slower
119	)
120
121	# 8-bit alternative (less compression, sometimes more stable)
122	eight_bit_config = BitsAndBytesConfig(
123	load_in_8bit=True,
124	)
125	```
126
127	## Memory Requirements
128
129	\| Model Size \| Full Fine-tuning \| LoRA (16-bit) \| QLoRA (4-bit) \|
130	\|------------\|------------------\|---------------\|---------------\|
131	\| 7B \| ~60 GB \| ~16 GB \| ~6 GB \|
132	\| 13B \| ~104 GB \| ~28 GB \| ~10 GB \|
133	\| 34B \| ~272 GB \| ~75 GB \| ~20 GB \|
134	\| 70B \| ~560 GB \| ~160 GB \| ~48 GB \|
135
136	Notes:
137	- QLoRA memory inclu

People who liked this also viewed

6 picks

Type

qlora

18installs0forks

View on GitHub

How to install qlora?

itsmostafa/llm-engineering-skills/qlora

$npx -y skills add itsmostafa/llm-engineering-skills --skill qlora

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# QLoRA: Quantized Low-Rank Adaptation
2
3	QLoRA enables fine-tuning of large language models on consumer GPUs by combining 4-bit quantization with LoRA adapters. A 65B model can be fine-tuned on a single 48GB GPU while matching 16-bit fine-tuning performance.
4
5	> Prerequisites: This skill assumes familiarity with LoRA. See the `lora` skill for LoRA fundamentals (LoraConfig, target_modules, training patterns).
6
7	## Table of Contents
8
9	- [Core Innovations](#core-innovations)
10	- [BitsAndBytesConfig Deep Dive](#bitsandbytesconfig-deep-dive)
11	- [Memory Requirements](#memory-requirements)
12	- [Complete Training Example](#complete-training-example)
13	- [Inference and Merging](#inference-and-merging)
14	- [Troubleshooting](#troubleshooting)
15	- [Best Practices](#best-practices)
16	- [References](#references)
17
18	## Core Innovations
19
20	QLoRA introduces three techniques that reduce memory usage without sacrificing performance:
21
22	### 4-bit NormalFloat (NF4)
23
24	NF4 is an information-theoretically optimal quantization data type for normally distributed weights. Neural network weights are typically normally distributed, making NF4 more efficient than standard 4-bit floats.
25
26	```
27	Storage: 4-bit NF4 (quantized weights)
28	Compute: 16-bit BF16 (dequantized for forward/backward pass)
29	```
30
31	The key insight: weights are stored in 4-bit but dequantized to bf16 for computation. Only the frozen base model is quantized; LoRA adapters remain in full precision.
32
33	NF4 vs FP4:
34
35	\| Quantization \| Description \| Use Case \|
36	\|--------------\|-------------\|----------\|
37	\| `nf4` \| Normalized Float 4-bit, optimal for normal distributions \| Default, recommended \|
38	\| `fp4` \| Standard 4-bit float \| Legacy, rarely needed \|
39
40	### Double Quantization
41
42	Standard quantization requires storing scaling constants (typically fp32) for each quantization block. Double quantization quantizes these constants too:
43
44	```
45	First quantization: weights → 4-bit + fp32 scaling constants
46	Double quantization: scaling constants → 8-bit + fp32 second-level constants
47	```
48
49	This saves approximately 0.37 bits per parameter—significant for billion-parameter models:
50	- 7B model: ~325 MB savings
51	- 70B model: ~3.2 GB savings
52
53	### Paged Optimizers
54
55	During training, gradient checkpointing can cause memory spikes when processing long sequences. Paged optimizers use NVIDIA unified memory to automatically transfer optimizer states between GPU and CPU:
56
57	```
58	Normal training: OOM on memory spike
59	Paged optimizers: GPU ↔ CPU transfer handles spikes gracefully
60	```
61
62	This is handled automatically by bitsandbytes when using 4-bit training.
63
64	## BitsAndBytesConfig Deep Dive
65
66	### All Parameters Explained
67
68	```python
69	from transformers import BitsAndBytesConfig
70	import torch
71
72	bnb_config = BitsAndBytesConfig(
73	# Core 4-bit settings
74	load_in_4bit=True, # Enable 4-bit quantization
75	bnb_4bit_quant_type="nf4", # "nf4" (recommended) or "fp4"
76
77	# Double quantization
78	bnb_4bit_use_double_quant=True, # Quantize the quantization constants
79
80	# Compute precision
81	bnb_4bit_compute_dtype=torch.bfloat16, # Dequantize to this dtype for compute
82
83	# Optional: specific storage type (usually auto-detected)
84	bnb_4bit_quant_storage=torch.uint8, # Storage dtype for quantized weights
85	)
86	```
87
88	### Compute Dtype Selection
89
90	\| Dtype \| Hardware \| Notes \|
91	\|-------\|----------\|-------\|
92	\| `torch.bfloat16` \| Ampere+ (RTX 30xx, A100) \| Recommended, faster \|
93	\| `torch.float16` \| Older GPUs (V100, RTX 20xx) \| Use if bf16 not supported \|
94	\| `torch.float32` \| Any \| Slower, only for debugging \|
95
96	Check bf16 support:
97	```python
98	import torch
99	print(torch.cuda.is_bf16_supported()) # True on Ampere+
100	```
101
102	### Comparison: Quantization Options
103
104	```python
105	# Recommended: NF4 + double quant + bf16
106	optimal_config = BitsAndBytesConfig(
107	load_in_4bit=True,
108	bnb_4bit_quant_type="nf4",
109	bnb_4bit_use_double_quant=True,
110	bnb_4bit_compute_dtype=torch.bfloat16,
111	)
112
113	# Fallback when bf16 is unsupported
114	fp16_config = BitsAndBytesConfig(
115	load_in_4bit=True,
116	bnb_4bit_quant_type="nf4",
117	bnb_4bit_use_double_quant=True,
118	bnb_4bit_compute_dtype=torch.float16, # use when bf16 is unsupported or slower
119	)
120
121	# 8-bit alternative (less compression, sometimes more stable)
122	eight_bit_config = BitsAndBytesConfig(
123	load_in_8bit=True,
124	)
125	```
126
127	## Memory Requirements
128
129	\| Model Size \| Full Fine-tuning \| LoRA (16-bit) \| QLoRA (4-bit) \|
130	\|------------\|------------------\|---------------\|---------------\|
131	\| 7B \| ~60 GB \| ~16 GB \| ~6 GB \|
132	\| 13B \| ~104 GB \| ~28 GB \| ~10 GB \|
133	\| 34B \| ~272 GB \| ~75 GB \| ~20 GB \|
134	\| 70B \| ~560 GB \| ~160 GB \| ~48 GB \|
135
136	Notes:
137	- QLoRA memory inclu

People who liked this also viewed

6 picks

Type