lora

Name: lora
Author: itsmostafa

23installs0forks

How to install lora?

itsmostafa/llm-engineering-skills/lora

$npx -y skills add itsmostafa/llm-engineering-skills --skill lora

Installs into the current project.

Summary

Parameter-efficient fine-tuning with Low-Rank Adaptation (LoRA). Use when fine-tuning large language models with limited GPU memory, creating task-specific adapters, or when you need to train multiple specialized models from a single base.

Files · 1

View on GitHub

SKILL.md

1	# Using LoRA for Fine-tuning
2
3	LoRA (Low-Rank Adaptation) enables efficient fine-tuning by freezing pretrained weights and injecting small trainable matrices into transformer layers. This reduces trainable parameters to ~0.1% of the original model while maintaining performance.
4
5	## Table of Contents
6
7	- [Core Concepts](#core-concepts)
8	- [Basic Setup](#basic-setup)
9	- [Configuration Parameters](#configuration-parameters)
10	- [QLoRA (Quantized LoRA)](#qlora-quantized-lora)
11	- [Training Patterns](#training-patterns)
12	- [Saving and Loading](#saving-and-loading)
13	- [Merging Adapters](#merging-adapters)
14	- [Best Practices](#best-practices)
15	- [References](#references)
16
17	## Core Concepts
18
19	### How LoRA Works
20
21	Instead of updating all weights during fine-tuning, LoRA decomposes weight updates into low-rank matrices:
22
23	```
24	W' = W + BA
25	```
26
27	Where:
28	- `W` is the frozen pretrained weight matrix (d × k)
29	- `B` is a trainable matrix (d × r)
30	- `A` is a trainable matrix (r × k)
31	- `r` is the rank, much smaller than d and k
32
33	The key insight: weight updates during fine-tuning have low intrinsic rank, so we can represent them efficiently with smaller matrices.
34
35	### Why Use LoRA
36
37	\| Aspect \| Full Fine-tuning \| LoRA \|
38	\|--------\|------------------\|------\|
39	\| Trainable params \| 100% \| ~0.1-1% \|
40	\| Memory usage \| High \| Low \|
41	\| Adapter size \| Full model \| ~3-100 MB \|
42	\| Training speed \| Slower \| Faster \|
43	\| Multiple tasks \| Separate models \| Swap adapters \|
44
45	## Basic Setup
46
47	### Installation
48
49	```bash
50	pip install peft transformers accelerate
51	```
52
53	### Minimal Example
54
55	```python
56	from transformers import AutoModelForCausalLM, AutoTokenizer
57	from peft import LoraConfig, get_peft_model, TaskType
58	import torch
59
60	# Load base model
61	model_name = "meta-llama/Llama-3.2-1B"
62	model = AutoModelForCausalLM.from_pretrained(
63	model_name,
64	dtype=torch.bfloat16,
65	device_map="auto",
66	)
67	tokenizer = AutoTokenizer.from_pretrained(model_name)
68	tokenizer.pad_token = tokenizer.eos_token
69
70	# Configure LoRA
71	lora_config = LoraConfig(
72	r=16,
73	lora_alpha=32,
74	target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
75	lora_dropout=0.05,
76	bias="none",
77	task_type=TaskType.CAUSAL_LM,
78	)
79
80	# Apply LoRA
81	model = get_peft_model(model, lora_config)
82	model.print_trainable_parameters()
83	# trainable params: 3,407,872 \|\| all params: 1,238,300,672 \|\| trainable%: 0.28%
84	```
85
86	## Configuration Parameters
87
88	### LoraConfig Options
89
90	```python
91	from peft import LoraConfig, TaskType
92
93	config = LoraConfig(
94	# Core parameters
95	r=16, # Rank of update matrices
96	lora_alpha=32, # Scaling factor (alpha/r applied to updates)
97	target_modules=["q_proj", "v_proj"], # Layers to adapt
98
99	# Regularization
100	lora_dropout=0.05, # Dropout on LoRA layers
101	bias="none", # "none", "all", or "lora_only"
102
103	# Task configuration
104	task_type=TaskType.CAUSAL_LM, # CAUSAL_LM, SEQ_CLS, SEQ_2_SEQ_LM, TOKEN_CLS
105
106	# Advanced
107	modules_to_save=None, # Additional modules to train (e.g., ["lm_head"])
108	layers_to_transform=None, # Specific layer indices to adapt
109	rank_pattern=None, # Per-module rank overrides
110	alpha_pattern=None, # Per-module alpha overrides
111	trainable_token_indices=None, # Train only selected new token embeddings
112	target_parameters=None, # Target nn.Parameter weights in some MoE layers
113	use_rslora=False, # Rank-stabilized LoRA scaling
114	use_dora=False, # Weight-Decomposed LoRA
115	)
116	```
117
118	### Target Modules by Architecture
119
120	```python
121	# Llama, Mistral, Qwen
122	target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
123
124	# GPT-2, GPT-J
125	target_modules = ["c_attn", "c_proj", "c_fc"]
126
127	# BERT, RoBERTa
128	target_modules = ["query", "key", "value", "dense"]
129
130	# Falcon
131	target_modules = ["query_key_value", "dense", "dense_h_to_4h", "dense_4h_to_h"]
132
133	# Phi
134	target_modules = ["q_proj", "k_proj", "v_proj", "dense", "fc1", "fc2"]
135
136	# MoE expert parameters stored as nn.Parameter tensors
137	target_parameters = ["feed_forward.experts.gate_up_proj", "feed_forward.experts.down_proj"]
138	```
139
140	For MoE models where expert weights are not `nn.Linear` modules, use `target_parameters` rather than `target_modules`. Merge adapters before latency-sensitive inference because materializing expert LoRA updates can add overhead.
141
142	### Finding Target Modules
143
144	```python
145	# Print all linear layer names
146	from peft.utils import get_peft_model_state_dict
147
148	def find_target_modules(model):
149	linear_modules = set()
150	for name, module in model.named_modules():
151	if isinstance(module, torch.nn.Linear):
152	# Get the last part of the name (e.g., "q_proj" from "model.lay

People who liked this also viewed

6 picks

Type

lora

23installs0forks

View on GitHub

How to install lora?

itsmostafa/llm-engineering-skills/lora

$npx -y skills add itsmostafa/llm-engineering-skills --skill lora

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Using LoRA for Fine-tuning
2
3	LoRA (Low-Rank Adaptation) enables efficient fine-tuning by freezing pretrained weights and injecting small trainable matrices into transformer layers. This reduces trainable parameters to ~0.1% of the original model while maintaining performance.
4
5	## Table of Contents
6
7	- [Core Concepts](#core-concepts)
8	- [Basic Setup](#basic-setup)
9	- [Configuration Parameters](#configuration-parameters)
10	- [QLoRA (Quantized LoRA)](#qlora-quantized-lora)
11	- [Training Patterns](#training-patterns)
12	- [Saving and Loading](#saving-and-loading)
13	- [Merging Adapters](#merging-adapters)
14	- [Best Practices](#best-practices)
15	- [References](#references)
16
17	## Core Concepts
18
19	### How LoRA Works
20
21	Instead of updating all weights during fine-tuning, LoRA decomposes weight updates into low-rank matrices:
22
23	```
24	W' = W + BA
25	```
26
27	Where:
28	- `W` is the frozen pretrained weight matrix (d × k)
29	- `B` is a trainable matrix (d × r)
30	- `A` is a trainable matrix (r × k)
31	- `r` is the rank, much smaller than d and k
32
33	The key insight: weight updates during fine-tuning have low intrinsic rank, so we can represent them efficiently with smaller matrices.
34
35	### Why Use LoRA
36
37	\| Aspect \| Full Fine-tuning \| LoRA \|
38	\|--------\|------------------\|------\|
39	\| Trainable params \| 100% \| ~0.1-1% \|
40	\| Memory usage \| High \| Low \|
41	\| Adapter size \| Full model \| ~3-100 MB \|
42	\| Training speed \| Slower \| Faster \|
43	\| Multiple tasks \| Separate models \| Swap adapters \|
44
45	## Basic Setup
46
47	### Installation
48
49	```bash
50	pip install peft transformers accelerate
51	```
52
53	### Minimal Example
54
55	```python
56	from transformers import AutoModelForCausalLM, AutoTokenizer
57	from peft import LoraConfig, get_peft_model, TaskType
58	import torch
59
60	# Load base model
61	model_name = "meta-llama/Llama-3.2-1B"
62	model = AutoModelForCausalLM.from_pretrained(
63	model_name,
64	dtype=torch.bfloat16,
65	device_map="auto",
66	)
67	tokenizer = AutoTokenizer.from_pretrained(model_name)
68	tokenizer.pad_token = tokenizer.eos_token
69
70	# Configure LoRA
71	lora_config = LoraConfig(
72	r=16,
73	lora_alpha=32,
74	target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
75	lora_dropout=0.05,
76	bias="none",
77	task_type=TaskType.CAUSAL_LM,
78	)
79
80	# Apply LoRA
81	model = get_peft_model(model, lora_config)
82	model.print_trainable_parameters()
83	# trainable params: 3,407,872 \|\| all params: 1,238,300,672 \|\| trainable%: 0.28%
84	```
85
86	## Configuration Parameters
87
88	### LoraConfig Options
89
90	```python
91	from peft import LoraConfig, TaskType
92
93	config = LoraConfig(
94	# Core parameters
95	r=16, # Rank of update matrices
96	lora_alpha=32, # Scaling factor (alpha/r applied to updates)
97	target_modules=["q_proj", "v_proj"], # Layers to adapt
98
99	# Regularization
100	lora_dropout=0.05, # Dropout on LoRA layers
101	bias="none", # "none", "all", or "lora_only"
102
103	# Task configuration
104	task_type=TaskType.CAUSAL_LM, # CAUSAL_LM, SEQ_CLS, SEQ_2_SEQ_LM, TOKEN_CLS
105
106	# Advanced
107	modules_to_save=None, # Additional modules to train (e.g., ["lm_head"])
108	layers_to_transform=None, # Specific layer indices to adapt
109	rank_pattern=None, # Per-module rank overrides
110	alpha_pattern=None, # Per-module alpha overrides
111	trainable_token_indices=None, # Train only selected new token embeddings
112	target_parameters=None, # Target nn.Parameter weights in some MoE layers
113	use_rslora=False, # Rank-stabilized LoRA scaling
114	use_dora=False, # Weight-Decomposed LoRA
115	)
116	```
117
118	### Target Modules by Architecture
119
120	```python
121	# Llama, Mistral, Qwen
122	target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
123
124	# GPT-2, GPT-J
125	target_modules = ["c_attn", "c_proj", "c_fc"]
126
127	# BERT, RoBERTa
128	target_modules = ["query", "key", "value", "dense"]
129
130	# Falcon
131	target_modules = ["query_key_value", "dense", "dense_h_to_4h", "dense_4h_to_h"]
132
133	# Phi
134	target_modules = ["q_proj", "k_proj", "v_proj", "dense", "fc1", "fc2"]
135
136	# MoE expert parameters stored as nn.Parameter tensors
137	target_parameters = ["feed_forward.experts.gate_up_proj", "feed_forward.experts.down_proj"]
138	```
139
140	For MoE models where expert weights are not `nn.Linear` modules, use `target_parameters` rather than `target_modules`. Merge adapters before latency-sensitive inference because materializing expert LoRA updates can add overhead.
141
142	### Finding Target Modules
143
144	```python
145	# Print all linear layer names
146	from peft.utils import get_peft_model_state_dict
147
148	def find_target_modules(model):
149	linear_modules = set()
150	for name, module in model.named_modules():
151	if isinstance(module, torch.nn.Linear):
152	# Get the last part of the name (e.g., "q_proj" from "model.lay

People who liked this also viewed

6 picks

Type