grpo-rl-training

Name: grpo-rl-training
Author: AlexAI-MCP

132stars24forks

How to install grpo-rl-training?

alexai-mcp/hermes-ccc/grpo-rl-training

$npx -y skills add AlexAI-MCP/hermes-CCC --skill grpo-rl-training

Installs into the current project.

Summary

Expert guidance for GRPO/RL fine-tuning with TRL for reasoning and task-specific model training.

Files · 1

View on GitHub

SKILL.md

1	# GRPO/RL Training with TRL
2
3	Expert guidance for Group Relative Policy Optimization (GRPO) using the TRL library. Battle-tested patterns for fine-tuning language models with custom reward functions.
4
5	## When to Use
6
7	- Teaching a model to follow specific output formats (JSON, structured reasoning)
8	- Improving accuracy on math, coding, or reasoning tasks
9	- Custom task-specific behavior without labeled datasets
10	- Distilling reasoning capabilities from larger models
11
12	---
13
14	## Setup
15
16	```bash
17	pip install transformers>=4.47.0 trl>=0.14.0 datasets>=3.2.0 peft>=0.14.0 torch accelerate
18	```
19
20	Optional for logging:
21	```bash
22	pip install wandb
23	wandb login
24	```
25
26	---
27
28	## Minimal GRPO Example
29
30	```python
31	from trl import GRPOConfig, GRPOTrainer
32	from transformers import AutoModelForCausalLM, AutoTokenizer
33	from datasets import load_dataset
34
35	model_name = "Qwen/Qwen2.5-1.5B-Instruct"
36	model = AutoModelForCausalLM.from_pretrained(model_name)
37	tokenizer = AutoTokenizer.from_pretrained(model_name)
38
39	dataset = load_dataset("your/dataset")
40
41	# Define reward function
42	def reward_fn(prompts, completions, **kwargs):
43	"""
44	Returns list of floats — one reward per completion.
45	Higher = better. Typically in range [-1, 1] or [0, 1].
46	"""
47	rewards = []
48	for completion in completions:
49	# Example: reward for correct format
50	if completion.strip().startswith("<answer>"):
51	rewards.append(1.0)
52	else:
53	rewards.append(-0.5)
54	return rewards
55
56	config = GRPOConfig(
57	output_dir="./grpo-output",
58	learning_rate=5e-6,
59	num_train_epochs=3,
60	per_device_train_batch_size=4,
61	gradient_accumulation_steps=4,
62	num_generations=8, # completions per prompt (G in GRPO)
63	max_prompt_length=512,
64	max_completion_length=256,
65	kl_coef=0.1, # KL penalty — start low, tune up if reward hacking
66	logging_steps=10,
67	save_steps=100,
68	report_to="wandb", # or "none"
69	)
70
71	trainer = GRPOTrainer(
72	model=model,
73	args=config,
74	reward_funcs=reward_fn,
75	train_dataset=dataset["train"],
76	tokenizer=tokenizer,
77	)
78
79	trainer.train()
80	trainer.save_model("./grpo-final")
81	```
82
83	---
84
85	## Reward Function Patterns
86
87	### Format Reward
88	```python
89	import re
90
91	def format_reward(prompts, completions, **kwargs):
92	pattern = r"<think>.?</think>\s<answer>.*?</answer>"
93	return [1.0 if re.fullmatch(pattern, c, re.DOTALL) else -1.0 for c in completions]
94	```
95
96	### Accuracy Reward
97	```python
98	def accuracy_reward(prompts, completions, ground_truth, **kwargs):
99	rewards = []
100	for completion, gt in zip(completions, ground_truth):
101	extracted = extract_answer(completion)
102	rewards.append(1.0 if extracted == gt else 0.0)
103	return rewards
104	```
105
106	### Length Penalty
107	```python
108	def length_penalty_reward(prompts, completions, **kwargs):
109	return [max(0, 1.0 - len(c) / 2000) for c in completions]
110	```
111
112	### Combined Rewards
113	```python
114	reward_funcs = [format_reward, accuracy_reward] # TRL averages them
115	```
116
117	---
118
119	## PEFT/LoRA Integration (Memory Efficient)
120
121	```python
122	from peft import LoraConfig, get_peft_model
123
124	lora_config = LoraConfig(
125	r=16,
126	lora_alpha=32,
127	target_modules=["q_proj", "v_proj"],
128	lora_dropout=0.05,
129	bias="none",
130	task_type="CAUSAL_LM",
131	)
132
133	model = get_peft_model(model, lora_config)
134	model.print_trainable_parameters()
135	```
136
137	---
138
139	## Key Hyperparameters
140
141	\| Param \| Default \| Effect \|
142	\|-------\|---------\|--------\|
143	\| `num_generations` \| 8 \| More = better gradient estimate, more memory \|
144	\| `kl_coef` \| 0.1 \| Higher = stays closer to reference model \|
145	\| `learning_rate` \| 5e-6 \| Lower than SFT — RL is unstable with high LR \|
146	\| `max_completion_length` \| 256 \| Controls max tokens generated \|
147
148	---
149
150	## Common Pitfalls
151
152	Reward Hacking: Model finds shortcuts to maximize reward without actually improving.
153	→ Fix: Add KL penalty (`kl_coef`), diverse reward signals, human evaluation
154
155	Mode Collapse: All completions become similar.
156	→ Fix: Increase `num_generations`, add temperature, check reward diversity
157
158	OOM: Large models with many generations blow up memory.
159	→ Fix: Use LoRA, reduce `num_generations`, use `gradient_checkpointing=True`
160
161	Reward too sparse: Model rarely gets positive reward → no learning signal.
162	→ Fix: Start with easier examples, use shaped reward (partial credit)
163
164	---
165
166	## Monitoring with W&B
167
168	Key metrics to watch:
169	- `train/reward`: should increase over time
170	- `train/kl`: should stay bounded (spike = reward hacking)
171	- `train/policy_loss`: learning signal
172	- `train/entropy`: diversity of completions (drop = mode collapse)
173
174	---
175
176	## Checkpoint and Resume
177
178	```bash
179	# Resume f

People who liked this also viewed

6 picks

Type

lark-drive飞书云空间（云盘/云存储）：管理 Drive 文件和文件夹，包含上传/下载、创建文件夹、复制/移动/删除、查看元数据、评论/权限/订阅、标题、版本、飞书文档密级标签（secure labels）和本地文件导入。用户需要整理云盘目录、处理云空间资源 URL/token、判断链接类型/真实 token/标题，或导入 Word/Markdown/Excel/CSV/PPTX/.base 为 docx/sSkillsJul 2026382k16k
lark-sharedUse for lark-cli setup/auth tasks: auth login/status/logout, user vs bot identity, business-domain permissions (--domain, including all/docs/drive), missing scopes, revoking authorization, or handlingSkillsJul 2026382k16k
lark-whiteboard飞书画板：查询和编辑飞书云文档中的画板。支持导出画板为预览图片、导出原始节点结构、使用多种格式更新画板内容。当用户需要查看画板内容、导出画板图片、编辑画板时使用此 skill。不负责：飞书云文档内容编辑（lark-doc）、文档内嵌电子表格/Base（lark-sheets / lark-base）。SkillsJul 2026380k16k
lark-task飞书任务：管理任务、清单和任务智能体。创建待办任务、查看和更新任务状态、拆分子任务、组织任务清单、分配协作成员、上传任务附件、注册或注销任务智能体、更新任务智能体的主页数据、写入智能体任务记录。当用户需要创建待办事项、查看任务列表、跟踪任务进度、管理项目清单或给他人分配任务、为任务上传附件文件、注册注销任务智能体、更新智能体主页数据、写入任务记录时使用。SkillsJul 2026380k16k
lark-attendance飞书考勤打卡：查询自己的考勤打卡记录SkillsJul 2026330k16k
lark-apps妙搭（Spark/Miaoda）应用开发与托管：应用创建、本地全栈开发、云端生成迭代、创意设计（UI mockup / 可交互原型 / 线框图 / 落地页 / 仪表盘 / 幻灯片 deck / 视觉探索）、AI相关能力和飞书平台能力或者其他外部能力集成、日志/Trace/监控指标/PV/UV 查询、环境变量管理、应用角色与成员管理、自动化触发器（定时/记录变更/Webhook/飞书审批）。当用户要SkillsJul 2026227k16k

grpo-rl-training

132stars24forks

View on GitHub

How to install grpo-rl-training?

alexai-mcp/hermes-ccc/grpo-rl-training

$npx -y skills add AlexAI-MCP/hermes-CCC --skill grpo-rl-training

Installs into the current project.

Summary

Expert guidance for GRPO/RL fine-tuning with TRL for reasoning and task-specific model training.

Files · 1

View on GitHub

SKILL.md

1	# GRPO/RL Training with TRL
2
3	Expert guidance for Group Relative Policy Optimization (GRPO) using the TRL library. Battle-tested patterns for fine-tuning language models with custom reward functions.
4
5	## When to Use
6
7	- Teaching a model to follow specific output formats (JSON, structured reasoning)
8	- Improving accuracy on math, coding, or reasoning tasks
9	- Custom task-specific behavior without labeled datasets
10	- Distilling reasoning capabilities from larger models
11
12	---
13
14	## Setup
15
16	```bash
17	pip install transformers>=4.47.0 trl>=0.14.0 datasets>=3.2.0 peft>=0.14.0 torch accelerate
18	```
19
20	Optional for logging:
21	```bash
22	pip install wandb
23	wandb login
24	```
25
26	---
27
28	## Minimal GRPO Example
29
30	```python
31	from trl import GRPOConfig, GRPOTrainer
32	from transformers import AutoModelForCausalLM, AutoTokenizer
33	from datasets import load_dataset
34
35	model_name = "Qwen/Qwen2.5-1.5B-Instruct"
36	model = AutoModelForCausalLM.from_pretrained(model_name)
37	tokenizer = AutoTokenizer.from_pretrained(model_name)
38
39	dataset = load_dataset("your/dataset")
40
41	# Define reward function
42	def reward_fn(prompts, completions, **kwargs):
43	"""
44	Returns list of floats — one reward per completion.
45	Higher = better. Typically in range [-1, 1] or [0, 1].
46	"""
47	rewards = []
48	for completion in completions:
49	# Example: reward for correct format
50	if completion.strip().startswith("<answer>"):
51	rewards.append(1.0)
52	else:
53	rewards.append(-0.5)
54	return rewards
55
56	config = GRPOConfig(
57	output_dir="./grpo-output",
58	learning_rate=5e-6,
59	num_train_epochs=3,
60	per_device_train_batch_size=4,
61	gradient_accumulation_steps=4,
62	num_generations=8, # completions per prompt (G in GRPO)
63	max_prompt_length=512,
64	max_completion_length=256,
65	kl_coef=0.1, # KL penalty — start low, tune up if reward hacking
66	logging_steps=10,
67	save_steps=100,
68	report_to="wandb", # or "none"
69	)
70
71	trainer = GRPOTrainer(
72	model=model,
73	args=config,
74	reward_funcs=reward_fn,
75	train_dataset=dataset["train"],
76	tokenizer=tokenizer,
77	)
78
79	trainer.train()
80	trainer.save_model("./grpo-final")
81	```
82
83	---
84
85	## Reward Function Patterns
86
87	### Format Reward
88	```python
89	import re
90
91	def format_reward(prompts, completions, **kwargs):
92	pattern = r"<think>.?</think>\s<answer>.*?</answer>"
93	return [1.0 if re.fullmatch(pattern, c, re.DOTALL) else -1.0 for c in completions]
94	```
95
96	### Accuracy Reward
97	```python
98	def accuracy_reward(prompts, completions, ground_truth, **kwargs):
99	rewards = []
100	for completion, gt in zip(completions, ground_truth):
101	extracted = extract_answer(completion)
102	rewards.append(1.0 if extracted == gt else 0.0)
103	return rewards
104	```
105
106	### Length Penalty
107	```python
108	def length_penalty_reward(prompts, completions, **kwargs):
109	return [max(0, 1.0 - len(c) / 2000) for c in completions]
110	```
111
112	### Combined Rewards
113	```python
114	reward_funcs = [format_reward, accuracy_reward] # TRL averages them
115	```
116
117	---
118
119	## PEFT/LoRA Integration (Memory Efficient)
120
121	```python
122	from peft import LoraConfig, get_peft_model
123
124	lora_config = LoraConfig(
125	r=16,
126	lora_alpha=32,
127	target_modules=["q_proj", "v_proj"],
128	lora_dropout=0.05,
129	bias="none",
130	task_type="CAUSAL_LM",
131	)
132
133	model = get_peft_model(model, lora_config)
134	model.print_trainable_parameters()
135	```
136
137	---
138
139	## Key Hyperparameters
140
141	\| Param \| Default \| Effect \|
142	\|-------\|---------\|--------\|
143	\| `num_generations` \| 8 \| More = better gradient estimate, more memory \|
144	\| `kl_coef` \| 0.1 \| Higher = stays closer to reference model \|
145	\| `learning_rate` \| 5e-6 \| Lower than SFT — RL is unstable with high LR \|
146	\| `max_completion_length` \| 256 \| Controls max tokens generated \|
147
148	---
149
150	## Common Pitfalls
151
152	Reward Hacking: Model finds shortcuts to maximize reward without actually improving.
153	→ Fix: Add KL penalty (`kl_coef`), diverse reward signals, human evaluation
154
155	Mode Collapse: All completions become similar.
156	→ Fix: Increase `num_generations`, add temperature, check reward diversity
157
158	OOM: Large models with many generations blow up memory.
159	→ Fix: Use LoRA, reduce `num_generations`, use `gradient_checkpointing=True`
160
161	Reward too sparse: Model rarely gets positive reward → no learning signal.
162	→ Fix: Start with easier examples, use shaped reward (partial credit)
163
164	---
165
166	## Monitoring with W&B
167
168	Key metrics to watch:
169	- `train/reward`: should increase over time
170	- `train/kl`: should stay bounded (spike = reward hacking)
171	- `train/policy_loss`: learning signal
172	- `train/entropy`: diversity of completions (drop = mode collapse)
173
174	---
175
176	## Checkpoint and Resume
177
178	```bash
179	# Resume f

People who liked this also viewed

6 picks

Type