minihf

Name: minihf
Author: JD-P

184stars12forks

How to install minihf?

jd-p/minihf

$git clone https://github.com/JD-P/minihf

Installs into the current project.

Summary

MiniHF is an inference, human preference data collection, and fine-tuning tool for local language models. It is intended to help the user develop their prompts into full models. Normally when we prompt a language model we're forced to think in that models latent space. MiniHF let

Files · 1

View on GitHub

README.md

1	![vector art logo of the brain patterned in the style of a pictorial history of
2	Portuguese textiles, painted in the 1700s. With the logotext "MiniHF"](static/minihf_logo_text.png)
3
4	MiniHF is an inference, human preference data collection, and fine-tuning tool
5	for local language models. It is intended to help the user develop their prompts
6	into full models. Normally when we prompt a language model we're forced to think
7	in that models latent space. MiniHF lets you go the other direction: Imagine the
8	ideal context in which your prompt could take place and then add it to the model.
9	To make this possible MiniHF provides several powerful features:
10
11	* Lightweight web interface and inference server that lets you easily branch your
12	session with the model into multiple completion chains and pick the best ones
13
14	* Make your own feedback dataset by writing with local
15	language models such as StableLM and NeoX 20b.
16
17	* A monte carlo tree search (MCTS) based inference algorithm, Weave, which rejection
18	samples from the model to improve output quality
19
20	* The ability to finetune both the underlying generator LoRa and the evaluator
21	reward LoRa used for the tree search on your own custom dataset
22
23	* Easy bootstrapping of new document contexts and models using reinforcement
24	learning from AI feedback (RLAIF)
25
26	* Easy install with minimal dependencies
27
28	If you want to discuss MiniHF with other users, [we have a discord server](https://discord.gg/Tfa6MabKVF).
29
30	## Setup
31
32	### DataCrunch
33
34	If you want to use MiniHF with a large amount of VRAM, https://datacrunch.io/ is
35	a good option. Provision an A6000, A6000 Ada, etc from their selection with
36	the `Ubuntu + Cuda + Docker` image and more than the default amount of storage
37	(I found it worked with 1tb, but you can probably make do with less, **the default
38	40gb is not enough**). Once you have it running SSH in with a listener on port 5000:
39
40	```
41	ssh root@IP_ADDRESS -L 5000:localhost:5000
42	```
43
44	Once you're in clone the repo and change directories to it:
45
46	```
47	git clone https://github.com/JD-P/minihf.git
48	cd minihf
49	```
50
51	Then run:
52
53	```
54	bash setup.sh
55	```
56
57	You should see the script update the server packages, install python dependencies
58	from pip, download the models, and then finally start the inference server. At
59	this point you can start using MiniHF by visiting http://localhost:5000/ in your
60	browser. You can change which models you're using in the `minihf_infer.py` file.
61	Later on we'll add a configuration file to change these settings.
62
63	To start the server yourself on subsequent logins use the commands:
64
65	```
66	cd minihf
67	source env_minihf/bin/activate
68	flask --app minihf_infer run
69	```
70
71	## Tuning Models
72
73	MiniHF lets you tune two model types, both of which are LoRa tunes on an underlying
74	foundation model such as [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6b),
75	[NeoX](https://github.com/EleutherAI/gpt-neox),
76	[OpenLlama](https://huggingface.co/openlm-research/open_llama_7b_v2), or [falcon-40b](https://huggingface.co/tiiuae/falcon-40b):
77
78	1. Generator LoRa - Generates the text that the user or Weave algorithm evaluates.
79
80	2. Evaluator LoRa - Reward model that selects between branches in the Weave tree search.
81
82	Furthermore each model has two kinds of tuning, self-supervised finetuning (SFT) and
83	reinforcement learning from AI feedback (RLAIF).
84
85	![A diagram of the training flow chart for the models and tuning scripts in MiniHF](static/minihf_training_diagram.png)
86
87	### Preparing The Tuning Dataset
88
89	The tuning dataset should consist of a zip file containing one or more plaintext
90	files or json conversations exported from MiniHF. Because the model might not be
91	adapted to your style or document context yet, it might be more efficient to write
92	out the first drafts of what you want in text files and then start using MiniHF
93	after you've tuned the generator on them.
94
95	### Tuning The Generator
96
97	Once you have the tuning dataset it's easy to make a generator LoRa from it with
98	the `sft_generator.py` script:
99
100	```
101	python3 sft_generator.

People who liked this also viewed

6 picks

Type

minihf

184stars12forks

View on GitHub

How to install minihf?

jd-p/minihf

$git clone https://github.com/JD-P/minihf

Installs into the current project.

Summary

Files · 1

View on GitHub

README.md

1	![vector art logo of the brain patterned in the style of a pictorial history of
2	Portuguese textiles, painted in the 1700s. With the logotext "MiniHF"](static/minihf_logo_text.png)
3
4	MiniHF is an inference, human preference data collection, and fine-tuning tool
5	for local language models. It is intended to help the user develop their prompts
6	into full models. Normally when we prompt a language model we're forced to think
7	in that models latent space. MiniHF lets you go the other direction: Imagine the
8	ideal context in which your prompt could take place and then add it to the model.
9	To make this possible MiniHF provides several powerful features:
10
11	* Lightweight web interface and inference server that lets you easily branch your
12	session with the model into multiple completion chains and pick the best ones
13
14	* Make your own feedback dataset by writing with local
15	language models such as StableLM and NeoX 20b.
16
17	* A monte carlo tree search (MCTS) based inference algorithm, Weave, which rejection
18	samples from the model to improve output quality
19
20	* The ability to finetune both the underlying generator LoRa and the evaluator
21	reward LoRa used for the tree search on your own custom dataset
22
23	* Easy bootstrapping of new document contexts and models using reinforcement
24	learning from AI feedback (RLAIF)
25
26	* Easy install with minimal dependencies
27
28	If you want to discuss MiniHF with other users, [we have a discord server](https://discord.gg/Tfa6MabKVF).
29
30	## Setup
31
32	### DataCrunch
33
34	If you want to use MiniHF with a large amount of VRAM, https://datacrunch.io/ is
35	a good option. Provision an A6000, A6000 Ada, etc from their selection with
36	the `Ubuntu + Cuda + Docker` image and more than the default amount of storage
37	(I found it worked with 1tb, but you can probably make do with less, **the default
38	40gb is not enough**). Once you have it running SSH in with a listener on port 5000:
39
40	```
41	ssh root@IP_ADDRESS -L 5000:localhost:5000
42	```
43
44	Once you're in clone the repo and change directories to it:
45
46	```
47	git clone https://github.com/JD-P/minihf.git
48	cd minihf
49	```
50
51	Then run:
52
53	```
54	bash setup.sh
55	```
56
57	You should see the script update the server packages, install python dependencies
58	from pip, download the models, and then finally start the inference server. At
59	this point you can start using MiniHF by visiting http://localhost:5000/ in your
60	browser. You can change which models you're using in the `minihf_infer.py` file.
61	Later on we'll add a configuration file to change these settings.
62
63	To start the server yourself on subsequent logins use the commands:
64
65	```
66	cd minihf
67	source env_minihf/bin/activate
68	flask --app minihf_infer run
69	```
70
71	## Tuning Models
72
73	MiniHF lets you tune two model types, both of which are LoRa tunes on an underlying
74	foundation model such as [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6b),
75	[NeoX](https://github.com/EleutherAI/gpt-neox),
76	[OpenLlama](https://huggingface.co/openlm-research/open_llama_7b_v2), or [falcon-40b](https://huggingface.co/tiiuae/falcon-40b):
77
78	1. Generator LoRa - Generates the text that the user or Weave algorithm evaluates.
79
80	2. Evaluator LoRa - Reward model that selects between branches in the Weave tree search.
81
82	Furthermore each model has two kinds of tuning, self-supervised finetuning (SFT) and
83	reinforcement learning from AI feedback (RLAIF).
84
85	![A diagram of the training flow chart for the models and tuning scripts in MiniHF](static/minihf_training_diagram.png)
86
87	### Preparing The Tuning Dataset
88
89	The tuning dataset should consist of a zip file containing one or more plaintext
90	files or json conversations exported from MiniHF. Because the model might not be
91	adapted to your style or document context yet, it might be more efficient to write
92	out the first drafts of what you want in text files and then start using MiniHF
93	after you've tuned the generator on them.
94
95	### Tuning The Generator
96
97	Once you have the tuning dataset it's easy to make a generator LoRa from it with
98	the `sft_generator.py` script:
99
100	```
101	python3 sft_generator.

People who liked this also viewed

6 picks

Type