AutoPrompter

Name: AutoPrompter
Author: gauravvij

50stars4forks

How to install AutoPrompter?

gauravvij/autoprompter

$git clone https://github.com/gauravvij/AutoPrompter

Installs into the current project.

Summary

AutoPrompter is an autonomous system designed to iteratively improve LLM prompts through a closed-loop optimization process. It merges the validation and metrics capabilities of tools like promptfoo with the iterative improvement logic of autoresearch.

Files · 1

View on GitHub

README.md

1	# AutoPrompter: Autonomous Prompt Optimization System
2
3	<p align="center">
4	<a href="https://heyneo.so" target="_blank">
5	<img src="https://img.shields.io/badge/Made%20by-NEO-ff3b30?style=for-the-badge&logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSJjdXJyZW50Q29sb3IiIHN0cm9rZS13aWR0aD0iMiI+PHBhdGggZD0iTTEyIDJMNCA3djZsOCA1IDgtNXYtNmwtOC01eiIvPjxwYXRoIGQ9Ik00IDEzbDggNSA4LTUiLz48L3N2Zz4=&logoColor=white" alt="Made by NEO">
6	</a>
7	</p>
8
9	AutoPrompter is an autonomous system designed to iteratively improve LLM prompts through a closed-loop optimization process. It merges the validation and metrics capabilities of tools like `promptfoo` with the iterative improvement logic of `autoresearch`.
10
11	## System Architecture
12
13	The system operates in a continuous loop where an Optimizer LLM refines prompts for a Target LLM based on empirical performance data.
14
15	1. Dataset Generation: The Optimizer LLM (Gemini 3.1 Flash - customizable through config.yaml) generates a synthetic dataset of input/output pairs based on the task description.
16	2. Iterative Improvement:
17	* The Target LLM (Qwen 3.5 9b) is tested against the current prompt using the generated dataset.
18	* Performance is measured using a defined metric (Accuracy, F1, Semantic Similarity, etc.).
19	* The Optimizer LLM analyzes failures and successes to generate a refined prompt.
20	3. Experiment Ledger: Every iteration is recorded in a persistent ledger to prevent duplicate experiments and track progress.
21	4. Context Management: The system manages the history of experiments to provide the Optimizer LLM with relevant context without exceeding window limits.
22
23	### Core Components
24	- Optimizer LLM: `google/gemini-3.1-flash-lite-preview` (via OpenRouter)
25	- Target LLM: `qwen/qwen3.5-9b` (via OpenRouter)
26	- Metrics: Automated evaluation against expected outputs.
27	- Ledger: JSON-based tracking of all experiments.
28
29	## Installation
30
31	1. Clone the repository:
32	```bash
33	git clone https://github.com/gauravvij/AutoPrompter.git
34	cd AutoPrompter
35	```
36
37	2. Create and activate a virtual environment:
38	```bash
39	python3 -m venv venv
40	source venv/bin/activate
41	```
42
43	3. Install dependencies:
44	```bash
45	pip install -r requirements.txt
46	```
47
48	4. Configure API Key (for OpenRouter backend):
49	Set the `OPENROUTER_API_KEY` environment variable or add it to `/root/.config/openrouter/config`.
50
51	Note: API key is only required when using OpenRouter backend. Local backends (Ollama, llama.cpp) do not require API keys.
52
53	## Configuration
54
55	The system is configured via YAML files. Key fields include:
56
57	- `optimizer_llm`: Model ID and parameters for the optimizer.
58	- `target_llm`: Model ID and parameters for the target.
59	- `experiment`: `max_iterations`, `batch_size`, and convergence thresholds.
60	- `task`: `name`, `description`, and `initial_prompt`.
61	- `metric`: `type` (e.g., `accuracy`, `semantic_similarity`) and `target_score`.
62	- `storage`: Paths for the ledger, dataset, and results.
63
64	### Supported Backends
65
66	AutoPrompter supports multiple LLM backends:
67
68	1. OpenRouter (default): Cloud-based LLM access via OpenRouter API
69	2. Ollama: Local LLM inference using Ollama server
70	3. llama.cpp: Local LLM inference using llama.cpp server
71
72	#### OpenRouter Configuration (Default)
73
74	```yaml
75	optimizer_llm:
76	backend: "openrouter"
77	model: "google/gemini-3.1-flash-lite-preview"
78	api_base: "https://openrouter.ai/api/v1"
79	temperature: 0.7
80	max_tokens: 4096
81	```
82
83	#### Ollama Configuration
84
85	```yaml
86	optimizer_llm:
87	backend: "ollama"
88	model: "llama3.2"
89	host: "http://localhost"
90	port: 11434
91	temperature: 0.7
92	max_tokens: 4096
93	```
94
95	#### llama.cpp Configuration
96
97	```yaml
98	optimizer_llm:
99	backend: "llama_cpp"
100	model: "llama-3.2-3b"
101	host: "http://localhost"
102	port: 8080
103	temperature: 0.7
104	max_tokens: 4096
105	```
106
107	### Setting Up Local Backends
108
109	#### Ollama Setup

People who liked this also viewed

6 picks

Type

AutoPrompter

50stars4forks

View on GitHub

How to install AutoPrompter?

gauravvij/autoprompter

$git clone https://github.com/gauravvij/AutoPrompter

Installs into the current project.

Summary

Files · 1

View on GitHub

README.md

1	# AutoPrompter: Autonomous Prompt Optimization System
2
3	<p align="center">
4	<a href="https://heyneo.so" target="_blank">
5	<img src="https://img.shields.io/badge/Made%20by-NEO-ff3b30?style=for-the-badge&logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSJjdXJyZW50Q29sb3IiIHN0cm9rZS13aWR0aD0iMiI+PHBhdGggZD0iTTEyIDJMNCA3djZsOCA1IDgtNXYtNmwtOC01eiIvPjxwYXRoIGQ9Ik00IDEzbDggNSA4LTUiLz48L3N2Zz4=&logoColor=white" alt="Made by NEO">
6	</a>
7	</p>
8
9	AutoPrompter is an autonomous system designed to iteratively improve LLM prompts through a closed-loop optimization process. It merges the validation and metrics capabilities of tools like `promptfoo` with the iterative improvement logic of `autoresearch`.
10
11	## System Architecture
12
13	The system operates in a continuous loop where an Optimizer LLM refines prompts for a Target LLM based on empirical performance data.
14
15	1. Dataset Generation: The Optimizer LLM (Gemini 3.1 Flash - customizable through config.yaml) generates a synthetic dataset of input/output pairs based on the task description.
16	2. Iterative Improvement:
17	* The Target LLM (Qwen 3.5 9b) is tested against the current prompt using the generated dataset.
18	* Performance is measured using a defined metric (Accuracy, F1, Semantic Similarity, etc.).
19	* The Optimizer LLM analyzes failures and successes to generate a refined prompt.
20	3. Experiment Ledger: Every iteration is recorded in a persistent ledger to prevent duplicate experiments and track progress.
21	4. Context Management: The system manages the history of experiments to provide the Optimizer LLM with relevant context without exceeding window limits.
22
23	### Core Components
24	- Optimizer LLM: `google/gemini-3.1-flash-lite-preview` (via OpenRouter)
25	- Target LLM: `qwen/qwen3.5-9b` (via OpenRouter)
26	- Metrics: Automated evaluation against expected outputs.
27	- Ledger: JSON-based tracking of all experiments.
28
29	## Installation
30
31	1. Clone the repository:
32	```bash
33	git clone https://github.com/gauravvij/AutoPrompter.git
34	cd AutoPrompter
35	```
36
37	2. Create and activate a virtual environment:
38	```bash
39	python3 -m venv venv
40	source venv/bin/activate
41	```
42
43	3. Install dependencies:
44	```bash
45	pip install -r requirements.txt
46	```
47
48	4. Configure API Key (for OpenRouter backend):
49	Set the `OPENROUTER_API_KEY` environment variable or add it to `/root/.config/openrouter/config`.
50
51	Note: API key is only required when using OpenRouter backend. Local backends (Ollama, llama.cpp) do not require API keys.
52
53	## Configuration
54
55	The system is configured via YAML files. Key fields include:
56
57	- `optimizer_llm`: Model ID and parameters for the optimizer.
58	- `target_llm`: Model ID and parameters for the target.
59	- `experiment`: `max_iterations`, `batch_size`, and convergence thresholds.
60	- `task`: `name`, `description`, and `initial_prompt`.
61	- `metric`: `type` (e.g., `accuracy`, `semantic_similarity`) and `target_score`.
62	- `storage`: Paths for the ledger, dataset, and results.
63
64	### Supported Backends
65
66	AutoPrompter supports multiple LLM backends:
67
68	1. OpenRouter (default): Cloud-based LLM access via OpenRouter API
69	2. Ollama: Local LLM inference using Ollama server
70	3. llama.cpp: Local LLM inference using llama.cpp server
71
72	#### OpenRouter Configuration (Default)
73
74	```yaml
75	optimizer_llm:
76	backend: "openrouter"
77	model: "google/gemini-3.1-flash-lite-preview"
78	api_base: "https://openrouter.ai/api/v1"
79	temperature: 0.7
80	max_tokens: 4096
81	```
82
83	#### Ollama Configuration
84
85	```yaml
86	optimizer_llm:
87	backend: "ollama"
88	model: "llama3.2"
89	host: "http://localhost"
90	port: 11434
91	temperature: 0.7
92	max_tokens: 4096
93	```
94
95	#### llama.cpp Configuration
96
97	```yaml
98	optimizer_llm:
99	backend: "llama_cpp"
100	model: "llama-3.2-3b"
101	host: "http://localhost"
102	port: 8080
103	temperature: 0.7
104	max_tokens: 4096
105	```
106
107	### Setting Up Local Backends
108
109	#### Ollama Setup

People who liked this also viewed

6 picks

Type