PromptAgent
starsforks
How to install PromptAgent?
maitrix-org/promptagent
$git clone https://github.com/maitrix-org/PromptAgentThe following command run PromptAgent to craft an expert prompt for a BIG-bench task, penguins_in_a_table. The running could take some time depending on the inference speed of OpenAI API
README.md
| 1 | <p align="center"> |
| 2 | <img src="./images/Header.png" alt="Expert-level Prompting" title="Expert-level Prompting"/> |
| 3 | </p> |
| 4 | |
| 5 | ## News |
| 6 | - Aug. 04, 2024: PromptAgent now support vllm LLMs, which speed up the local inference. Thanks Jiazheng for the commit! |
| 7 | - Jun. 27, 2024: LLM Reasoners library now integrates PromptAgent! See the **adapted PromptAgent** here: [Link](https://github.com/maitrix-org/llm-reasoners/tree/main/examples/PromptAgent/MedQA) |
| 8 | - May 17, 2024: Modify arguments control, use yaml file to set arguments for PromptAgent. |
| 9 | - May 15, 2024: Different models (openai, palm, huggingface models) are supported by PromptAgent. |
| 10 | - Jan. 16, 2024: PromptAgent has been accepted by ICLR 2024! |
| 11 | - Dec. 17, 2023: Refined the code for ease of reading and use by users. |
| 12 | # PromptAgent |
| 13 | |
| 14 | |
| 15 | |
| 16 | This is the official repo for "PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization". PromptAgent is a novel automatic prompt optimization method that autonomously crafts prompts equivalent in quality to those handcrafted by experts, i.e., expert-level prompts. [[arXiv](https://arxiv.org/abs/2310.16427)] |
| 17 | <p align="center"> |
| 18 | <img src="./images/expert_prompt_00.jpg" alt="Expert-level Prompting" width="700" title="Expert-level Prompting"/> |
| 19 | </p> |
| 20 | Unlike discovering magic/local prompt variants as existing prompt optimization methods, expert-level prompting is still an untapped area that solves challenging problems. And PromptAgent serves as a principled framework to study prompt optimization by unifying prompt sampling and rewarding using MCTS algorithm. |
| 21 | <p align="center"> |
| 22 | <img src="./images/mcts_00.jpg" alt="Expert-level Prompting" width="700" title="Expert-level Prompting"/> |
| 23 | </p> |
| 24 | |
| 25 | ## Installation |
| 26 | |
| 27 | ```bash |
| 28 | git clone https://github.com/XinyuanWangCS/PromptAgent.git |
| 29 | cd PromptAgent |
| 30 | conda create -n prompt_agent |
| 31 | conda activate prompt_agent |
| 32 | pip install -r requirements.txt |
| 33 | ``` |
| 34 | |
| 35 | |
| 36 | ## Quick Start |
| 37 | |
| 38 | The following command run PromptAgent to craft an expert prompt for a BIG-bench task, [penguins_in_a_table](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/penguins_in_a_table). The running could take some time depending on the inference speed of OpenAI APIs and size of datasets. |
| 39 | |
| 40 | **Note**: Before running this command, please add your (OpenAI) api key to the example_config.yaml file (base_model_setting: api_key and optim_model_setting: api_key). You can also check all the other auguments in the yaml file. |
| 41 | ```bash |
| 42 | python src/main.py --config_dir example_config.yaml |
| 43 | ``` |
| 44 | |
| 45 | `penguins_in_a_table` is an table understanding task to answer questions about animals contained in tables. An example from the original dataset looks like this: |
| 46 | ``` |
| 47 | Here is a table where the first line is a header and each subsequent line is a penguin: |
| 48 | |
| 49 | name, age, height (cm), weight (kg) |
| 50 | Louis, 7, 50, 11 |
| 51 | Bernard, 5, 80, 13 |
| 52 | Vincent, 9, 60, 11 |
| 53 | Gwen, 8, 70, 15 |
| 54 | |
| 55 | For example: the age of Louis is 7, the weight of Gwen is 15 kg, the height of |
| 56 | Bernard is 80 cm. |
| 57 | |
| 58 | Which penguin is taller than the other ones? Answer: |
| 59 | ``` |
| 60 | Then, the expected result is `Bernard`. |
| 61 | |
| 62 | The initial query from the BIG-bench dataset is `Answer questions about a table of penguins and their attributes.` Starting with such an ordinary prompt, PromptAgent will strategically sample model errors (from the base model), generate error feedbacks (actions), simulate future rewards, and search for high-reward paths leadning to expert prompts. The optimized prompt for `penguins_in_a_table` will look like this (exact results may vary as this is not deterministic): |
| 63 | ``` |
| 64 | As you delve into a dataset of penguins, assess essential attributes like names, ages, |
| 65 | and gender. Decode the significance of each attribute in the context of every penguin |
| 66 | while keeping in mind that the dataset may be modified, including addition or removal |
| 67 | of penguins. When such modifications are made, immediately revise your understanding, |
| 68 | redo your computations, and ensure that your subsequent calculations consider these |
| 69 | chang |
People who liked this also viewed
6 picksType
- components-helperBased on the docs to provide code prompt files for vue component libraryPromptsNov 2023
- VibeCode-Protocol-SuiteA comprehensive suite of protocols, meta-prompts, and orchestration tools designed to streamline software development workflows, project management, and team collaboration. Includes the VibeCode Protocol Suite with web interface, user manuals, and reusable templates for efficient development processes.PromptsJul 2026
- Promptable-web-sdkWeb SDK for Promptable Website.PromptsDec 2022
- LLMFlexA python package for developing AI applications with local LLMs.PromptsJan 2025
- cli-repo-to-promptexport any codebase to a single LLM-ready markdown promptPromptsJul 2026
- Files2PromptVS Code extension to copy files for an LLM promptPromptsApr 2025
