cappr
starsforks
How to install cappr?
kddubey/cappr
$git clone https://github.com/kddubey/capprREADME.md
| 1 | # CAPPr: Completion After Prompt Probability |
| 2 | |
| 3 | [](https://www.python.org/downloads/release/python-380/) |
| 4 | [](https://github.com/kddubey/cappr/actions/workflows/test.yml) |
| 5 | [](https://codecov.io/gh/kddubey/cappr) |
| 6 | [](https://pypi.org/project/cappr/) |
| 7 | [](https://opensource.org/licenses/Apache-2.0) |
| 8 | |
| 9 | <!-- [](https://cappr.readthedocs.io/en/latest/?badge=latest) --> |
| 10 | |
| 11 | Make your LLM pick from a list of choices. <br> |
| 12 | Or compute the probability of a completion given a prompt, which may be |
| 13 | [useful](https://cappr.readthedocs.io/en/latest/related_work.html). <br> |
| 14 | Squeeze [more](https://cappr.readthedocs.io/en/latest/statistical_performance.html) out |
| 15 | of open source LLMs. |
| 16 | |
| 17 | |
| 18 | ## Usage |
| 19 | |
| 20 | <details> |
| 21 | <summary>Use a GGUF model</summary> |
| 22 | |
| 23 | ```python |
| 24 | from llama_cpp import Llama |
| 25 | from cappr.llama_cpp.classify import predict |
| 26 | |
| 27 | model = Llama("./TinyLLama-v0.Q8_0.gguf", verbose=False) |
| 28 | |
| 29 | prompt = """Gary told Spongebob a story: |
| 30 | There once was a man from Peru; who dreamed he was eating his shoe. He |
| 31 | woke with a fright, in the middle of the night, to find that his dream |
| 32 | had come true. |
| 33 | |
| 34 | The moral of the story is to""" |
| 35 | |
| 36 | completions = ( |
| 37 | "look at the bright side", |
| 38 | "use your imagination", |
| 39 | "eat shoes", |
| 40 | ) |
| 41 | |
| 42 | pred = predict(prompt, completions, model) |
| 43 | print(pred) |
| 44 | # use your imagination |
| 45 | ``` |
| 46 | |
| 47 | See [this page of the |
| 48 | documentation](https://cappr.readthedocs.io/en/latest/select_a_language_model.html#llama-cpp) |
| 49 | for more info on using GGUF models. |
| 50 | </details> |
| 51 | |
| 52 | |
| 53 | <details> |
| 54 | <summary>Use a Hugging Face transformers model</summary> |
| 55 | |
| 56 | ```python |
| 57 | from transformers import AutoModelForCausalLM, AutoTokenizer |
| 58 | from cappr.huggingface.classify import predict |
| 59 | |
| 60 | model_name = "gpt2" |
| 61 | model = AutoModelForCausalLM.from_pretrained(model_name) |
| 62 | tokenizer = AutoTokenizer.from_pretrained(model_name) |
| 63 | |
| 64 | prompt = "Which planet is closer to the Sun: Mercury or Earth?" |
| 65 | completions = ("Mercury", "Earth") |
| 66 | |
| 67 | pred = predict(prompt, completions, model_and_tokenizer=(model, tokenizer)) |
| 68 | print(pred) |
| 69 | # Mercury |
| 70 | ``` |
| 71 | |
| 72 | See [this page of the |
| 73 | documentation](https://cappr.readthedocs.io/en/latest/select_a_language_model.html#hugging-face) |
| 74 | for more info on using ``transformers`` models. |
| 75 | </details> |
| 76 | |
| 77 | |
| 78 | <details> |
| 79 | <summary>Cache instructions to save time</summary> |
| 80 | |
| 81 | Many prompts start with the same set of instructions, e.g., a system prompt plus a |
| 82 | handful of example input-output pairs. Instead of repeatedly running the model on common |
| 83 | instructions, cache them so that future computations are faster. |
| 84 | |
| 85 | Here's an |
| 86 | example using |
| 87 | [`cappr.huggingface.classify.cache_model`](https://cappr.readthedocs.io/en/latest/cappr.huggingface.classify.html#cappr.huggingface.classify.cache_model). |
| 88 | |
| 89 | ```python |
| 90 | from transformers import AutoModelForCausalLM, AutoTokenizer |
| 91 | from cappr.huggingface.classify import cache_model, predict |
| 92 | |
| 93 | # Load model and tokenizer |
| 94 | model = AutoModelForCausalLM.from_pretrained("gpt2") |
| 95 | tokenizer = AutoTokenizer.from_pretrained("gpt2") |
| 96 | model_and_tokenizer = (model, tokenizer) |
| 97 | |
| 98 | # Create data |
| 99 | prompt_prefix = '''Instructions: complete the sequence. |
| 100 | Here are examples: |
| 101 | A, B, C => D |
| 102 | 1, 2, 3 => 4 |
| 103 | |
| 104 | Complete this sequence:''' |
| 105 | |
| 106 | prompts = ["X, Y =>", "10, 9, 8 =>"] |
| 107 | completions = ["7", "Z", "Hi"] |
| 108 | |
| 109 | # Cache prompt_prefix because it's used for all prompts |
| 110 | cached_model_and_tokenizer = cache_model( |
| 111 | model_and_tokenizer, prompt_prefix |
| 112 | ) |
| 113 | |
| 114 | # Compute |
| 115 | preds = predict( |
| 116 | prompts, completions, cached_m |
People who liked this also viewed
6 picksType
- components-helperBased on the docs to provide code prompt files for vue component libraryPromptsNov 2023
- VibeCode-Protocol-SuiteA comprehensive suite of protocols, meta-prompts, and orchestration tools designed to streamline software development workflows, project management, and team collaboration. Includes the VibeCode Protocol Suite with web interface, user manuals, and reusable templates for efficient development processes.PromptsJul 2026
- Promptable-web-sdkWeb SDK for Promptable Website.PromptsDec 2022
- LLMFlexA python package for developing AI applications with local LLMs.PromptsJan 2025
- cli-repo-to-promptexport any codebase to a single LLM-ready markdown promptPromptsJul 2026
- Files2PromptVS Code extension to copy files for an LLM promptPromptsApr 2025
