tokuin
starsforks
How to install tokuin?
nooscraft/tokuin
$git clone https://github.com/nooscraft/tokuinA fast, CLI tool to estimate token usage, control API costs, and compress prompts for LLM providers. Built in Rust for performance, portability, and safety.
README.md
| 1 | # 🧮 Tokuin |
| 2 | |
| 3 | [](https://github.com/nooscraft/tokuin) |
| 4 | [](https://www.rust-lang.org/) |
| 5 | [](https://github.com/nooscraft/tokuin/actions) |
| 6 | |
| 7 | A fast, CLI tool to **estimate token usage**, **control API costs**, and **compress prompts** for LLM providers. Built in Rust for performance, portability, and safety. |
| 8 | |
| 9 | ## ✨ What's in v0.2.0 |
| 10 | |
| 11 | - **Prompt Compression** — Reduce prompt tokens by 70-90% using the [Hieratic format](#️-prompt-compression) |
| 12 | - **Quality Metrics** — Measure compression fidelity (semantic similarity, critical instruction preservation, structural integrity) |
| 13 | - **LLM-as-a-Judge** — Compare outputs from original vs compressed prompts using an LLM judge |
| 14 | - **Incremental Compression** — Compress long conversations turn-by-turn without re-processing history |
| 15 | - **Structured Mode** — Compression-aware handling of JSON, code blocks, HTML tables, and technical docs |
| 16 | - Everything from v0.1.x: token counting, cost estimation, multi-model comparison, load testing |
| 17 | |
| 18 | --- |
| 19 | |
| 20 | ## 🚀 Installation |
| 21 | |
| 22 | ### Quick Install (macOS & Linux) |
| 23 | |
| 24 | ```bash |
| 25 | curl -fsSL https://raw.githubusercontent.com/nooscraft/tokuin/main/install.sh | bash |
| 26 | ``` |
| 27 | |
| 28 | Detects your platform, downloads the latest release, verifies its checksum, and installs `tokuin` to `/usr/local/bin` (or `~/.local/bin` if root access is unavailable). |
| 29 | |
| 30 | ### What's included per platform |
| 31 | |
| 32 | Release binaries are built with `--features all`. Embedding-based semantic scoring (`compression-embeddings`) requires the ONNX Runtime, which only has compatible prebuilt binaries for some platforms: |
| 33 | |
| 34 | | Platform | Token counting | Compression | Embedding scoring | LLM judge | |
| 35 | |---|---|---|---|---| |
| 36 | | Linux x86_64 | ✅ | ✅ | ✅ | ✅ | |
| 37 | | macOS Apple Silicon (aarch64) | ✅ | ✅ | ✅ | ✅ | |
| 38 | | macOS Intel (x86_64) | ✅ | ✅ | ❌ heuristic only | ✅ | |
| 39 | | Linux aarch64 | ✅ | ✅ | ❌ heuristic only | ✅ | |
| 40 | | Windows x86_64 | ✅ | ✅ | ❌ heuristic only | ✅ | |
| 41 | |
| 42 | Platforms without embedding scoring still have full compression — quality metrics use heuristic scoring instead of embedding-based semantic similarity. See [Platform Support for Embedding Model Loading](#platform-support-for-embedding-model-loading) for the technical reasons and how to get full support on any platform. |
| 43 | |
| 44 | ### Quick Install (Windows PowerShell) |
| 45 | |
| 46 | ```powershell |
| 47 | irm https://raw.githubusercontent.com/nooscraft/tokuin/main/install.ps1 | iex |
| 48 | ``` |
| 49 | |
| 50 | Binary is placed in `%LOCALAPPDATA%\Programs\tokuin`. To customize: download the script first and invoke `.\install.ps1 -InstallDir "C:\Tools"`. |
| 51 | |
| 52 | ### From Releases |
| 53 | |
| 54 | Release archives for all platforms are at [GitHub Releases](https://github.com/nooscraft/tokuin/releases). Download the archive for your OS/architecture, verify against `checksums.txt`, and place `tokuin` on your `PATH`. |
| 55 | |
| 56 | ### From Source |
| 57 | |
| 58 | **Minimal build (token counting only):** |
| 59 | ```bash |
| 60 | git clone https://github.com/nooscraft/tokuin.git |
| 61 | cd tokuin |
| 62 | cargo build --release |
| 63 | ``` |
| 64 | |
| 65 | **Build with all features:** |
| 66 | ```bash |
| 67 | cargo build --release --features all |
| 68 | ``` |
| 69 | |
| 70 | **Build with embedding support (semantic scoring):** |
| 71 | ```bash |
| 72 | cargo build --release --features all,compression-embeddings |
| 73 | # Then set up models: |
| 74 | ./target/release/tokuin setup models |
| 75 | ``` |
| 76 | |
| 77 | --- |
| 78 | |
| 79 | ## 📖 Usage |
| 80 | |
| 81 | ### Token Counting |
| 82 | |
| 83 | ```bash |
| 84 | echo "Hello, world!" | tokuin --model gpt-4 |
| 85 | tokuin prompt.txt --model gpt-4 --price |
| 86 | ``` |
| 87 | |
| 88 | ### Multi-Model Comparison |
| 89 | |
| 90 | ```bash |
| 91 | echo "Hello, world!" | tokuin --compare gpt-4 gpt-3.5-turbo --price |
| 92 | ``` |
| 93 | |
| 94 | ### JSON Output / Breakdown |
| 95 | |
| 96 | ```bash |
| 97 | echo "Hello, world!" | tokuin --model gpt-4 --format json |
| 98 | echo '[{"role":"system","content":"You are helpful"},{"role":"user","content":"Hi!"}]' | \ |
| 99 | tokuin --model gpt-4 --breakdown --price |
| 100 | ``` |
| 101 | |
| 102 | ### Diff & Watch |
| 103 | |
| 104 | ```bash |
| 105 | tokuin prompt.txt --model gpt-4 --diff prompt-v2.txt --price |
| 106 | tokuin prompt.txt --model gpt-4 --watch |
| 107 | ``` |
| 108 | |
| 109 | --- |
| 110 | |
| 111 | ## 🗜️ Prompt Compression |
| 112 | |
| 113 | Tok |
People who liked this also viewed
6 picksType
- bun-promptxTerminal prompt library for the Bun runtimePromptsSep 2023
- lintlanglintlang is a static linter for AI agent configs, tool descriptions, and system prompts that runs zero-LLM quality gating in CI. Catches language-level failures (vague tool descriptions, missing stop conditions, schema gaps) before they reach runtime, with deterministic regex + structural detectors and no model calls.PromptsJul 2026
- gpt-cost-estimatorA cost estimator for OpenAI API calls in tqdm loops.PromptsNov 2024
- shadowquillappShadowQuill is a privacy‑first, offline and free prompt engineering/generation app powered by self-hosted local LLMsPromptsJul 2026
- prompt-patternsPrompt 编写模式：如何将思维框架赋予机器，以设计模式的形式来思考 promptPromptsMar 2023
- OpenPipeTurn expensive prompts into cheap fine-tuned modelsPromptsMay 2024
