ctoc
starsforks
How to install ctoc?
yaohui-wyh/ctoc
$git clone https://github.com/yaohui-wyh/ctoc_Count Tokens of Code_.
README.md
| 1 | # ctoc |
| 2 | |
| 3 | _Count Tokens of Code_. |
| 4 | |
| 5 | **Token counts** play a key role in shaping a Large Language Model's (LLM) memory and conversation history. They're vital for prompt engineering and token cost estimation. Various strategies in prompt engineering (e.g., contextual filtering and reranking) predominantly aim at token compression to counteract LLM's context size limit. |
| 6 | |
| 7 | **ctoc** provides a lightweight tool for analyzing codebases at the token level. It incorporates all the features of [cloc](https://github.com/AlDanial/cloc). (You can use `ctoc` in a `cloc`-consistent manner.) |
| 8 | |
| 9 | Built on top of [gocloc](https://github.com/hhatto/gocloc), ctoc is extremely fast. |
| 10 | |
| 11 | [](https://godoc.org/github.com/yaohui-wyh/ctoc) |
| 12 | [](https://github.com/yaohui-wyh/ctoc/actions) |
| 13 | [](https://goreportcard.com/report/github.com/yaohui-wyh/ctoc) |
| 14 | |
| 15 | <details> |
| 16 | <summary>What are <b>Tokens</b>? (in the context of Large Language Model)</summary> |
| 17 | |
| 18 | - **Tokens**: basic units of text/code for LLM AI models to process/generate language. |
| 19 | - **Tokenization**: splitting input/output texts into smaller units for LLM AI models. |
| 20 | - **Vocabulary size**: the number of tokens each model uses, which varies among different GPT models. |
| 21 | - **Tokenization cost**: affects the memory and computational resources that a model needs, which influences the cost and performance of running an OpenAI or Azure OpenAI model. |
| 22 | |
| 23 | refs: https://learn.microsoft.com/en-us/semantic-kernel/prompt-engineering/tokens |
| 24 | |
| 25 | </details> |
| 26 | |
| 27 | |
| 28 | ## Installation |
| 29 | |
| 30 | Install from GitHub release: |
| 31 | ```bash |
| 32 | curl -sL "https://github.com/yaohui-wyh/ctoc/releases/download/v0.0.1/ctoc_$(uname)_$(uname -m).tar.gz" | tar xz && chmod +x ctoc && ctoc -h |
| 33 | ``` |
| 34 | |
| 35 | Alternatively, you can install via `go install` (requires Go 1.19+): |
| 36 | |
| 37 | ```bash |
| 38 | go install github.com/yaohui-wyh/ctoc/cmd/ctoc@latest |
| 39 | ``` |
| 40 | |
| 41 | ## Usage |
| 42 | |
| 43 | ### Basic Usage |
| 44 | |
| 45 | ``` |
| 46 | $ ctoc -h |
| 47 | Usage: |
| 48 | ctoc [OPTIONS] |
| 49 | |
| 50 | Application Options: |
| 51 | --by-file report results for every encountered source file |
| 52 | --sort=[name|files|blank|comment|code|tokens] sort based on a certain column (default: code) |
| 53 | --output-type= output type [values: default,cloc-xml,sloccount,json] (default: default) |
| 54 | --exclude-ext= exclude file name extensions (separated commas) |
| 55 | --include-lang= include language name (separated commas) |
| 56 | --match= include file name (regex) |
| 57 | --not-match= exclude file name (regex) |
| 58 | --match-d= include dir name (regex) |
| 59 | --not-match-d= exclude dir name (regex) |
| 60 | --debug dump debug log for developer |
| 61 | --skip-duplicated skip duplicated files |
| 62 | --show-lang print about all languages and extensions |
| 63 | --version print version info |
| 64 | --show-encoding print about all LLM models and their corresponding encodings |
| 65 | --encoding=[cl100k_base|p50k_base|p50k_edit|r50k_base] specify tokenizer encoding (default: cl100k_base) |
| 66 | |
| 67 | Help Options: |
| 68 | -h, --help Show this help message |
| 69 | ``` |
| 70 | |
| 71 | ``` |
| 72 | $ ctoc . |
| 73 | ------------------------------------------------------------------------------------------------ |
| 74 | Language files blank comment code tokens |
| 75 | -------------------------------------------------------------------------------------------- |
People who liked this also viewed
6 picksType
- bun-promptxTerminal prompt library for the Bun runtimePromptsSep 2023
- lintlanglintlang is a static linter for AI agent configs, tool descriptions, and system prompts that runs zero-LLM quality gating in CI. Catches language-level failures (vague tool descriptions, missing stop conditions, schema gaps) before they reach runtime, with deterministic regex + structural detectors and no model calls.PromptsJul 2026
- gpt-cost-estimatorA cost estimator for OpenAI API calls in tqdm loops.PromptsNov 2024
- shadowquillappShadowQuill is a privacy‑first, offline and free prompt engineering/generation app powered by self-hosted local LLMsPromptsJul 2026
- prompt-patternsPrompt 编写模式：如何将思维框架赋予机器，以设计模式的形式来思考 promptPromptsMar 2023
- OpenPipeTurn expensive prompts into cheap fine-tuned modelsPromptsMay 2024
