tiktoken
installs/mo · pypistarsforks
How to install tiktoken?
tiktoken
$pip install tiktokenOpenAI's fast BPE tokenizer. Count tokens exactly before you spend them, cost planning, context budgeting, chunking. A five-line dependency that pays for itself daily.
README.md
| 1 | # tiktoken |
| 2 | 3-6x faster than comparable tokenizers. |
| 3 | Exact token counts for cost planning. |
