huggingface-tokenizers
byfirecrawl· 131 skills
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
How to install huggingface-tokenizers?
firecrawl/ai-research-skills/huggingface-tokenizers
$npx -y skills add firecrawl/ai-research-skills --skill huggingface-tokenizersInstalls into the current project.
›Prefer a prompt? Paste this to your agent
Run `npx skills use "https://github.com/firecrawl/ai-research-skills" --skill "firecrawl/ai-research-skills/huggingface-tokenizers"` and follow the generated skill instructions now. Read its complete output, redirecting it to a temporary file first if necessary. Resolve relative paths from the supporting-files directory it provides.
Use the skills in "https://github.com/firecrawl/ai-research-skills" that are relevant to the current task. Run `npx skills add "https://github.com/firecrawl/ai-research-skills"` and select the relevant skills, then follow their instructions.
Related
6 picksType
- azure-aiUse for Azure AI: Search, Speech, OpenAI, Document Intelligence. Helps with search, vector/hybrid search, speech-to-text, text-to-speech, transcription, OCR.Jul 2026
- ai-research-exploreRigor Explore compatible skill slug for meaningful and potentially novel deep learning research candidates.Jul 2026
- ai-research-reproductionRigor Reproduce compatible skill slug for README-first deep learning repository reproduction.Jul 2026
- explore-codeRigor Improve implementation leaf skill for auditable candidate implementation in deep learning research repositories.Jul 2026
- run-trainRigor Train skill for deep learning research repositories. Use when a documented or selected training command should be run conservatively for startup…Jul 2026
- explore-runRigor Improve / Rigor Explore run leaf skill for bounded exploratory evidence in deep learning research repositories.Jul 2026
