Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
How to install sentencepiece?
firecrawl/ai-research-skills/sentencepiece
$npx -y skills add firecrawl/ai-research-skills --skill sentencepieceInstalls into the current project.
›Prefer a prompt? Paste this to your agent
Run `npx skills use "https://github.com/firecrawl/ai-research-skills" --skill "firecrawl/ai-research-skills/sentencepiece"` and follow the generated skill instructions now. Read its complete output, redirecting it to a temporary file first if necessary. Resolve relative paths from the supporting-files directory it provides.
Use the skills in "https://github.com/firecrawl/ai-research-skills" that are relevant to the current task. Run `npx skills add "https://github.com/firecrawl/ai-research-skills"` and select the relevant skills, then follow their instructions.
Related
6 picksType
- azure-messagingTroubleshoot and resolve issues with Azure Messaging SDKs for Event Hubs and Service Bus.Jul 2026
- lark-openapi-explorer飞书/Lark 原生 OpenAPI 探索：从官方文档库中挖掘未经 CLI 封装的原生 OpenAPI 接口。当用户的需求无法被现有 lark-* skill 或 lark-cli 已注册命令满足，需要查找并调用原生飞书 OpenAPI 时使用。Jul 2026
- lark-skill-maker创建 lark-cli 的自定义 Skill。当用户需要把飞书 API 操作封装成可复用的 Skill（包装原子 API 或编排多步流程）时使用。Jul 2026
- implementImplement a piece of work based on a spec or set of tickets.Jul 2026
- supabaseUse when doing ANY task involving Supabase. Triggers: Supabase products (Database, Auth, Edge Functions, Realtime, Storage, Vectors, Cron, Queues); client…Jul 2026
- firebase-basicsProvides foundational setup, authentication, and project management workflows for Firebase using the Firebase CLI.Jul 2026
