Overview
sentencepiece is an open-source productivity skill for Claude Code and compatible agents, published by Orchestra-Research. Its author describes it as: “Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train o…”. The project has 12k stars on GitHub and is available under the MIT license. Add it to your setup with `/plugin marketplace add Orchestra-Research/AI-Research-SKILLs`.
What sentencepiece does
Unsupervised tokenizer that works on raw text without language-specific preprocessing.
What’s inside
The SKILL.md for sentencepiece is organised into these sections:
- When to use SentencePiece
- Quick start
- Installation
- Train model
- Encode and decode
- Language-independent design
- Whitespace as symbol (▁)
- Tokenization algorithms
- BPE (Byte-Pair Encoding)
- Unigram (default)
- Training configuration
- Essential parameters
Installation
Add sentencepiece to your agent with:
/plugin marketplace add Orchestra-Research/AI-Research-SKILLsReview a skill’s source before installing it — skills can instruct your agent to run commands. This command comes from the skill’s public repository, which is the source of truth for setup steps. New to skills? Read how to install Claude Code skills.
Supported agents
- Claude Code — ships a claude code plugin marketplace
- Codex — declared in the repository topics
License
sentencepiece is published under the MIT license.
Strengths
- Clear MIT license — safe to read and adapt within its terms
- Ships in Orchestra-Research/AI-Research-SKILLs, an established project with 11,807 GitHub stars
- Works with Claude Code, Codex
Author
Topics
Frequently asked questions
- What does sentencepiece do?
- Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
- How do I install sentencepiece?
- Run /plugin marketplace add Orchestra-Research/AI-Research-SKILLs in your agent, then reload your skills. Review the source at https://github.com/Orchestra-Research/AI-Research-SKILLs before installing.
- Is sentencepiece free to use?
- Yes. sentencepiece is free and open source under the MIT license, so you can read, run, and adapt it within that license's terms.
- Which agents does sentencepiece work with?
- sentencepiece lists Claude Code and Codex — named in its SKILL.md, its plugin marketplace or its repository topics. Skills use the portable SKILL.md format, so other agents that read that format may load it too.
- Where does sentencepiece come from?
- sentencepiece ships inside Orchestra-Research/AI-Research-SKILLs, a repository that contains 41 catalogued skills in total. The repository's 11,807 GitHub stars apply to that whole collection, not to this skill on its own.