Overview
huggingface-tokenizers is an open-source backend skill for Claude Code and compatible agents, published by Orchestra-Research. Its author describes it as: “Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle paddi…”. The project has 12k stars on GitHub and is available under the MIT license. Add it to your setup with `/plugin marketplace add Orchestra-Research/AI-Research-SKILLs`.
What huggingface-tokenizers does
Fast, production-ready tokenizers with Rust performance and Python ease-of-use.
What’s inside
The SKILL.md for huggingface-tokenizers is organised into these sections:
- When to use HuggingFace Tokenizers
- Quick start
- Installation
- Load pretrained tokenizer
- Train custom BPE tokenizer
- Batch encoding with padding
- Tokenization algorithms
- BPE (Byte-Pair Encoding)
- WordPiece
- Unigram
- Tokenization pipeline
- Normalization
Installation
Add huggingface-tokenizers to your agent with:
/plugin marketplace add Orchestra-Research/AI-Research-SKILLsReview a skill’s source before installing it — skills can instruct your agent to run commands. This command comes from the skill’s public repository, which is the source of truth for setup steps. New to skills? Read how to install Claude Code skills.
Supported agents
- Claude Code — ships a claude code plugin marketplace
- Codex — declared in the repository topics
License
huggingface-tokenizers is published under the MIT license.
Strengths
- Clear MIT license — safe to read and adapt within its terms
- Ships in Orchestra-Research/AI-Research-SKILLs, an established project with 11,807 GitHub stars
- Works with Claude Code, Codex
Author
Topics
Frequently asked questions
- What does huggingface-tokenizers do?
- Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
- How do I install huggingface-tokenizers?
- Run /plugin marketplace add Orchestra-Research/AI-Research-SKILLs in your agent, then reload your skills. Review the source at https://github.com/Orchestra-Research/AI-Research-SKILLs before installing.
- Is huggingface-tokenizers free to use?
- Yes. huggingface-tokenizers is free and open source under the MIT license, so you can read, run, and adapt it within that license's terms.
- Which agents does huggingface-tokenizers work with?
- huggingface-tokenizers lists Claude Code and Codex — named in its SKILL.md, its plugin marketplace or its repository topics. Skills use the portable SKILL.md format, so other agents that read that format may load it too.
- Where does huggingface-tokenizers come from?
- huggingface-tokenizers ships inside Orchestra-Research/AI-Research-SKILLs, a repository that contains 41 catalogued skills in total. The repository's 11,807 GitHub stars apply to that whole collection, not to this skill on its own.
