Overview
nemo-curator is an open-source data skill for Claude Code and compatible agents, published by Orchestra-Research. Its author describes it as: “GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Sc…”. The project has 12k stars on GitHub and is available under the MIT license. Add it to your setup with `/plugin marketplace add Orchestra-Research/AI-Research-SKILLs`.
What nemo-curator does
NVIDIA's toolkit for preparing high-quality training data for LLMs.
Best for
What’s inside
The SKILL.md for nemo-curator is organised into these sections:
- When to use NeMo Curator
- Quick start
- Installation
- Basic text curation pipeline
- Data curation pipeline
- Stage 1: Quality filtering
- Stage 2: Deduplication
- Stage 3: PII redaction
- Stage 4: Classifier filtering
- GPU acceleration
- GPU vs CPU performance
- Multi-GPU scaling
Installation
Add nemo-curator to your agent with:
/plugin marketplace add Orchestra-Research/AI-Research-SKILLsReview a skill’s source before installing it — skills can instruct your agent to run commands. This command comes from the skill’s public repository, which is the source of truth for setup steps. New to skills? Read how to install Claude Code skills.
Supported agents
- Claude Code — ships a claude code plugin marketplace
- Codex — declared in the repository topics
License
nemo-curator is published under the MIT license.
Strengths
- Clear MIT license — safe to read and adapt within its terms
- Ships in Orchestra-Research/AI-Research-SKILLs, an established project with 11,807 GitHub stars
- Works with Claude Code, Codex
Author
Topics
Frequently asked questions
- What does nemo-curator do?
- GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.
- How do I install nemo-curator?
- Run /plugin marketplace add Orchestra-Research/AI-Research-SKILLs in your agent, then reload your skills. Review the source at https://github.com/Orchestra-Research/AI-Research-SKILLs before installing.
- Is nemo-curator free to use?
- Yes. nemo-curator is free and open source under the MIT license, so you can read, run, and adapt it within that license's terms.
- Which agents does nemo-curator work with?
- nemo-curator lists Claude Code and Codex — named in its SKILL.md, its plugin marketplace or its repository topics. Skills use the portable SKILL.md format, so other agents that read that format may load it too.
- Where does nemo-curator come from?
- nemo-curator ships inside Orchestra-Research/AI-Research-SKILLs, a repository that contains 41 catalogued skills in total. The repository's 11,807 GitHub stars apply to that whole collection, not to this skill on its own.
