Overview
distributed-llm-pretraining-torchtitan is an open-source workflow skill for Claude Code and compatible agents, published by Orchestra-Research. Its author describes it as: “Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, tor…”. The project has 12k stars on GitHub and is available under the MIT license. Add it to your setup with `/plugin marketplace add Orchestra-Research/AI-Research-SKILLs`.
What distributed-llm-pretraining-torchtitan does
TorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs.
What’s inside
The SKILL.md for distributed-llm-pretraining-torchtitan is organised into these sections:
- Quick start
- Common workflows
- Workflow 1: Pretrain Llama 3.1 8B on single node
- Workflow 2: Multi-node training with SLURM
- Workflow 3: Enable Float8 training for H100s
- Workflow 4: 4D parallelism for 405B models
- When to use vs alternatives
- Common issues
- Supported models
- Performance benchmarks (H100)
- Advanced topics
- Resources
Installation
Add distributed-llm-pretraining-torchtitan to your agent with:
/plugin marketplace add Orchestra-Research/AI-Research-SKILLsReview a skill’s source before installing it — skills can instruct your agent to run commands. This command comes from the skill’s public repository, which is the source of truth for setup steps. New to skills? Read how to install Claude Code skills.
Supported agents
- Claude Code — ships a claude code plugin marketplace
- Codex — declared in the repository topics
License
distributed-llm-pretraining-torchtitan is published under the MIT license.
Strengths
- Clear MIT license — safe to read and adapt within its terms
- Ships in Orchestra-Research/AI-Research-SKILLs, an established project with 11,807 GitHub stars
- Works with Claude Code, Codex
Author
Topics
Frequently asked questions
- What does distributed-llm-pretraining-torchtitan do?
- Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.
- How do I install distributed-llm-pretraining-torchtitan?
- Run /plugin marketplace add Orchestra-Research/AI-Research-SKILLs in your agent, then reload your skills. Review the source at https://github.com/Orchestra-Research/AI-Research-SKILLs before installing.
- Is distributed-llm-pretraining-torchtitan free to use?
- Yes. distributed-llm-pretraining-torchtitan is free and open source under the MIT license, so you can read, run, and adapt it within that license's terms.
- Which agents does distributed-llm-pretraining-torchtitan work with?
- distributed-llm-pretraining-torchtitan lists Claude Code and Codex — named in its SKILL.md, its plugin marketplace or its repository topics. Skills use the portable SKILL.md format, so other agents that read that format may load it too.
- Where does distributed-llm-pretraining-torchtitan come from?
- distributed-llm-pretraining-torchtitan ships inside Orchestra-Research/AI-Research-SKILLs, a repository that contains 41 catalogued skills in total. The repository's 11,807 GitHub stars apply to that whole collection, not to this skill on its own.