Expert guidance for distributed training with DeepSpeed - ZeRO optimization stages, pipeline parallelism, FP16/BF16/FP8, 1-bit Adam, sparse attention
training-llms-megatron
training-llms-megatron is an open-source workflow skill for Claude Code and compatible agents, published by Orchestra-Research. Its author describes it as: “Trains large language models (2B-462B parameters) using NVIDIA Megatron-Core with advanced parallelism strategies. Use when training models >1B parameters, need maximum GPU efficiency (47% MFU on H100), or require ten…”. The project has 12k stars on GitHub and is available under the MIT license. Add it to your setup with `/plugin marketplace add Orchestra-Research/AI-Research-SKILLs`.
What training-llms-megatron does
Megatron-Core trains LLMs from 2B to 462B parameters with up to 47% Model FLOP Utilization on H100 GPUs through advanced parallelism strategies.
Installation
Add training-llms-megatron to your agent with:
/plugin marketplace add Orchestra-Research/AI-Research-SKILLsAlways review a skill's source before installing it. This command comes from the skill's public repository; the linked repo is the source of truth for exact setup steps.
What's inside
The SKILL.md for training-llms-megatron is organised into these sections:
- Quick start
- Common workflows
- Workflow 1: Train LLaMA-style model with 3D parallelism
- Workflow 2: Configure Mixture of Experts (MoE) training
- Workflow 3: Optimize for maximum throughput
- When to use vs alternatives
- Common issues
- Advanced topics
- Hardware requirements
- Resources
When to use it
Reach for training-llms-megatron when you want workflow help from your agent without writing the same instructions every session. Load the skill and the agent picks it up automatically for relevant tasks.
Strengths
- Clear MIT license — safe to read and adapt
- Ships in Orchestra-Research/AI-Research-SKILLs, an established project with 11,807 GitHub stars
- Actively maintained (recent commits)
Topics
Frequently asked questions
- What does training-llms-megatron do?
- Trains large language models (2B-462B parameters) using NVIDIA Megatron-Core with advanced parallelism strategies. Use when training models >1B parameters, need maximum GPU efficiency (47% MFU on H100), or require tensor/pipeline/sequence/context/expert parallelism. Production-ready framework used for Nemotron, LLaMA, DeepSeek.
- How do I install training-llms-megatron?
- Run /plugin marketplace add Orchestra-Research/AI-Research-SKILLs in your agent, then reload your skills. Review the source at https://github.com/Orchestra-Research/AI-Research-SKILLs before installing.
- Is training-llms-megatron free to use?
- Yes. training-llms-megatron is free and open source under the MIT license, so you can read, run, and adapt it within that license's terms.
- Where does training-llms-megatron come from?
- training-llms-megatron ships inside Orchestra-Research/AI-Research-SKILLs, a repository that contains 41 catalogued skills in total. The repository's 11,807 GitHub stars apply to that whole collection, not to this skill on its own.
Related skills
More Workflow →Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.
Parameter-efficient fine-tuning for LLMs using LoRA, QLoRA, and 25+ methods. Use when fine-tuning large models (7B-70B) with limited GPU memory, when you need to train <1% of parameters with minimal accuracy loss, or for multi-adapter serving. HuggingFace's official library integrated with transformers ecosystem.
Distributed training orchestration across clusters. Scales PyTorch/TensorFlow/HuggingFace from laptop to 1000s of nodes. Built-in hyperparameter tuning with Ray Tune, fault tolerance, elastic scaling. Use when training massive models across multiple machines or running distributed hyperparameter sweeps.
Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss. Use when deploying large models (7B-70B) on limited GPU memory, when you need faster inference than GPTQ with better accuracy preservation, or for instruction-tuned and multimodal models. MLSys 2024 Best Paper Award winner.