Expert guidance for GRPO/RL fine-tuning with TRL for reasoning and task-specific model training
openrlhf-training
openrlhf-training is an open-source productivity skill for Claude Code and compatible agents, published by Orchestra-Research. Its author describes it as: “High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and G…”. The project has 12k stars on GitHub and is available under the MIT license. Add it to your setup with `/plugin marketplace add Orchestra-Research/AI-Research-SKILLs`.
What openrlhf-training does
OpenRLHF is a Ray-based RLHF framework optimized for distributed training with vLLM inference acceleration.
Installation
Add openrlhf-training to your agent with:
/plugin marketplace add Orchestra-Research/AI-Research-SKILLsAlways review a skill's source before installing it. This command comes from the skill's public repository; the linked repo is the source of truth for exact setup steps.
What's inside
The SKILL.md for openrlhf-training is organised into these sections:
- Quick start
- Common workflows
- Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)
- Workflow 2: GRPO training (no critic model needed)
- Workflow 3: DPO training (preference optimization)
- When to use vs alternatives
- Common issues
- Advanced topics
- Hardware requirements
- Resources
When to use it
Reach for openrlhf-training when you want productivity help from your agent without writing the same instructions every session. Load the skill and the agent picks it up automatically for relevant tasks.
Strengths
- Clear MIT license — safe to read and adapt
- Ships in Orchestra-Research/AI-Research-SKILLs, an established project with 11,807 GitHub stars
- Actively maintained (recent commits)
Topics
Frequently asked questions
- What does openrlhf-training do?
- High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.
- How do I install openrlhf-training?
- Run /plugin marketplace add Orchestra-Research/AI-Research-SKILLs in your agent, then reload your skills. Review the source at https://github.com/Orchestra-Research/AI-Research-SKILLs before installing.
- Is openrlhf-training free to use?
- Yes. openrlhf-training is free and open source under the MIT license, so you can read, run, and adapt it within that license's terms.
- Where does openrlhf-training come from?
- openrlhf-training ships inside Orchestra-Research/AI-Research-SKILLs, a repository that contains 41 catalogued skills in total. The repository's 11,807 GitHub stars apply to that whole collection, not to this skill on its own.
Related skills
More Productivity →Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.
Provides guidance for LLM post-training with RL using slime, a Megatron+SGLang framework. Use when training GLM models, implementing custom data generation workflows, or needing tight Megatron-LM integration for RL scaling.
Orchestrates end-to-end autonomous AI research projects using a two-loop architecture. The inner loop runs rapid experiment iterations with clear optimization targets. The outer loop synthesizes results, identifies patterns, and steers research direction. Routes to domain-specific skills for execution, supports continuous agent operation via Claude Code /loop and OpenClaw heartbeat, and produces research presentations and papers. Use when starting a research project, running autonomous experiments, or managing a multi-hypothesis research effort.
Provides guidance for interpreting and manipulating neural network internals using nnsight with optional NDIF remote execution. Use when needing to run interpretability experiments on massive models (70B+) without local GPU resources, or when working with any PyTorch architecture.