OpenRLHF

OpenRLHF is an open-source reinforcement learning from human feedback framework that combines Ray and vLLM for scalable distributed training.

OpenRLHF is the first high-performance, production-ready open-source reinforcement learning from human feedback framework that combines Ray and vLLM distributed architecture with a unified agent-based design paradigm. It is built for researchers and developers seeking scalable and extensible reinforcement learning. The platform separates actor, reward, reference, and critic models across different GPUs using Ray and utilizes vLLM for high-throughput inference and DeepSpeed for memory-efficient training. It supports various algorithms such as PPO, REINFORCE++, GRPO, and RLOO, along with multi-turn vision-language model reinforcement learning. Users can execute single-turn or multi-turn agent pipelines with HuggingFace models.

OpenRLHF at a glance

Best for: Software engineer, Researcher
Runs on: Linux, open source, self-hosted
Pricing: Free
Released: July 2023
Category: AI coding tool
Website: github.com/OpenRLHF/OpenRLHF

Key features

Ray distributed scheduler and controller
vLLM high-performance inference engine
DeepSpeed ZeRO-3 memory-efficient training
Single-turn and multi-turn agent execution modes
Support for PPO, REINFORCE++, GRPO, and RLOO algorithms
Multi-turn vision-language model reinforcement learning
HuggingFace Transformers model interface

Use it for

Train large language models with reinforcement learning
Execute multi-step interactive reasoning tasks
Fine-tune vision-language models with image inputs
Run distributed model training across multiple GPUs

Alternatives to OpenRLHF

RLinf — Scalable open-source framework supporting real-world robots and simulators.
rLLM — Enables reinforcement learning training for AI agents with minimal code changes.
ART — Enhances LLM agent performance and reliability using reinforcement learning techniques.
Agent Lightning — Train existing artificial intelligence agents with reinforcement learning and zero code changes.
OpenVLHarness — Lifts six multimodal model backbones by 9.7 to 12.6 points across 22 benchmarks.
AReaL — Bridges foundation model training with modern agent-based applications.
Unsloth — Runs and trains models locally with lower memory usage.
AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
XTuner — High-efficiency training for 200B+ scale MoE models without expert parallelism.
EasyR1 — It supports vision-language models and high-performance RL training algorithms.
Karotte — It runs tasks in isolated virtual machines for safety and consistency.
Stable Baselines3 — Provides reliable, standard PyTorch implementations for reinforcement learning.

All OpenRLHF alternatives