XTuner

XTuner is an LLM training engine designed to facilitate the pre-training, instruction fine-tuning, and reinforcement learning of ultra-large MoE models.

XTuner V1 functions as a specialized training engine for ultra-large-scale Mixture-of-Experts (MoE) models. It optimizes parallel training architectures to handle models up to 1T parameters, surpassing traditional 3D parallel schemes in FSDP throughput for models above 200B scale. The engine supports long sequence training, enabling 200B MoE model training on 64k sequence lengths through memory optimization. It accommodates multimodal pre-training and supervised fine-tuning, alongside reinforcement learning techniques like GRPO. The framework integrates with various hardware, including NVIDIA GPUs and Ascend NPUs, and supports models such as Intern S1, Intern VL, and Qwen3.

XTuner at a glance

Best for: Software engineer, Researcher
Runs on: open source
Pricing: Free
Released: July 2023
Category: AI coding tool
Website: github.com/InternLM/xtuner

Key features

Dropless training for scalable MoE model development
Support for 64k sequence lengths in 200B MoE models
Integration with DeepSpeed Ulysses for sequence parallelism
Optimization for Ascend A3 Supernode hardware
Support for multimodal pre-training and supervised fine-tuning
Implementation of GRPO for reinforcement learning
Deployment compatibility with LMDeploy

Use it for

Pre-training ultra-large MoE models
Instruction fine-tuning for multimodal models
Reinforcement learning for large language models
Creating synthetic data for fine-tuning via GraphGen

Alternatives to XTuner

AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
LMDeploy — Accelerates inference with high-performance engines and quantization tools.
Strata — Runs large AI models locally on consumer gaming PCs without server reliance.
ART — Enhances LLM agent performance and reliability using reinforcement learning techniques.
rLLM — Enables reinforcement learning training for AI agents with minimal code changes.
PEFT — Reduces computational requirements and storage needs for large model training.
fal — Access 1,000+ generative models and high-performance serverless GPUs.
Soup — Fine-tune and align models like Llama-3.1-8B on a 4 GB laptop GPU.
ms-swift — Supports hundreds of text and multimodal models with a full training pipeline.
LlamaFactory — Provides a code-free CLI and Web UI for training 100+ different LLMs.
xTuring — Provides complete control over customizing open-source LLMs.
NeMo Molt — It scales fully-async agentic RL to frontier MoE scale with a small PyTorch stack.

All XTuner alternatives