We propose QeRL, a Quantization-enhanced Reinforcement Learning framework for large language models (LLMs). While RL is essential for LLMs' reasoning capabilities, it is resource-intensive, requiring substantial GPU memory and long rollout durations. QeRL addresses these issues by combining NVFP4 quantization with Low-Rank Adaptation (LoRA), accelerating rollout phase of RL while reducing memory overhead. Beyond efficiency, our findings show that quantization noise increases policy entropy, enhancing exploration, and enabling the discovery of better strategies during RL. To further optimize exploration, QeRL introduces an Adaptive Quantization Noise (AQN) mechanism, which dynamically adjusts noise during training. Experiments demonstrate that QeRL delivers over 1.5 times speedup in the rollout phase. Moreover, this is the first framework to enable RL training of a 32B LLM on a single H100 80GB GPU, while delivering overall speedups for RL training. It also achieves faster reward growth and higher final accuracy than 16-bit LoRA and QLoRA, while matching the performance of full-parameter fine-tuning on mathematical benchmarks such as GSM8K (90.8%) and MATH 500 (77.4%) in the 7B model. These results establish QeRL as an efficient and effective framework for RL training in LLMs.
QeRL achieves faster RL rollout and end-to-end training speeds, while delivering performance superior to vanilla LoRA and QLoRA, also comparable to full-parameter RL on mathematical benchmarks.
Its dual benefits in memory efficiency and training speed make QeRL highly effective for end-to-end RL workflows, especially in scenarios requiring extensive rollouts. QeRL shows rollout performance across various LoRA ranks, with QeRL achieving over 2× speedups on14B and 32B models.
NVFP4 quantization with LoRA, reducing memory and enabling faster RL while matching full-parameter finetuning performance with adaptive quantization noise. AQN dynamically adjusts quantization noise with an exponential scheduler, enhancing exploration.
Quantization Helps Exploration:
Multi-step reasoning is essential for LLMs to solve complex tasks, yet prevailing training paradigms face key limitations. In RL, large model sizes inflate memory use and a multistage pipeline—spanning rollouts, reward computation, logits-evaluation, and gradient updates—slows training, with long-sequence rollouts being especially expensive. Our experiments indicate a promising remedy:quantized LLMs combined with LoRA not only reduce training resources but also outperform vanilla LoRA in reward growth and evaluation, challenging the conventional view from SFT that quantization degrades training. We further observe that quantization error behaves like beneficial parameter noise, increasing entropy and broadening exploration. These findings motivate leveraging low-bit quantization in RL to improve both efficiency and reasoning quality.
Adaptive Quantization Noise(AQN)
Specifically, for each quantized linear layer, we sample a stochastic noise vector, This vector is not fixed but is resampled for each forward pass. The additive noise is defined as:
In the early stages, we leverage the inherent quantization noise to enhance the model's exploration capabilities. As training progresses, noise gradually reduces following an exponential decay scheduler:
Noise Merging for Injecting Noise into Low-precision Weight
We propose a simple solution that integrates this noise vector directly into the layer normalization parameters of LLM architectures. When westart with the default quantized noise and periodically inject additional noise in later stages, the reward curve grows more steadily.
Training Reward of7B/14B-Instruct Models on BigMath dataset (level 3~5),32B-Instruct Model on BigMath dataset (level 4~5).
QeRL achieves a rapid reward increase within200 steps (left), while vanilla LoRA requires over 500 steps to show improvement. In the hardest training data for 32B (right), QeRL still show higher accuracy reward curve than vanilla LoRA.
Reward visualization on Different RL Algorithms.
In both GRPO and DAPO, quantized model delivers better converged reward score and faster reward growth.
Comparison with Relevant Baselines on GSM8K.
We report the GSM8k training results of the 3B and 7B models using GRPO.
Comparison with Relevant Baselines on More Benchmarks.
We report the BigMath(level 3~5) training results of other models using DAPO.
Memory Saving and End-to-End Training Speedup on 7B and 14B Models.
We compare the quantized model sizes and end-to-end RL training speedup of these PEFT methods, with all experiments conducted on a single NVIDIA H100-80GB GPU.
Memory Saving and End-to-End Training Speedup on 32B Model.
We compare the throughput of rollout phase and the final end-to-end training speedup of 32B model.
Ablation of Learning Rate.
When the learning rate is increased to 3e-5, the larger update magnitude in the adapter results in faster reward growth and quicker model convergence. However, in 16-bit models, the excessive update magnitude leads to instability, often causing the training process to collapse. In contrast, QeRL demonstrates remarkable robustness to larger learning rates due to the presence of NVFP4 quantization noise, which helps stabilize updates.
@article{
huang2025qerl,
title={QeRL: Beyond Efficiency--Quantization-enhanced Reinforcement Learning for LLMs},
author={Huang, Wei and Ge, Yi and Yang, Shuai and Xiao, Yicheng and Mao, Huizi and Lin, Yujun and Ye, Hanrong and Liu, Sifei and Cheung, Ka Chun and Yin, Hongxu and others},
journal={arXiv preprint arXiv:2510.11696},
year={2025}
}
