Large-scale online reinforcement learning is becoming a major way to improve reasoning, code generation, and tool use in large language models. Its cost, however, rises quickly as training expands across domains. Every policy update requires fresh sampling, generation remains tightly coupled to backpropagation, and expensive expert experience is often set aside once a single training run ends.
REGEN, short for Replay-recycling for Expert-to-Generalist Distillation with Offline Reinforcement Learning, asks a direct question: if domain experts have already produced extensive replay data during reinforcement learning, why should a generalist explore the same ground online once again? Rather than repeatedly querying teachers for new trajectories, REGEN recycles the replay memory naturally produced while training specialists and uses offline reinforcement learning to consolidate their capabilities into one generalist model.
From domain experts to a unified generalist
Specialized models often perform better within individual domains such as mathematical reasoning, code generation, and instruction following, but deploying and maintaining many experts is rarely ideal. Multi-teacher on-policy distillation (MOPD) offers one solution: a student generates trajectories from its own policy, and a teacher for the corresponding domain supplies the learning signal.
MOPD can integrate expert capabilities effectively, but it still depends on online rollouts. Student generation, teacher inference, and parameter updates must operate together, so inference cannot be fully separated from backpropagation. The more tasks, teachers, and training stages a system contains, the greater the computational and engineering burden of this coupling becomes.
REGEN instead reframes expert reinforcement learning as a form of data synthesis. The trajectories produced during specialist training are not disposable intermediate artifacts. They are experience assets containing exploration, successful attempts, and failure feedback. Once distribution shift and data imbalance are handled, the same experience can train new generalist policies repeatedly.
Recycle replay memory instead of regenerating experience
REGEN compared with multi-teacher on-policy distillation. REGEN aggregates replay memory from specialist training and learns a generalist through class-balanced sampling and offline policy optimization with importance sampling. MOPD repeatedly performs on-policy rollouts and teacher distillation.
As the figure shows, REGEN consists of three key stages:
- Replay-memory recycling. Reinforcement learning for each domain expert leaves behind replay memory. REGEN aggregates these existing trajectories instead of repeating expensive online sampling for generalist training.
- Class-balanced sampling. Positive- and negative-reward examples are often highly imbalanced in replay data. REGEN retains valid queries containing both positive and negative responses, then balances the two classes so that one outcome does not dominate the learning signal.
- Offline policy optimization. Historical trajectories come from specialists trained at different stages and in different domains, creating substantial distribution shift from the current generalist. REGEN uses offline reinforcement learning with truncated importance sampling to retain useful learning signals while controlling the variance of off-policy updates.
Because rollouts already exist, the training system only needs to read data and perform backward updates. Sampling and learning become fully decoupled. The same expert experience can be reused across models, training recipes, and later stages of post-training.
Lower cost without giving up generality
REGEN is evaluated across mathematical reasoning, code generation, and instruction following. It matches the accuracy of MOPD at substantially lower training cost. The significance extends beyond making one distillation run cheaper: data that once served a single expert-training run can continue to support generalist training and subsequent iterations.
This design also makes the system easier to extend. When a new domain is added, a specialist can be trained independently and its replay data merged into the existing experience pool. There is no need to launch every teacher simultaneously for another round of joint online distillation. Expert experience produced by different teams and at different times can also be reorganized in one offline training stage.
From one-off reinforcement learning to reusable experience infrastructure
LLM post-training often treats online reinforcement learning as an expensive but temporary phase: when training ends, the value of its rollouts ends with it. REGEN demonstrates another possibility by treating replay memory as infrastructure that can be accumulated, filtered, and reused. Online reinforcement learning explores and produces high-value experience; offline reinforcement learning disseminates and consolidates capabilities at lower cost.
This does not mean online reinforcement learning can be removed entirely. REGEN still depends on capable specialists and sufficiently broad experience, while a static dataset cannot automatically cover behavior that has never been explored. When multi-domain experts already exist, however, REGEN turns “train it again” into “reuse the experience we have already paid for,” providing a more sustainable path toward large-scale, multi-stage model post-training.