How Can Coding-Agent RL Generalize Across Harnesses?

Why does the harness matter?

Coding agents solve tasks through ongoing interaction. A model inspects a repository, edits files, runs tests, and uses feedback to decide its next action. Reinforcement learning trains the model on these task attempts, using their outcomes as a learning signal.

The harness shapes the experience the model learns from. OpenHands SDK, Claude Code, and OpenCode define different tool interfaces, execution feedback, context management, and interaction loops. During RL, these choices shape the trajectories used to train the model. At deployment, the harness exposes tools for reading files, editing code, and running tests. It also controls how much interaction history is retained and how tool outputs are presented to the model.

This raises a practical question: how much of the learned behavior transfers when the harness changes? Training a separate model for each harness is one option, but it leaves multiple checkpoints to maintain. We investigate whether one model trained across harnesses can retain their average performance and reduce transfer gaps. Model merging provides an alternative way to combine separately trained models into one checkpoint.

Train together, or merge afterward?

We run the single-harness and mixed-harness RL experiments on Lego-RL. It connects the three native harnesses to a shared pipeline for rollout capture, sandboxed task execution, and policy updates.

MH-RL trains one shared model through all three harnesses. For each rollout, we randomly select a harness with probability 1/3, run the task, and score the result. Accepted rollouts update the same model. Over training, that model learns from all three interaction environments.

Model merging combines three separately trained models. Equal-weight parameter averaging (AVG) averages their weights. Task-vector summation (SUM) adds all three learned updates to the shared starting model, with coefficient 1. Both produce one deployment checkpoint; SUM also applies a larger combined update than AVG.

We start from the Qwen3.5-35B-A3B instruction-tuned baseline. The three single-harness RL models are OH-RL, CC-RL, and OC-RL, named for their training harness. We evaluate on SWE-bench Verified, SWE-bench Pro, and SWE-bench Multilingual.

Can one checkpoint replace three?

One shared checkpoint retains nearly the same average accuracy as three single-harness models. Across all three benchmarks, its mean score stays within 0.27 percentage points of deploying each single-harness model through its training harness.

Figure 1. One shared checkpoint versus one model per harness. Each single-harness model is paired with its training harness. Benchmark means weight the three harnesses equally.

MH-RL reduces selected cross-harness transfer gaps. On SWE-bench Multilingual through OpenCode, it improves on CC-RL while staying close to the OpenCode-trained model. The scores are 57.67% for MH-RL and 52.67% for CC-RL. MH-RL also has the smallest average cross-harness score spread among the four RL training strategies, though it does not lead in every setting.

Training rewards increase across all three harnesses within MH-RL. Tracking each harness separately helps check whether the overall reward increase is concentrated in a single interface. The gains differ across harnesses, but the positive trend in each suggests that the shared policy can learn across multiple interaction environments within one run. Figure 2 places these trends alongside the single-harness training curves.

Figure 2. Training progress. Left: MH-RL and single-harness runs. Right: rewards by harness within MH-RL. Faint lines show individual steps; solid lines use trailing 10-step windows. The dotted line marks the evaluated MH-RL checkpoint (step 90). These are training rewards, not held-out benchmark scores.

Experiment notes. All RL runs use Group Sequence Policy Optimization, with Group Relative Policy Optimization advantage estimation. The OpenSWE-derived Lego-RL-2699 dataset provides 2,699 training tasks. Policy updates use VeOmni, AdamW, a learning rate of 10⁻⁶, and mini-batches of 64; rollouts use asynchronous vLLM. The evaluated MH-RL checkpoint is step 90. The one-third probabilities govern rollout selection; filtering and completion times can change the mix of accepted trajectories.

Supplement: PDF · Markdown.

Does mixed-harness RL beat model merging?

MH-RL improves on AVG in the mean scores for SWE-bench Verified and SWE-bench Multilingual. On SWE-bench Pro, their three-harness means are nearly identical. On SWE-bench Verified, MH-RL reaches 68.67%, compared with 66.40% for AVG and 69.00% for SUM. SUM scores higher in the reported accuracy comparisons, so the advantage over AVG does not extend to model merging in general.

Table 1. Model merging vs. mixed-harness RL: resolved tasks (%). Verified and Multilingual average all three evaluation harnesses; Pro uses OpenHands SDK only because SUM results are unavailable through the other two harnesses. Select a column heading to sort.

SUM uses archived reference runs. Runtime configurations are not fully matched, so these scores do not isolate the causal effect of the training strategy.

Does it take fewer interactions?

We compare MH-RL, the three single-harness models, and SUM on SWE-bench Verified tasks solved by all five models within each evaluation harness. MH-RL uses fewer turns and has a lower estimated response length than SUM across all three harnesses. Comparing the same successful tasks helps reveal how their interaction patterns differ.

The shorter trajectories mainly reflect fewer turns. Generated tokens per turn are much closer, so the reduction does not come from consistently shorter individual replies. Figure 3 shows these quantities side by side. Switching tool replies on or off further separates the model’s own generation from the feedback accumulated during tool use.

Figure 3. Response length, tokens per turn, and turns on shared successful SWE-bench Verified tasks. Response length includes cumulative model generation plus saved tool replies by default; switch tool replies off to view generation alone. Generation includes recorded CoT and tool-call tokens. Tool replies are re-tokenized estimates and counted once. Tokens per turn exclude tool replies; turns count model calls.

What should we take away?

One model can work well across several harnesses. In our experiments, MH-RL came close to the average accuracy of three separately trained models and narrowed some of the gaps when switching harnesses. That is encouraging, but all three harnesses were part of training; we still do not know how well the model would handle a new one. Limited compute also left two questions open: which harnesses to train together, and how often to sample each. We gave all three an equal share—a simple starting point, with room to explore better mixtures.

Citation

If you use Lego-RL, please cite the Lego-RL paper.

@article{du2026legorlharnessnativereinforcementlearning,
  title = {LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents},
  author = {Yiming Du and Yuxin Jiang and Tao Yuan and Jianbo Dai
            and Shaowei Wang and Jierun Chen and Chaofan Tao
            and Xianzhi Yu and Lifeng Shang and Kam-Fai Wong
            and Xiaohui Li and Haoli Bai},
  year = {2026},
  journal = {arXiv preprint arXiv:2608.17393},
  url = {https://arxiv.org/abs/2608.17393}
}