[train] feat: Add OpenClaw online RL/OPD/Combine training path

[jamindy]

What Does This PR Do?

This PR adds the OpenClaw-RL online training path to uni-agent on top of verl fully async training, with support for client-driven RL / OPD / Combine rollout flows.

Training data no longer comes from static datasets. Instead, an external OpenClaw client connects to an embedded OpenAI-compatible proxy, drives multi-turn sessions, and submits per-turn training samples directly into the fully async rollout queue.

Specifically:

• Gateway TurnCapture: extends uni_agent.gateway with per-turn export support for OpenClaw traffic. GatewaySession can capture prompt_ids, response_ids, and backend response_logprobs after successful generation, reusing the gateway MessageCodec and avoiding local decode/re-encode.
• Gateway RPC interface: exposes chat_completions and pop_turn_captures through GatewayActor / GatewayManager, allowing OpenClaw rollouters to drive generation through the existing gateway session runtime.
• OpenClaw RL path: adds uni_agent.openclaw.rl, including a fully async rollouter, gateway-backed proxy server, sample builder, trainer, and entrypoint for client-driven binary RL.
• OpenClaw OPD path: adds uni_agent.openclaw.opd, including hint judging, teacher-logprob channel, OPD sample construction, distillation trainer wiring, and gateway-only proxy behavior.
• OpenClaw Combine path: adds uni_agent.openclaw.combine, including hybrid RL + OPD sample dispatch, combine loss integration, and mode-specific rollouter/server/trainer wiring.
• OpenClaw shared layer: adds protocol helpers, PRM/OPD scorers, shared HTTP/entry/sample-builder utilities, and losses registered under uni_agent.openclaw.losses.
• Examples and eval scripts: adds training scripts for RL / OPD / Combine and a 4-GPU OpenClaw RL eval script. The external role-playing LLM launcher has been updated to use the vLLM OpenAI API server.

The default OpenClaw online training path is now:

OpenClaw client → embedded FastAPI proxy → GatewayManager / GatewaySession → rollout vLLM → TurnCapture → PRM / teacher scoring → per-turn RolloutSample → fully async trainer.

PR Scope

This PR focuses on the OpenClaw online fully async training integration:

• Extensions in uni_agent.gateway required for per-turn capture.
• uni_agent.openclaw.rl
• uni_agent.openclaw.opd
• uni_agent.openclaw.combine
• OpenClaw losses / scorers / protocol helpers.
• OpenClaw training and eval examples.

Tests

Important regression coverage includes:

• Gateway per-turn capture export.
• GatewayManager proxy methods for chat completion and capture draining.
• OpenClaw RL per-turn sample construction.
• OPD teacher-logprob sample construction.
• Combine dispatch behavior for OPD+RL / OPD-only / RL-only samples.
• YAML parse checks for openclaw_base, openclaw_rl, openclaw_opd, and openclaw_combine.
• Bash syntax checks for OpenClaw launch scripts.

API and Usage

Example launch scripts:

bash examples/openclaw/train_rl.sh
bash examples/openclaw/train_opd.sh
bash examples/openclaw/train_combine.sh

End-to-End Evaluation for OpenClaw-RL Training Methods:

bash examples/openclaw/eval/run_openclaw_rl_eval_4gpu.sh

Design and Code Changes

High-level structure:

• GatewaySession owns session state and TurnCapture emission.
• GatewayActor provides OpenAI-compatible chat completion RPC and capture draining.
• GatewayManager routes session IDs to gateway actors and exposes the RPCs needed by OpenClaw.
• OpenClawRLServer is the embedded client-facing proxy.
• OpenClawRLRollouter owns the GatewayManager lifecycle and pushes per-turn samples to the fully async message queue.
• OpenClawOPDRollouter and OpenClawCombineRollouter reuse the RL gateway path and customize scoring/sample construction.
• openclaw_base.yaml carries shared fully async config; mode-specific configs only keep deltas.

Key invariants:

• OpenClaw generation goes through gateway only.
• Token truth comes from gateway/backend token IDs and logprobs.
• Training samples do not use a decode → re-encode path.
• Failed generation or capture does not create partial OpenClaw samples.
• Main turns are trainable; side turns are served but not captured.
• The final pending main turn without next-state is submitted as unjudged / neutral.
• Loss registration happens in Ray workers through worker_process_setup_hook.

Checklist Before Submitting

• Read the Contribute Guide
• Run pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add tests or explain why tests are not practical
• Confirm the PR title matches the required format
• Confirm the placeholder text in this template has been replaced with real content

[gemini-code-assist]

Code Review

This pull request introduces an automated evaluation suite and training pipelines for OpenClaw-RL methods, including Binary RL, On-Policy Distillation (OPD), and a Combine/Top-K-Select hybrid. It adds evaluation scripts, vLLM server launchers, training scripts, custom loss functions, and a gateway session manager. Key feedback from the review highlights critical issues, including potential runtime crashes (ReadonlyContainerError) due to direct modifications of read-only Hydra configurations in entry.py and rollouter.py, a potential memory leak in the RL server from unclosed sessions, and an inaccurate overlap metric in topk_select.py caused by including padded positions in the mean calculation.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Hydra-instantiated configurations are read-only by default. Modifying config directly (e.g., config.actor_rollout_ref.rollout.nnodes = ...) will raise a ReadonlyContainerError at runtime. To prevent this crash, use OmegaConf.set_readonly(config, False) before making any modifications.

    from omegaconf import OmegaConf\n    OmegaConf.set_readonly(config, False)\n    start_time = time()\n    auto_set_device(config)\n    # Align rollout pool config with actor_rollout_ref rollouter config.\n    config.actor_rollout_ref.rollout.nnodes = config.rollout.nnodes\n    config.actor_rollout_ref.rollout.n_gpus_per_node = config.rollout.n_gpus_per_node

[gemini-code-assist]

If self.config is a read-only DictConfig (which is standard for Hydra-instantiated applications), attempting to modify self.config.distillation.enabled will raise a ReadonlyContainerError. While the try-except block catches this exception and logs a warning, it leaves enabled as True, which subsequently causes the worker initialization to crash on a None teacher client. To ensure this modification succeeds, use OmegaConf.set_readonly(self.config, False) first.

        try:\n            if self.config.get(\"distillation\") is not None and self.config.distillation.get(\"enabled\"):\n                from omegaconf import OmegaConf\n                OmegaConf.set_readonly(self.config, False)\n                self.config.distillation.enabled = False\n                logger.info(\n                    \"[OpenClawOPDRollouter] native distillation teacher infra disabled on \"\n                    \"rollouter (teacher log-probs computed externally via GenRM)\"\n                )\n        except Exception as e:  # pragma: no cover - defensive\n            logger.warning(\"[OpenClawOPDRollouter] could not disable rollouter distillation: %s\", e)

[gemini-code-assist]

The server stores active sessions in self._sessions and self._gateway_sessions indefinitely. If a client disconnects or fails to send a session_done signal, these sessions will never be cleaned up, leading to a memory/resource leak over time. Consider implementing a session timeout or a periodic background cleanup task to evict stale sessions.

[gemini-code-assist]

The overlap_metric is computed as a simple mean over the entire tensor, which includes padded positions in the batch. Since padded positions will have matching padding tokens (e.g., 0 == 0), the metric distillation/topk_select_overlap will be artificially inflated. To get an accurate metric, mask the gathered overlap with response_mask before computing the mean.

Suggested change

	else: # token_optimal
	sel = overlap.argmax(dim=-1) # (B,S)
	overlap_metric = overlap.gather(-1, sel.unsqueeze(-1)).squeeze(-1).mean()
	else: # token_optimal\n sel = overlap.argmax(dim=-1) # (B,S)\n gathered = overlap.gather(-1, sel.unsqueeze(-1)).squeeze(-1)\n response_mask = data.get(\"response_mask\", None)\n if response_mask is not None:\n mask_bool = response_mask.bool().to_padded_tensor(False) if response_mask.is_nested else response_mask.bool()\n overlap_metric = gathered[mask_bool].mean() if mask_bool.any() else torch.tensor(0.0)\n else:\n overlap_metric = gathered.mean()