ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents

In plain words

A GUI agent operates a phone or computer the way a person does: it looks at the screen and taps, swipes and types. Training one by trial and error is hard. The emulators crash, a real phone cannot report whether a task worked, and a long task yields one pass-or-fail signal at the end, which says little about which steps helped. ClawGUI is a bundle that covers the whole path: training the agent with reinforcement learning, testing it on public benchmarks in a fixed way, and running the trained agent on real phones through a chat app.

The training part adds a score for each step. A separate judge model looks at a step and says whether it moved the task forward, and the learning rule credits individual steps as well as the whole attempt. The authors trained a 2B model this way, starting from an existing 2B model, and it scored higher on a mobile benchmark than that starting point.

The evaluation part pins the settings, such as prompt, resolution and temperature, for each model. The aim is that a published number can be re-checked and compared with others.

TL;DR

• ClawGUI is an open-source stack for GUI agents: online RL training, a standardised evaluation pipeline, and a chat-driven agent for Android, HarmonyOS and iOS §1 §3.1.
• Its RL method is GiGPO with a per-step Process Reward Model (PRM). On MobileWorld GUI-Only (117 tasks) the authors report 17.1% SR with GiGPO against 14.5% with GRPO §4.3 tab. 2.
• The trained ClawGUI-2B scores 17.1% against 11.1% for its MAI-UI-2B base tab. 1. The same table lists MAI-UI-8B at 19.7% and Doubao-1.5-UI-TARS at 26.3%, both above it.
• The shipped MobileWorld scripts do not reproduce the paper's training setup, and the shipped step judge sees less context than the paper says. See "Paper vs code".
• All numbers are author-reported. The paper reports no repeated runs or variance.

Problem

Online RL for GUI agents is limited by three things, as the paper describes them §1. First, emulators drift into unhealthy states over long runs, and real devices expose no system-level checks, so environments are unstable. Second, rewards are sparse over long tasks. Third, reported benchmark numbers shift with choices that are often not documented, such as prompt format, coordinate normalisation, image resolution and sampling settings §1 §2.2. The paper also states that existing open RL infrastructure is validated only in emulators, and that real-device RL is largely unexplored §2.2.

Method

The paper presents ClawGUI as three modules §3.1: ClawGUI-RL (training), ClawGUI-Eval (evaluation) and ClawGUI-Agent (deployment). The method that matters for training is ClawGUI-RL.

ClawGUI-RL environments §3.2.1 fig. 2. One interface covers Docker-based Android emulators (run through MobileWorld) and physical or cloud phones. Each emulator episode has four stages: reset to a clean device and a new task; evaluation, using system-level checks on app state and database records, plus an MLLM judge that reads the final screen; spare-server rotation, where an unhealthy container is swapped for a spare; and periodic teardown and restart. On real devices the tasks are written by hand, and evaluation relies on the MLLM judge alone, because no system-level access is available §3.2.1.

Reward §3.2.2 eq. 1. The outcome reward is binary: 1 for success, 0 for failure, given at episode end. A PRM is run after every action. It receives the previous screenshot, the current screenshot and the actions taken so far, and returns a score for whether the action contributed to the task. The paper combines the two as R = R_outcome + R_step eq. 1. The paper does not give the range or scale of the step score.

Trainer §3.2.3. ClawGUI-RL builds on verl and verl-agent and supports Reinforce++, PPO, GSPO, GRPO and GiGPO. The experiments use GRPO and GiGPO. GRPO normalises the returns of a group of rollouts on the same task and gives every step of a trajectory the same advantage §3.2.3. GiGPO has two levels. The episode level is the same as GRPO's. The step level clusters steps that reach the same anchor state across rollouts into sub-groups, and normalises each step's discounted return inside its sub-group. It needs no learned critic and no extra rollouts §3.2.3. The paper does not write out the equations and refers to the GiGPO paper, so the pseudocode below follows the shipped code.

# GiGPO advantage, as the shipped code computes it
# [code: clawgui-rl/gigpo/core_gigpo.py#L138-L171]
# Input:  one row per env step: step_reward, episode_score,
#         group_id, traj_id, anchor_obs
# Output: one advantage per row (applied to all its tokens)

def gigpo_advantage(rows, gamma, step_weight):
    # 1. Discounted return per step, computed backwards inside
    #    its trajectory  [code: clawgui-rl/gigpo/core_gigpo.py#L117-L119]
    step_returns = discounted_returns(
        rows.step_reward, rows.traj_id, gamma
    )
    # 2. Episode term: centre episode scores within each group,
    #    pooling all step rows of the group (not one value per
    #    trajectory). Std is divided only in mode mean_std_norm.
    #    [code: clawgui-rl/gigpo/core_gigpo.py#L161-L180]
    #    [code: clawgui-rl/gigpo/core_gigpo.py#L217-L223]
    episode_adv = centre_within(rows.episode_score, rows.group_id)
    # 3. Anchor sub-groups: inside a group, rows whose anchor
    #    observation is identical share one id  [code:
    #    clawgui-rl/gigpo/core_gigpo.py#L243-L331]
    anchor_id = cluster_identical(rows.anchor_obs, rows.group_id)
    # 4. Step term: centre discounted returns inside each sub-group
    #    [code: clawgui-rl/gigpo/core_gigpo.py#L334-L384]
    step_adv = centre_within(step_returns, anchor_id)
    # 5. Joint advantage; the shipped script uses step_weight = 1.0
    #    [code: clawgui-rl/gigpo/core_gigpo.py#L170]
    return episode_adv + step_weight * step_adv

The shipped MobileWorld judge (not described in the paper) turns each step into a 0 or 1 label. It marks the action point on the previous screenshot and asks a judge model whether the action helped the task, using the task, the previous action only, and both screenshots judge.py judge.py judge.py.

ClawGUI-Eval §3.3. It covers six benchmarks: ScreenSpot-Pro, ScreenSpot-V2, UI-Vision, MMBench-GUI, OSWorld-G and AndroidControl §3.3.1. Evaluation has three decoupled stages §3.3.2 fig. 3. Infer runs a model either locally through transformers or remotely through an OpenAI-compatible API. Multi-GPU runs use one process per GPU, with shard-level checkpoints. Judge applies a benchmark-specific rule: point-in-box for grounding, a polygon and refusal-aware rule for OSWorld-G, and a multi-action rule for AndroidControl. Metric aggregates per-sample labels, broken down by platform, element type and task category. Any stage can be re-run alone, for example to re-judge saved predictions §3.3.2. A reproduced result counts as successful when it is at least the official number or within 2 absolute points §4.4.

ClawGUI-Agent §3.4. The agent takes instructions from 12+ chat platforms. The paper names Feishu, DingTalk, Telegram, Discord, Slack and QQ §3.4.3. In remote mode a phone is driven from another device. In local mode the chat app runs on the phone itself §3.4.3. Control is hybrid: CLI where the app offers an interface, GUI otherwise §3.4.1. A personalised memory stores structured facts such as contacts, frequently used apps and habits as embeddings, retrieves the top-k similar memories for each task, and merges duplicates §3.4.2. The evaluation pipeline is also exposed as an agent skill §3.4.4. This part was not checked against code for this card.

ClawGUI-2B training setup §4.1. The base model is MAI-UI-2B. Training uses 64 parallel virtual environments on 8 A6000 GPUs (48 GB each). The algorithm is GiGPO with a rollout group size of 8, temperature 0.7, learning rate 1e-6, 3 epochs and a training batch size of 8. The PRM judge is Qwen3.5-72B. The evaluation cap is 50 steps.

Hyperparameters

Results

Author-reported. Setting: MobileWorld GUI-Only, 117 tasks, success rate (SR), at most 50 steps per task §4.1. ClawGUI-2B is ClawGUI-RL with GiGPO applied to MAI-UI-2B §4.2.

Table 1: GUI-Only, 117 tasks tab. 1

Table 2: reward design on GUI-Only tab. 2

The paper reports a relative gain of 17.9% for GiGPO over GRPO and an absolute gain of 2.6 points §4.3.

Table 3: grounding reproduction against official numbers tab. 3

Each cell is official → reproduced. "–" means no official number. Reproduced counts as successful if it is at least the official value or within 2 points. The paper reports 46 of 48 cells with an official number as successful (95.8%) §4.4. Failures: Qwen3-VL-2B and UI-TARS 1.5-7B on ScreenSpot-Pro.

The closed-source rows use a two-stage Zoom method with 25% crop tiles for Gemini and 50% for Seed §4.4.

Paper vs code

Repository: https://github.com/zju-real/ClawGUI, linked in the paper's front matter (main.tex, title block). It carries an Apache-2.0 LICENSE. The commit checked is d990d3e3f2390ced708771d9681d3c4f9a6cbef5. The repository also contains clawgui-eval, clawgui-agent, clawgui-skills and clawgui-app directories. Only clawgui-rl and clawgui-eval were compared with the paper.

• No shipped script reproduces §4.1. The paper's ClawGUI-2B setup (64 environments, 8 GPUs, group size 8, batch 8) is not in the repository. The GiGPO MobileWorld script uses GUI-Owl-1.5-2B, one GPU, group size 1, history length 1, a 1-step cap, and no KL term run_mobileworld.sh run_mobileworld.sh. The PPO MobileWorld script uses four GPUs, batch size 4, history length 3, a 50-step cap, GAE with a critic, and KL 0.01 run_mobileworld.sh run_mobileworld.sh.
• Shared values. Learning rate 1e-6 and temperature 0.7 match the paper run_mobileworld.sh run_mobileworld.sh.
• Epochs. Both scripts pass total_epochs=3 to data preparation but set trainer.total_epochs=1 run_mobileworld.sh run_mobileworld.sh. Which value governs the run was not traced.
• Step judge off. Both MobileWorld scripts set step_reward_judge=False run_mobileworld.sh. The paper uses a 72B PRM judge §4.1. The real-device GiGPO script turns the judge on run_realdevice.sh.
• Judge context. The paper says the PRM sees all previous actions §3.2.2. The shipped judge sees only the previous action: "We only use the previous step's action" judge.py.
• Judge failures score as helpful. The judge returns 1 on an API error judge.py and on an unparseable reply judge.py. The paper says nothing about failure handling.
• Reward sum versus mean. Eq. 1 sums the outcome and step rewards. When the judge is on, the rollout loop divides each intermediate step reward by the number of intermediate steps and sets the episode reward to the mean of those plus the final evaluation rollout_loop.py rollout_loop.py. The code's comment gives a range of about 0 to 2. The code therefore does not compute R_outcome + R_step as written.
• Episode statistics pooled over steps. The episode-level mean and std use compute_mean_std_cross_steps=True by default, and the call does not override it core_gigpo.py core_gigpo.py. Every step row is added to the statistics, with no per-trajectory de-duplication core_gigpo.py. The paper describes a trajectory-level macro advantage §3.2.3. A longer trajectory therefore adds more rows to its group.
• No std division by default. The default mode, mean_norm, subtracts the mean and does not divide by std core_gigpo.py core_gigpo.py. The paper says "normalization" without saying which.
• Discount. Step returns are discounted sums of per-step rewards ray_trainer.py core_gigpo.py. The shipped gamma is 0.95 run_mobileworld.sh. The paper does not state gamma.
• Invalid-action penalty. Each invalid action subtracts a coefficient from the step and episode rewards ray_trainer.py. The scripts set it to 0.1. The paper does not mention it.
• Anchor matching. Anchor groups use exact string equality by default; similarity matching is off core_gigpo.py core_gigpo.py. The paper describes "the same intermediate state". Which observation becomes anchor_obs on MobileWorld was not traced.
• Evaluation settings. The paper says every model's settings are pinned but gives no values §3.3. The README default for grounding temperature is 0 README.md. It states that resolution changes move results by about ±0.1% README.md.
• Reproduction count matches. The eval README reports 44 of 46 open-source cells, 95.7% README.md. This agrees with the paper's table.

Pitfalls

• The headline comparison leaves out stronger rows. The paper says small trained models beat larger untrained ones, and names Qwen3-VL-32B and UI-Venus-72B §4.2 (2). The same table lists MAI-UI-8B at 19.7% and Doubao-1.5-UI-TARS at 26.3%, both above ClawGUI-2B's 17.1% tab. 1.
• Two different "gain" figures. §4.2 says "a relative margin of 6.0%" §4.2 (1). 17.1 against 11.1 is 6.0 absolute points, a relative gain of about 54%, which the conclusion states §6. The contributions list say "6.0 absolute points" §1. The abstract says "6.0%", without saying which.
• Small numbers, one run. With 117 tasks, one task is 0.85 percentage points. 17.1% is about 20 tasks, 14.5% about 17, and 11.1% about 13. The GiGPO-over-GRPO gap is about three tasks. No seed count, repeat or variance is reported.
• Causal claim. The paper says both models share base weights and that "the gain stems entirely from ClawGUI-RL" §4.2 (1). The only controlled comparison is GiGPO against GRPO tab. 2. The GRPO run's settings are described only as "the same pipeline" §1. It is also not stated whether the 11.1% baseline was re-measured under this harness.
• Reproduction criterion. A result counts as reproduced if it is at least the official value, however far above. Otherwise a 2-point margin applies §4.4. The two failures are attributed to undisclosed prompt or resolution choices §4.4.
• Benchmark count. The headline says six benchmarks §1. Table 3 covers five benchmarks and omits AndroidControl. The Figure 3 caption also says five fig. 3 tab. 3.
• Real-device training is judge-only. Without system checks, the MLLM judge is the only reward signal on real phones §3.2.1. Its error rate is not reported, and the shipped judge fails open to 1 (see "Paper vs code").
• Cost is hidden. Each step is judged by a 72B model §4.1. The paper gives no judge call counts or cost. Container restarts and spare servers also add load §3.2.1.
• Unstated RL details. The paper leaves out the optimiser, KL setting, gradient clipping, maximum response length, image resolution, restart interval and spare-server count (see Hyperparameters). An implementer must take these from the shipped scripts, which differ from §4.1.
• Claim not checked. "First open-source GUI agent RL infrastructure" §1 is a priority claim. It was not checked against the literature in this card.

Lineage

• Builds on: GiGPO (arXiv:2505.10978) for the step-level advantage, and verl and verl-agent as the training base §3.2.3. MAI-UI-2B (arXiv:2512.22047) is the base model and baseline §4.1. MobileWorld (arXiv:2512.19432) is the benchmark and emulator source §3.2.1.
• Compared against: GRPO tab. 2; MAI-UI-2B and MAI-UI-8B, Qwen3-VL, UI-Venus, GUI-Owl, Doubao-1.5-UI-TARS tab. 1; GUI-G² and other grounding models tab. 3.
• Superseded by: not known from wiki content.