KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation

In plain words

Phone assistants are supposed to act for you without being told everything. A benchmark usually checks whether an agent can follow a clear instruction on a phone. This paper asks harder questions: does the agent know what you like, does it ask the right follow-up when an instruction is vague, and does it know when to offer help, wait for a yes, or stay quiet.

KnowU-Bench runs these tests in a rooted Android emulator, so each task can be checked against what actually happened on the screen. The agent gets a history of the user's phone activity but not the user's profile, so it has to work out preferences from the history. When the agent asks a question, an AI stand-in for the user answers in character.

The authors report that models which handle clear tasks well drop sharply on the vague and proactive ones. In their analysis the main weak points are learning preferences through questions and judging when to step in, not moving around the screen. The best model in the study reaches about 60% overall success.

TL;DR

• Online benchmark (Android emulator) with 42 general, 86 personalized and 64 proactive tasks §1 §3.3.
• The agent sees user logs H, the hidden profile P stays with a user simulator §3.2.
• Scoring combines deterministic checks and an LLM judge §3.4.
• Author-reported: 11 models; Claude Sonnet 4.6 has the best overall success (60.4%) §4.2 tab. 2.
• Largest errors are insufficient clarification and unwarranted intervention, not GUI navigation §4.4 fig. 5.

Problem

Existing mobile-agent benchmarks test execution of explicit instructions §2. The few that cover personalization either use offline trajectory matching or intent prediction on a static history, and none measures whether an agent can acquire a missing preference by asking during execution §1 §2. Proactive evaluation usually stops at intent prediction or suggestion ranking, so the full decision chain (intervene, seek consent, or stay silent after a refusal) goes unmeasured §1 §2. The paper also argues that online benchmarks are realistic but hard to reproduce §1.

Method

Environment

The benchmark is formalised as a partially observable decision process over states, observations (instruction plus screenshots), UI actions and a binary completion reward §3.1. The runtime is a containerised Android stack: a rooted Pixel 8 virtual device, driven by a Python web-framework server that coordinates the run. One shared controller translates each agent action into device commands sent over ADB §3.1. The code targets Android API 34 Pixel_8_API_34.ini. Each task starts from a fixed emulator snapshot and resets background processes, callback files and interaction history. Time-sensitive tasks override the device clock at initialisation §3.1.

The action space has 13 actions: tap, double tap, long press, drag, text input, scroll, home, back, enter, wait, answer, status (complete or infeasible) and ask_user tab. 4 app. B.

Apps: the paper says the environment covers 23 applications §3.1. Four service apps were added to the MobileWorld set: two shopping apps and two food-delivery apps. They are rebuilt from third-party projects with local mock data, and order-submission events are sent to a host callback to verify results app. C.2.

User agent

Each simulated user has two parts §3.2:

• a structured profile P, visible only to the simulator. It has seven top-level fields: identity, locations, digital context, habits, preferences, decision criteria and social graph tab. 6 app. D.1;
• a timestamped log H of prior on-device actions, visible only to the agent. Each entry has time, location, action, label and category; the agent sees time, location and action as free text tab. 7 app. D.2.

When the agent calls ask_user, the simulator answers from a role-grounded prompt built from P, the current state S and the dialogue so far §3.2 app. E.2. Four roles are used: Developer, Grandma, Student and Researcher app. D.1. The code names the Researcher profile user.yaml user.yaml app. E.2.

Task definition

The agent receives instruction g, the exposed logs H, and the state S (time, place and, for proactive tasks, the GUI state) §3.3. At each step it samples an action conditioned on g, the current screenshot, the history of past actions, H, S and optional feedback such as the latest ask_user reply §3.3. Three task families §3.3:

• General: explicit instructions, no personal context needed. Baseline for execution alone §3.3.
• Personalized: vague instructions whose details depend on P. The agent may ask, and the simulator replies in free text §3.3.
• Proactive: no instruction. The agent must pick direct execution, a proposal for confirmation, or silence. A confirmation request gets an accept or reject reply, and the agent must follow it §3.3.

Preference categories are described as platform choice, beverage, diet, shopping priority, payment method and navigation app app. D.1.

Hybrid scoring

Each task score is a combination of a rule-based score S_rule (0 or 1 for deterministic tasks, or a bounded base score for hybrid tasks) and an LLM judge score S_llm. The rubric-conditioned judge returns a normalised score and a rationale §3.4. The paper writes the final score as a weighted sum, S_i = lambda_i * S_rule + (1 - lambda_i) * S_llm, with lambda_i in [0, 1]. It sets lambda_i = 1 for fully deterministic tasks, lambda_i = 0 for purely semantic tasks, and otherwise a value proportional to the share of preference-dependent requirements §3.4. Numeric lambda values are not stated. Rule checks cover recipient, order creation, alarm or setting values, time windows and forbidden actions after a rejection §3.4.

Each episode is scored as successful when its score is above 0.99 §4.1.

Pseudocode of one episode, reconstructed from §3.1–§3.4:

# One KnowU-Bench episode  [§3.1] [§3.2] [§3.3] [§3.4]
# Input:  instruction g, hidden profile P, exposed logs H, state S
# Output: score in [0, 1], number of ask_user queries c

def run_episode(instruction, profile, logs, state):
    # Restore the fixed emulator snapshot and clear transient state  [§3.1]
    device = reset_emulator()
    # The agent sees g, H and S, but never P  [§3.2]
    context = [instruction, logs, state]
    queries = 0
    for step in range(max_steps):  # max_steps is not stated in the paper
        # One UI action or one ask_user question per step  [§3.3]
        action = agent.act(context, device.screenshot())
        if action.type == "status":
            break
        if action.type == "ask_user":
            queries += 1
            # The simulator answers from P, S and the dialogue  [§3.2]
            reply = user_simulator(profile, state, context, action.text)
            context.append((action.text, reply))
        else:
            device.execute(action)
    # Deterministic checks over the final device state  [§3.4]
    rule_score = rule_check(device, profile)
    # Rubric-conditioned LLM judge over evidence and dialogue  [§3.4]
    llm_score = llm_judge(device, profile, context)
    # Per-task weight, lambda_i in [0, 1]; numeric values not stated  [§3.4]
    weight = task_weight
    score = weight * rule_score + (1 - weight) * llm_score
    return score, queries

Metrics

• Success Rate (SR): share of tasks in a split with score above 0.99 §4.1.
• Efficiency: the paper defines it as 50 divided by the average number of steps §4.1. It is plotted in fig. 3 panel (b), not in tab. 2.
• Average Score: mean score over personalized tasks §4.1.
• Interaction Efficiency (IE): over personalized tasks, the mean of S_i / max(c_i, 1), where c_i is the number of ask_user calls §4.1. The paper plots it in fig. 3 panel (b).
• Proactive Act, Silent, Stop: Act is the rate at which the agent intervenes when intervention is warranted; Silent is the rate at which it stays quiet when it is not; Stop is the rate at which it stops after an explicit rejection §4.1. The repository computes these from a per-episode bucket (below) and not from the paper's words, see Paper vs code.

# Bucket one proactive episode  [code: scripts/calc_paper_metrics.py#L289-L296]
# Input:  user_decision ("accept", "reject" or None)
#         habit_should_act (True, False or None, from the profile habits)
# Output: "need", "noneed", "reject" or None

def proactive_bucket(user_decision, habit_should_act):
    # An explicit rejection overrides the habit label
    if user_decision == "reject":
        return "reject"
    # Otherwise the profile habit says whether action is warranted
    if habit_should_act is True:
        return "need"
    if habit_should_act is False:
        return "noneed"
    return None

Hyperparameters

Results

Author-reported. Setup: 11 models in three groups (GUI-specific, general open-source, closed-source); the main setting is full history with noisy logs §4.1. Table 2 is reproduced from the paper tab. 2. Per-column bold and underline marks are omitted. "Gen." is General, "Pers." is Personalized and "Pro." is Proactive. Easy and hard are the paper's splits.

Source: tab. 2. Units are percent for SR and a 0 to 1 scale for Score.

Memory settings, computed over personalized and proactive tasks only tab. 3:

Findings stated in the text §4.2 §4.3:

• Easy general tasks are largely solved by the strongest models: MAI-UI-8B and Seed 2.0 Pro both score 100.0% on the easy general split tab. 2. The authors say strong models lose about 30% on average once success depends on personalization or proactivity §1.
• Role matters. Claude Sonnet 4.6 stays between 71.7% and 79.4% across the four roles; Seed 2.0 Pro ranges from 48.5% (Grandma) to 71.3% (Researcher) §4.2 fig. 3 panel (a). Grandma is the hardest role on average §4.2.
• Asking more does not help by itself. Claude Sonnet 4.6 asks about 0.4 questions per task on average. Seed 2.0 Pro asks about twice as many but lags behind on personalized tasks §4.2 fig. 3 panel (b).
• Proactive calibration is the main axis. Claude Sonnet 4.6 has the best Act rate, 70.8%. Qwen3.5-397B-A17B leads Silent (73.7%) and Stop (75.0%) but has 31.8% on Act §4.2 fig. 3 panel (c).
• Memory interface is model dependent. Qwen3-VL-8B gains from retrieval (13.6% to 20.4% on clean logs). UI-Venus-1.5-8B does better with full logs. MAI-UI-8B drops to 9.3% under noisy retrieval §4.3 tab. 3.
• Failure breakdown for Claude Sonnet 4.6 §4.4 fig. 5. Personalized failures: Clarify 66.7%, Partial 27.1%, GUI 4.2%, Preference 2.1%. Proactive failures: Intervention 60.0%, Passive 20.0%, GUI 15.0%, Rejection 5.0%.
• Judge check §4.3 fig. 4: on 26 shared trajectories, a hybrid of LLM judge and rule checks has lower error against four human raters than rules alone. Numbers are in the figure inset only.

Figures in words: fig. 1 contrasts clear and vague instructions. fig. 2 gives the framework overview. fig. 3 has three panels (roles, interaction metrics, proactive safety). fig. 4 shows judge agreement. fig. 5 shows the failure categories. Appendix figures fig. 6–fig. 17 show the pipeline and example trajectories. None are reproduced here.

Paper vs code

Repository: https://github.com/ZJU-REAL/KnowU-Bench (commit c03a825991ede13add6631f2ed19b90755930dc6, Apache-2.0 LICENSE). The paper's own metadata links this repository, which is the basis for treating it as official.

• Score fusion is not a single lambda rule. The paper's weighted sum with lambda_i §3.4 does not appear in the code. Each preference task hard-codes its own combination. Examples: placing an order is worth a fixed 0.5 and the judge adds up to 0.5 order_lunch_tradeoff.py; 0.4 times a base score plus 0.6 times the judge buy_computer_ask_user.py; 0.8 times a sort match plus 0.2 for asking, then 0.4 times base plus 0.6 times the judge sort_files_by_habit_ask_user.py sort_files_by_habit_ask_user.py; and a minimum of the base score and a weighted judge highlight_important_sms_sender_ask_user.py. A pattern search found such fixed combinations in 15 of the 27 preference task files; I did not read every file.
• Efficiency is not in the code. The formula 50 / average steps §4.1 was not found in src/ or scripts/.
• Interaction efficiency matches for the paper's splits, computed as mean of score / max(queries, 1) calc_paper_metrics.py. The code also reports it for proactive episodes, which the paper does not define. The web log viewer uses a different UIQ: binary success, q_i = s_i / c_i only when c_i > 0, and a denominator that also counts non-interaction tasks that asked the user utils.py utils.py.
• Act, Silent and Stop are computed differently from the text. Act is success rate on "need" episodes, so an intervention that fails execution counts as a miss calc_paper_metrics.py. Silent is the absence of any engagement action on "noneed" episodes calc_paper_metrics.py. Stop is the absence of stop-violating actions after the first rejection calc_paper_metrics.py. The need/noneed label comes from profile habits; an explicit rejection overrides it calc_paper_metrics.py.
• User simulator and judge model. The paper says gpt-4o §4.1. The code default for the preference simulator and judge is gpt-4o base_prefrence_task.py, but the shipped .env.example sets gpt-4.1 .env.example, and the base task falls back to gpt-4o-mini base.py.
• Total task count. The log viewer divides by 201 tasks utils.py. The paper's counts sum to 192.
• Logs. The repository has clean logs and 25%-noise logs for developer, grandma, student and user (file names only; the ratio was not counted).
• No discrepancy found in the success threshold (0.99) calc_paper_metrics.py.

Pitfalls

• Step budget. The paper does not give one. The code's default is unlimited unless --max-round is passed eval.py. Reported numbers depend on the budget used, which the card cannot determine.
• Judge failures give half credit. If the judge call raises, the judge returns 0.5 with an error message base_prefrence_task.py. A run with API errors therefore mixes in partial scores.
• Judge parsing. If the judge output has no score tag, the code takes the last number in the text base_prefrence_task.py. That number can come from the reasoning.
• Near-perfect judge needed for success. In the lunch-ordering task, any placed order gives 0.5. Passing the 0.99 threshold then requires a judge score of at least 0.98 order_lunch_tradeoff.py. Success on hybrid tasks is therefore close to a judge verdict.
• Claude Average Score is inconsistent in the text. §4.2 says 78.9%; tab. 2 shows 0.78 (easy) and 0.80 (hard).
• Application count. The text says 23 applications §3.1; tab. 5 lists 17.
• Two different overall rates. tab. 2 counts general tasks and tab. 3 does not. Don't compare them directly.
• Proactive percentages in §4.2 are only in the figure. Act, Silent and Stop are in fig. 3 panel (c), not in any table.
• Error-analysis counts. The paper gives percentages only. They fit exactly 48 personalized and 20 proactive failures (e.g. 2.1% = 1 of 48, 60.0% = 12 of 20). That is my inference, not stated.
• Judge agreement numbers are only in fig. 4. The text gives direction, not values.
• Judge model not named. The paper does not say which model judges the episodes §3.4. The code uses the same environment setting as the user simulator base_prefrence_task.py. Changing either model can shift scores; the paper's sensitivity check is the human comparison in fig. 4 only.
• Third-party app sources. The shopping apps are adapted from qykong/mall_fork and the food apps from Flash Waimai app. C.2. Their licences were not checked, and nothing from them is used on this card.
• Hidden profile and logs are LLM-generated. Profiles and logs are synthetic, then manually reviewed app. D.1 app. D.2. Real-user behaviour is not measured.
• Model versions. Several closed models are named as previews or with version numbers (Gemini 3.1 Pro Preview, Seed 2.0 Pro, Claude Sonnet 4.6). Access dates are not in the paper.

Lineage

• Builds on: AndroidWorld, for the reproducible full-stack Android environment and programmatic checks §2; MobileWorld, whose app ecosystem this work extends and whose UIQ metric is adapted §2 app. C.2 §4.1.
• Compared against: PersonalAlign and Me-Agent (preference recovery from history), FingerTip 20K (proactive suggestions from usage logs), ProactiveMobile, PIRA-Bench and Pare (proactive intent and API-level execution) §2 tab. 1.
• Superseded by: none stated.