RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies

In plain words

Testing a robot policy, the learned controller that decides how the arm moves, is hard to do fairly. Standard tests fix the tasks, the objects, the lighting and the camera angles, and those setups are hard to recreate across labs. They also cover only a few tasks.

RoboArena takes a different route. Evaluators at several institutions each run their own head-to-head trials of two policies, on whatever task and scene they pick. They do not know which policy is which. Each trial ends with a winner or a tie and a short written reason. A ranking model turns the many win and loss records into one ordering. It also accounts for the fact that some tasks are so easy or so hard that both policies look the same there, so the tasks do not have to match between trials. A language model reads the written feedback to summarise each policy's strengths and weaknesses. The whole system is built on the DROID robot platform.

TL;DR

• Policies are compared by crowd-sourced, double-blind pairwise real-robot trials, not on fixed tasks §1 §3.1.
• Rankings come from a Bradley-Terry model with latent task buckets, fit by EM, with ties handled by the Davidson extension §3.2 eq. 1 alg. 2.
• Author-reported: 612 pairwise comparisons; ranking closer to an exhaustive "oracle" than the standard DROID protocol, and stable after about 100 comparisons §1 §5.2 §5.3.
• The public code no longer serves the task-aware model. Its pair sampling and EM gradients also differ from the paper in places. See Paper vs code.

Problem

Comparing generalist robot policies needs the same scenes, lighting, camera views and objects. Reproducing those conditions is hard, especially across institutions, and fixed benchmarks cover few tasks §1. The paper asks whether many distributed, unstandardized evaluations can rank policies more accurately than a centralised, standardised one §1 §5.2. It assumes every policy in the pool can be run safely across a broad range of scenes and tasks §3.1.

Method

Evaluation protocol alg. 1 §3.1

1. An evaluator asks the central server C for a session. C draws two policies, π_A and π_B, from the pool at random and sends them to the evaluator, who sees only the remote server addresses §3.1 alg. 1.
2. The evaluator arranges the scene, writes a natural-language task T_i, and runs π_A and π_B back to back until the task ends or a fixed timeout is reached. Initial conditions must match within a pair but may change between pairs §3.1.
3. The evaluator reports three things: a progress score from 0 to 100 for the best progress reached, a binary preference, and a free-text reason §3.1.
4. The task, feedback and recordings are sent to C. Sessions run asynchronously §3.1.

Ranking model: task-aware Bradley-Terry §3.2 app. B.1

• Plain Bradley-Terry gives the win probability as σ(θ_A − θ_B) §3.2. The paper argues that this treats task effects as noise, which hurts the ranking §3.2.
• Each trial is assigned a latent task bucket t out of T, with prior ν_t summing to one. A bucket has a difficulty τ_t. Each policy has an offset ψ_{p,t} for each bucket. The win probability is given by eq. 1 §3.2 eq. 1.
• Within a trial, each policy solves the task independently with probability q = σ(θ + ψ − τ) app. B.1.
• Outcome probabilities for a trial in bucket t: win = q_i(1 − q_j); loss = (1 − q_i)q_j; tie = 2ν_tie √(q_i(1 − q_i)q_j(1 − q_j)), where ν_tie is the Davidson tie rate app. B.1.
• Parameters are fit by approximate maximum likelihood with EM. The E-step computes bucket responsibilities. The M-step does clipped Newton updates for θ, ψ and τ, using the gradients and Hessians in app. B.2, then centres the parameters, then updates ν and ν_tie alg. 2 §3.2.
• The final ranking sorts policies by θ, highest first alg. 2.
• An optional partial-success term adds a Gaussian likelihood on the progress scores app. B.3.

# Task-aware Bradley-Terry fit by EM  [alg. 2] [app. B.2]
# Input:  policy_a, policy_b (M,) indices; outcome (M,) in {0, 1, 2}
# Output: theta (N,) abilities, best policy first

def fit_task_aware_bt(policy_a, policy_b, outcome, num_tasks):
    # Initialise; the spread 0.1 is a variance or a std (not stated)  [alg. 2]
    theta = normal(0.0, 0.1, num_policies)            # (N,)
    psi = zeros((num_policies, num_tasks))            # (N, T)
    tau = normal(0.0, 0.1, num_tasks)                 # (T,)
    nu = full(num_tasks, 1.0 / num_tasks)             # (T,) bucket prior
    nu_tie = 0.5                                      # Davidson tie rate

    for iteration in range(em_iters):
        # Solve probabilities for both policies in each bucket  [app. B.1]
        solve_a = sigmoid(theta[policy_a, None] + psi[policy_a] - tau)
        solve_b = sigmoid(theta[policy_b, None] + psi[policy_b] - tau)

        # Outcome probability given each bucket  [app. B.1]
        p_win = solve_a * (1 - solve_b)               # (M, T)
        p_loss = (1 - solve_a) * solve_b              # (M, T)
        p_tie = 2 * nu_tie * sqrt(p_win * p_loss)     # (M, T)
        # outcome 0 selects p_loss, 1 selects p_tie, 2 selects p_win
        likelihood = select(outcome, p_loss, p_tie, p_win)

        # E-step: posterior over the bucket of each trial  [alg. 2]
        responsibility = nu * likelihood              # (M, T)
        responsibility /= responsibility.sum(axis=1, keepdims=True)

        # M-step: clipped Newton step on abilities, then centre  [alg. 2]
        theta_previous = theta
        theta_grad, theta_hess = theta_derivatives(theta, psi, tau, responsibility)
        theta = theta - clip(theta_grad / theta_hess, clip_value)
        theta = theta - mean(theta)

        # M-step: policy-bucket offsets, centred per policy
        # [code: central_server/central_server.py#L806]
        psi_grad, psi_hess = psi_derivatives(theta, psi, tau, responsibility)
        psi = psi - clip(psi_grad / psi_hess, clip_value)
        psi = psi - mean(psi, axis=1, keepdims=True)

        # M-step: bucket difficulties, centred  [alg. 2]
        tau_grad, tau_hess = tau_derivatives(theta, psi, tau, responsibility)
        tau = tau - clip(tau_grad / tau_hess, clip_value)
        tau = tau - mean(tau)

        # Bucket prior and tie rate  [alg. 2]
        nu = responsibility.mean(axis=0)
        ties = sum(responsibility * p_tie)
        wins = sum(responsibility * p_win)
        nu_tie = 0.5 * ties / wins

        # Stop when the largest change in theta is below tol  [alg. 2]
        if max(abs(theta - theta_previous)) < tol:
            break

    return argsort(-theta)                            # best policy first

The clipping bound is step_clip times step_decay to the power of the iteration app. B.5 central_server.py. The function names above (theta_derivatives and so on) stand for the closed-form expressions in app. B.2; the paper does not give them as code.

Qualitative policy characteristics §3.3 fig. 2 app. E

• A vision-language model reads the first frame of each evaluation video, with the instruction, and assigns a task category and describes the scene's lighting, clutter and object visibility. The paper names GPT-4.5 for this step §3.3.
• For each policy, a language model reads the preference labels, the task categories and the free-text feedback, and writes a report. The report compares the policy with others per category and lists strengths and weaknesses. It must cite evaluation episodes, and the citations link to videos §3.3. The paper names o3 for this step §3.3 app. E.
• The prompt is shown in fig. 14, and a summarising prompt in fig. 15. The full report for π0-FAST-DROID is shown in fig. 13 with video references removed app. E.

Evaluation system §4 fig. 3 fig. 4

• The prototype runs on DROID, which has a Franka Panda 7-DoF arm, a Robotiq 2F-85 gripper, and ZED cameras §4 fig. 3.
• Four parts: remote policy inference servers, which submitters host themselves; evaluation clients, a script that needs no inference compute; an evaluation database; and a central server that assigns policies, tracks the pool and cancels stalled evaluations §4.1 fig. 4.
• Safety: every new policy server is checked for input and output format. Then it runs in a test environment under a trained evaluator who can intervene §4.2.
• Evaluation credits: each evaluation earns one credit, which buys one evaluation of the evaluator's own policy against the pool §4.2.

Hyperparameters

Paper defaults and search ranges are from the EM hyperparameter table tab. 1.

Results

Author-reported throughout. Setup: a pool of seven generalist DROID policies, namely π0-flow-DROID, π0-FAST-DROID and five PaliGemma-based models using flow, FAST, FAST+, FSQ and binning action representations §5.1. The paper states the policies use the public openpi implementation §5.1. The oracle ranking scores every policy on every task, using 4,284 scored rollouts §5.1. The baseline is the standard DROID protocol with 17 tasks and 44 episodes per policy §5.1.

• Agreement with the oracle: the task-aware model ("TASK") gives the best ranking. Progress-based ("PROG"), standard Bradley-Terry and Elo rank below it, and the standard protocol ("Regular") is least reliable §5.2 fig. 6. The text does not order PROG against plain Bradley-Terry and Elo, and the exact values are in the figure only.
• The authors state that progress-based and preference-based rankings are complementary and both should be reported §5.2.
• LLM-assisted claims about π0-FAST-DROID are checked against win rates by category. Most claims that it outperforms, matches or underperforms another policy agree with the data §5.4 fig. 9.

Paper vs code

Search and code used. The paper links the project site robo-arena.github.io. The fetched page text contained no links to code. A GitHub search for "roboarena" found two candidates. robo-arena/roboarena (MIT) is a policy-serving SDK and appears to contain no ranking code, so it was not used. pranavatreya/roboarena_backend (MIT, copyright Pranav Atreya, one of the paper's authors) contains the leaderboard and the ranking code, and is recorded as code at commit ca7b8116c68c2f75313441d751ccfae7d9daacb3. Its LICENSE is dated 2026, so the code is later than paper v2 (2025-11-29). Discrepancies below are read from the code, not run.

Discrepancies between the paper and the official code:

• The live leaderboard does not use the task-aware model. The EM routine em_hybrid is marked "Legacy… intentionally disabled" central_server.py. The leaderboard calls fit_bt_davidson instead, a Bradley-Terry fit with Davidson ties, fit by Newton's method central_server.py central_server.py. The paper's baseline in app. B.4 uses gradient ascent with ties counted as half-wins, so the baseline differs too.
• Bucket count. The code sets 100 task buckets central_server.py. The paper's default is 60 tab. 1.
• Pair sampling. The paper says C draws two policies at random §3.1. The code draws the first policy with weights that favour policies with fewer A/B evaluations central_server.py central_server.py, draws the second uniformly central_server.py, and randomises which one is shown as A central_server.py. The threshold is 500 by default config.py.
• Possible crash in the weighting. The weight function calls np.pow central_server.py. NumPy provides np.power, not np.pow. As written, this would raise AttributeError for any policy with fewer evaluations than the threshold, which is nearly every policy by default. I did not run this, so check it before relying on the sampler.
• θ and ψ gradients use the opponent's probability. Appendix B.2 gives ∂ log P(win)/∂z_i = 1 − σ(z_i), with the policy's own solve probability app. B.2. The code's win and loss terms use sj_i, which is the other policy's solve probability central_server.py central_server.py. The code's tie term is σ_opponent − σ_self, while the appendix's analytic form is ½ − σ(z_i). The τ gradient matches the appendix for all three outcomes central_server.py. A finite-difference gradient check would settle the θ and ψ question. This affects only the EM model, which the live leaderboard no longer uses.
• Step clipping decays in code. The code clips each step at step_clip · step_decay^iteration central_server.py. Algorithm 2 has no decay, but the hyperparameter table lists step_decay alg. 2 tab. 1. The code also skips a Newton update when |hessian| < 1e-8 central_server.py.
• Matching defaults. These match: EM iterations 60, step_clip 1.0, both L2 terms 1e-2, tol 1e-4, the E-step tie formula, the ν_tie update, the ν_t update, and the initialisation of ν and ν_tie tab. 1 alg. 2 central_server.py central_server.py central_server.py. Random initialisation uses standard deviation 0.1 central_server.py.
• Partial-success defaults in code. The code uses σ_partial = 0.3 and partial weight 1.0 central_server.py. The paper does not state these values app. B.3.
• Progress scale. The paper's progress score runs from 0 to 100 §3.1. The server stores the submitted partial_success as a float with no range check visible in the code, and the partial-success model expects values in [0, 1] central_server.py central_server.py.
• Leaderboard transform. The public board reports 200 × θ + 1500 as an Elo-like score, with the standard deviation scaled the same way central_server.py central_server.py. The paper does not describe this scale.
• Exclusions. The public board drops two policy names, PI0 and PI0_FAST, and drops sessions without a VALID_SESSION note central_server.py central_server.py. The paper's pool includes π0 variants §5.1.
• Session timeout. Stale sessions are cancelled after 0.5 hours central_server.py central_server.py. The paper says only "a fixed timeout" for a rollout §3.1.
• Credit mechanics. The code's default base credit is 10 config.py, tops it up weekly with at most one week carried over central_server.py, and deducts the credit from the owner of the sampled first policy central_server.py.

Pitfalls

• Ranking scale. θ is a log-ability with no fixed unit, and the paper reports no Elo-style scale for the task-aware model. Any published score needs a stated transform, such as the code's 200θ + 1500 central_server.py.
• No uncertainty for the EM ranking. The paper describes no standard errors for the task-aware model. The code's Bradley-Terry fit gets standard deviations from the inverse observed Hessian central_server.py. Do not carry those over to the EM model without deriving them.
• Ambiguous initialisation. "N(0, 0.1)" does not say whether 0.1 is a variance or a standard deviation. The code uses a standard deviation central_server.py.
• Centring axis. The paper says the parameters are centred to zero mean but does not say over which axis for ψ. The code centres each policy's offsets across buckets central_server.py.
• Clipping. Step clipping bounds each Newton step. The decay schedule is not in Algorithm 2 but is in the code central_server.py.
• Local optima. The paper gives no restart count or seed count for EM. Because EM can stop at a local optimum, results depend on the initialisation and should be checked with several seeds.
• Rollouts and comparisons are different counts. The 612 pairwise comparisons §1 are not the 4,284 scored rollouts, which include the oracle's exhaustive scoring §5.1.
• Preference and progress disagree. In 11% of trials, the preference label disagrees with the progress scores app. D. Progress-only rankings lose this information §5.2.
• Drift results are simulated. The distribution-shift result is built by rerunning the ranking with artificially drifting tasks and policies app. B.6, not from observed drift.
• Model names are inconsistent. Categorisation is said to use GPT-4.5 §3.3, but the accuracy table names GPT-4o §5.4. The report prompt names o3-2025-04-16 app. E.
• LLM reports are only partly checked. The paper checks claims about one policy against win rates §5.4 fig. 9. It does not report a systematic check of the other reports.
• Limitations the authors name. Results cover only DROID; controlled single-factor experiments are not possible in this design; adversarial evaluators are not studied; and rankings may over-optimise once they become a target §7.

Lineage

• Builds on: Bradley-Terry §3.2, the Davidson tie extension app. B.1, Elo §5.2, the crowd-sourced arenas for chat and image models cited in the introduction §1, the DROID platform and dataset §4, and the standard DROID evaluation protocol §5.1.
• Compared against: the standard DROID protocol ("Regular"), progress-based ranking ("PROG"), a plain Bradley-Terry fit, Elo, and the exhaustive oracle §5.1 §5.2.
• Later code change: the public leaderboard code replaces the task-aware fit with Bradley-Terry plus Davidson ties (see Paper vs code). This is a code change, not a later paper.