Principles and Guidelines for Evaluating Social Robot Navigation Algorithms

In plain words

Robots that move through crowds are judged on more than whether they reached the goal. People also care whether the robot kept a safe and comfortable distance, moved in a way others could read, and fit the place it was in: a hospital corridor and a busy office call for different manners. This paper argues that the field needs shared rules for testing such robots, in the same way that computer vision and language research have shared benchmarks.

The authors gathered many researchers and wrote down what a socially navigating robot should respect, and how to measure it. They suggest a common set of simple, computable measurements, a way to sort any metric by what it measures and how it is scored, and advice on how to design surveys, scenarios, benchmarks, datasets and simulators so that results can be compared. They also propose a shared software interface for computing these measurements across simulators and robots. The paper runs no experiments of its own; it is a set of principles and guidelines.

TL;DR

• The paper defines a socially navigating robot through eight principles that cover physical harm, annoyance, readability of intent, courtesy, norm-following, modelling other agents, taking initiative, and the surrounding context §III-B.
• Metrics are sorted on three axes: algorithmic or surveyed, non-social or social or all-encompassing, and stepwise or taskwise. Table I lists suggested hand-crafted metrics §VI-A tab. I.
• Eight metric guidelines (M1–M8) and six benchmark criteria (B1–B6) are given, for example: report a battery of metrics, report distributions, and ground social claims in human data §VI-E §VIII-D.
• A common metrics API is proposed. The linked repository, as read at the cited commit, holds one Gym wrapper file with no metric implementations §X-D main_api.py.
• No experiments are reported; all outputs are principles, taxonomies and recommendations.

Problem

Social navigation is hard to evaluate fairly. The authors state that, unlike traditional navigation where SPL is a common reference, social navigation has no single agreed metric, and that surveyed human ratings, the gold standard, are costly, hard to scale and sometimes disruptive §VI-A §VI-C. Benchmarks differ in scope, rarely ground their evaluations in human data, and often lack coverage of edge cases §VIII-C. The paper's stated premise is that a shared language and shared criteria would let results from different simulators, robots and datasets be compared §I §X-D. It does not claim one protocol can fit all research goals; it gives a method for choosing an evaluation §IV.

Method

The paper is a framework, not an algorithm. The parts an implementer or evaluator can use are listed below, in the order of the paper.

1. Definition and principles §III-B

Social behaviour is treated as extra objectives on top of the main navigation goal. The first seven principles are objectives; the eighth, context, sets their relative weights §III-B (context figure, §III-B).

• P1 Safety: avoid harm to humans, other robots and the environment, not only collisions with people §III-B.
• P2 Comfort: avoid annoyance and stress. Factors named include distance to humans, not cutting people off, and motion naturalness. Proxemics distances are cited as an example: intimate at 0.45 m and personal at 1.2 m §III-B.
• P3 Legibility: behave so that other agents can infer the robot's goal. Announcing intent is mentioned as an option §III-B.
• P4 Politeness: physical politeness (not cutting people off) and communicative politeness (signals such as "excuse me") §III-B.
• P5 Social competency: follow social, political and legal norms for shared space. Some competencies are conventions rather than optimisation targets §III-B.
• P6 Understanding other agents: predict and accommodate others' motion and goals, including whether people are in conversation. Interaction potential should be minimised or maximised depending on the task §III-B.
• P7 Proactivity: take the initiative to prevent and resolve conflicts, for example offering a doorway or an elevator §III-B.
• P8 Contextual appropriateness: context has five named forms: cultural, diversity, environmental (geometric and operational), task and interpersonal §III-B (context figure, §III-B). Context should be defined well enough that others can judge applicability §III-B.

2. Metric taxonomy §VI-A (taxonomy figure, §VI-A)

Each metric should be classified on all three axes; the paper proposes a three-letter code such as NHT for Success Rate (taxonomy figure, §VI-A).

• Nature: surveyed (questionnaire-based in situ or ex situ, or sensor-based) or algorithmic (hand-crafted, or learned with distribution-specific or distribution-agnostic models) §VI-A.
• Variable modelled: non-social, social, or all-encompassing §VI-A.
• Temporal scope: stepwise (one score per time step) or taskwise (one score per task). The paper considers taskwise metrics the most suitable for benchmarking, and notes that stepwise scores can be averaged into taskwise ones, although not every moment weighs equally §VI-A.

3. Recommended hand-crafted metrics §VI-D tab. I

The paper recommends a common subset of hand-crafted metrics: a traditional set to check that social systems do not regress, and a second set for quality and socialness tab. I. The table gives a definition, class, parameters, unit, range and source for each. The condensed version below keeps the definitions in our own words; check tab. I for the full list and sources.

Class codes are read from the taxonomy in §VI-A: N = non-social, S = social, H = hand-crafted, T = taskwise §VI-A tab. I. The "Space compliance" description above restates the table in our own words; the table gives the exact definition. Personal Space Compliance is SC with a 0.5 m threshold tab. I.

# Space compliance (SC) for one trajectory  [tab. I]
# Input:  robot_xy (T, 2); human_xy (T, H, 2); threshold_m
# Output: fraction of steps with the nearest human inside the threshold

def space_compliance(robot_xy, human_xy, threshold_m):
    # Distance from robot to nearest human at each step  [tab. I] (SC)
    nearest = min_over_humans(distance(robot_xy, human_xy))   # (T,)
    # Count steps where the nearest human is inside the threshold  [tab. I]
    violating_steps = count(nearest < threshold_m)
    return violating_steps / len(nearest)

The table's SC row is defined as a ratio whose range is [0, 1); the code above is our reading of "ratio of the trajectory with minimum distance under a threshold", and the paper does not give a formula or the sign convention. The paper gives no implementation for any of these metrics.

4. Metric guidelines M1–M8 §VI-E

• M1 Robustness: report as many standard metrics from tab. I as feasible.
• M2 Simulation first: use algorithmic metrics in simulation to filter policies before deployment.
• M3 Parameterise in context: state the parameters of metrics such as FP or SC and the context in which they were used §VI-D.
• M4 Learned metrics: where human-data-trained metrics exist, use them to iterate on behaviour, or as acceptance tests §VI-E.
• M5 Validated surveys: use validated instruments once the system is robust enough to test socially §VI-E §VI-D.
• M6 Consistent setup: environment complexity, subject selection, robot familiarity and survey fatigue can all distort metrics §VI-E.
• M7 Iterate: analyse experiments and refine scenarios and surveys iteratively §VI-E.
• M8 Report in depth: report a battery of traditional, learned and surveyed metrics, both stepwise and taskwise, with histograms or other distributional information, not single point estimates §VI-E §VI-D.

5. Survey design §VI-D

• Test the survey in iterations for test-retest reliability, construct validity and bias §VI-D.
• Keep surveys short, because longer surveys produce less reliable answers, and avoid issuing them too often §VI-D.
• Use validated response scales, and phrase each item as a direct question with a "not applicable" option. The paper's example is "How safe was the robot's motion?" with Unsafe to Safe and Not Applicable §VI-D.
• Use balanced conditions and tools such as ANOVA and Cronbach's alpha to check items §VI-D.
• Do not use "significance" unless a proper statistical test was run §VI-D.

6. Benchmark criteria B1–B6 §VIII-D

A benchmark should (B1) evaluate social behaviour, not only navigation metrics such as SPL or success rate; (B2) provide quantitative metrics on several dimensions; (B3) include baselines, at minimum a worst-case planner and, if feasible, an oracle such as human demonstrations or a state-of-the-art method; (B4) be efficient, repeatable and scalable, with at least 30 samples for real robot trials as a rule of thumb; (B5) ground human evaluations in human data; and (B6) use validated instruments §VIII-D.

7. Research-type lifecycle §IV-B §IV-C (lifecycle figure, §IV-C)

The paper describes five research types, from field studies through staged social interactions, and a lifecycle: data collection, issue discovery, laboratory experiments, scenario development, social benchmarking and benchmark challenges §IV-B §IV-C (lifecycle figure, §IV-C). Scenario validity is discussed in terms of ecological and construct validity §IV-B.

8. Simulator and real-world guidelines

Simulator guidelines S1–S9 cover standard APIs, standard metrics, extensibility, dataset generation, benchmark creation, human labelling, common robot morphologies, detailed pedestrians, and behaviour authoring including ORCA and recorded playback. Simulators should also be validated periodically §X-E. Real-world guidelines R1–R3 cover safety (experiment monitors, safety layers), respect for participants (consent, IRB approval where required, privacy review), and clear scientific objectives §IV-D.

9. Common metrics API §X-D (data-flow figure, §X-D)

The proposed API has three parts: input data (pedestrians with at least a pose per time step, the robot under test with observations, actions and resulting trajectory, and static and dynamic obstacles), metric computation in shared library code that is extensible, and a defined output format for downstream tools §X-D. The planned reference implementation includes a JSON Schema for input with generators for ROS and GymCollisionAvoidance, reference metric libraries in C++ and Python, and a JSON Schema for output. The paper says these are in development §X-D.

Hyperparameters

The paper has no training hyperparameters. The numeric values it states are metric thresholds and study-design rules:

Results

No experiments. The paper's outputs are the principle set §III-B, the taxonomy §VI-A, the recommended metric list tab. I, the eight metric guidelines §VI-E, the benchmark criteria §VIII-D, the simulator guidelines §X-E and the API proposal §X-D. The authors report a consensus from a symposium and its position papers; the paper does not quantify agreement §XI §Contributions.

The paper also surveys existing benchmarks (for example SocNavBench, SEANavBench, CrowdBot and the iGibson challenge) and datasets against its criteria §VIII-B §VIII-C. Those comparisons are qualitative, and the paper gives no benchmark scores of its own.

Paper vs code

The code is the only associated artefact, and it is a prototype rather than the reference implementation the paper describes.

• Paper: a JSON Schema for input, reference C++ and Python metric libraries and a JSON Schema for output, with ROS and Gym bridges §X-D. Code: the repository at fc3d9c74a5bd04ed4e1e9551a5416d133a0dc878 contains a single file, main_api.py, with no schema, no C++, no ROS bridge and no metric library main_api.py.
• Paper: the API computes the metrics of tab. I §X-D. Code: get_metrics returns only the distance from the robot to the goal main_api.py. None of the tab. I metrics are implemented.
• Code: step reads the new state but has no return statement, so it returns None rather than the Gym tuple (observation, reward, done, info) main_api.py.
• Code: reset only calls the wrapped environment and does not clear the wrapper's own state, including self.time main_api.py main_api.py.
• Code: the body of render has indentation that differs from its for line (lines 64–66 use 8, 11 and 12 spaces). This is probably an IndentationError, so the file likely does not import as shipped. Not run here main_api.py.
• Code: no LICENSE file. The paper links the repository but does not state a licence, so reuse of the code is not granted by default.

No discrepancies in the numbers were checked, because the paper contains no numerical results beyond the thresholds in the Hyperparameters section.

Pitfalls

• Metric units and definitions. tab. I prints the unit of TTC as metres, although the description is a time to collision. The paper does not say which is intended. Check the unit before implementing it.
• Space compliance formula. The table defines SC as a ratio of trajectory with a distance threshold, but gives no formula, no tie-breaking when several humans are present, and no treatment of time steps where no human is in view. The pseudocode above is one reading, not the paper's definition.
• Thresholds are not given. The timeout, FP, ST and collision-count thresholds are parameters with no values. Results from different papers using different values are not comparable, as M3 notes §VI-E.
• Averages hide outliers. The paper asks for distributions and histograms because safety depends on consistency §VI-D. Reporting only means may mislead.
• Survey response burden. Longer or more frequent surveys degrade answers, so the paper advises fewer, validated items §VI-D.
• Context is not defined in machine-readable terms. The paper concedes that defining context precisely for a robot is an open problem §III-B.
• Learned metrics are not ready. The paper says learned metrics are not ready for adoption, and that no task-wise learned metric existed at publication §VI-D. Treat them as research.
• Repository state. The linked repository was last pushed in May 2023 and has no licence main_api.py. Do not depend on it as a metrics library; implement the table definitions from the paper.
• Version. The licence and content were recorded from arXiv v4 (updated 2023-09-19). The licence recorded by fetch for v4 is CC BY 4.0 [raw/arxiv/2306.16740.json].
• Authorship. The author list is a symposium-based consortium, and the contributions section describes working-group roles, not a single method owner §Contributions.

Lineage

• Builds on: the metric reviews the paper cites, Gao et al. (2021) and Mirsky et al. (2021) §VI-A; and the Success weighted by Path Length metric of Anderson et al. (2018), which the paper uses as the traditional reference §VI-A tab. I.
• Compared against: existing social navigation benchmarks and challenges, including SocNavBench, SEANavBench, CrowdBot and the iGibson challenge, judged against the paper's own criteria §VIII-B §VIII-C.
• Superseded by: not stated.