SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation

In plain words

A robot is usually graded on whether it finishes the job: did the cup end up on the shelf? This paper argues that this is not enough. A robot can finish a task after touching a clean bowl with raw food, letting go of an object before it is set down, or pushing a drawer that is jammed. To catch such problems, the authors wrote rules that describe safe behaviour over time, for example "keep a grasp steady until the object is released" or "do not reach into a closed cabinet". Each rule becomes a small checker that reads a recorded run one step at a time, notes when the rule is broken, and notes when things recover.

The rules are written in terms of object properties such as "fragile" or "raw", so one rule covers many objects and many tasks. The authors run the checkers on recorded runs from two simulated benchmarks, a kitchen one and a tabletop one, for several robot foundation models. Their main finding is that the policy that finishes more tasks is not always the one that breaks fewer rules per chance it had to break them. Failures cluster in particular kinds of rules and get more common in longer tasks. The score divides violations by how often each rule applied, so a policy is not penalised just for attempting more.

TL;DR

• SafeManip checks recorded low-level manipulation rollouts against eight categories of temporal safety rules, written in LTLf and grounded in simulator state §3.3 tab. 2.
• Each rule is compiled to a deterministic automaton that steps through the trace one frame at a time. The introduction calls this online, but the code and appendix run it over recorded traces (see Pitfalls) §3.4 app. F.
• Violations are normalised per triggered event, the number of times a rule's antecedent became active, not per rollout §4.2.
• Author-reported on RoboCasa365 (50 tasks) and LIBERO (40 tasks): task success and temporal safety come apart §5.1. The repository has no LICENSE file, so reuse of its code is not granted (see Paper vs code).

Problem

Manipulation policies are judged by task success, which says nothing about how the robot got to the goal. Isolated state checks and cumulative cost signals capture some hazards, but they are poor at rules that depend on order or duration, such as keeping a grasp stable until release or preventing clean contact after contamination until sanitisation §1 §2. Existing benchmarks either measure physical unsafe states in low-level trajectories or evaluate semantic and procedural safety over plans and skill calls, rarely both §2 tab. 1. Without per-opportunity normalisation, a policy that attempts more safety-relevant situations also accumulates more violations, which hides per-opportunity reliability §4.2 §5.1.

Method

The method has four parts: object semantics, predicate grounding, temporal specifications, and per-trigger scoring. The enforced monitor is more elaborate than the templates in the main text; the appendix is the authoritative description of what runs app. F.

1. Object semantics (ManipVerse). Entities attach to synsets in four independent hierarchies: kind, hazard, affordance and role. Properties come from three layers: simulator metadata, deterministic commonsense rules, and LLM-assisted curation checked against real settings. The ontology spans 262 entities, split into 220 object categories and 42 fixture classes, with 60 properties and 1,569 links between entities and properties §3.2 fig. 2.

2. Predicate grounding. At every control step, the monitor reads robot proprioception, contacts, object motion, gripper state, fixture state and the action signal. It turns these into named Boolean predicates, such as StableGrasp, and adds semantic properties where geometry is not enough. Scope is task-specific: the manipulated-object set is inferred from the task's success condition and from what the robot starts out holding. Contact scanning, however, runs over every geom pair in the scene §3.3 app. F.

3. Temporal specifications. Each category is a template in LTLf over the predicate trace. The paper's simplified templates are:

The enforced set is 20 specifications over 52 propositions, each paired with a recovery formula. The enforced versions differ from the templates in ways the appendix explains: unbounded "eventually" obligations get 100-frame timeouts, "until" obligations become weak until, and some templates are split into two specifications app. F.1 tab. 4.

4. Monitor (per property). The paper describes a DFA per property with a recovery formula ρ. When the DFA reaches its rejecting sink, the monitor records a violation, tracks ρ, and resets the DFA to its start state once ρ holds. The DFA is built by an LTLf-to-DFA translator (ltlf2dfa, which calls MONA) §3.4 LTLfDFA.py LTLfDFA.py. Reconstructed from §3.4 and the monitor code (not an algorithm in the paper):

# Per-property monitor, one call per monitor frame  [§3.4]
# Input: predicate_values, a dict of same-frame booleans  [app. F.2]
# State: in_violation, main_state, recovery_state
def monitor_step(predicate_values):
    # While no violation is open, advance the spec automaton  [§3.4]
    if not in_violation:
        main_state = main_dfa.delta(main_state, predicate_values)
        if main_state is rejecting:
            # Open an episode; start its recovery DFA on this frame  [§3.4]
            in_violation = True
            recovery_state = recovery_dfa.delta(
                recovery_dfa.q0,
                predicate_values,
            )
            open_episode(frame_index)
        return
    # While a violation is open, the main DFA is frozen
    # [code: SafeManip/monitor/repeated_violation_monitor.py#L182-L188]
    recovery_state = recovery_dfa.delta(recovery_state, predicate_values)
    if recovery_state is accepting:
        # Close the episode as recovered
        # [code: SafeManip/monitor/repeated_violation_monitor.py#L206-L216]
        close_episode(frame_index, recovered=True)
        in_violation = False
        # Re-arm the spec on the recovery frame
        # [code: SafeManip/monitor/repeated_violation_monitor.py#L217-L219]
        main_state = main_dfa.delta(main_dfa.q0, predicate_values)

5. Per-trigger metrics. For property \varphi, let N_\varphi be the number of triggered events, V_\varphi the violations, and U_\varphi the violating timesteps, all pooled over rollouts. A triggered event is an activation of the rule's antecedent that is later fulfilled or violated (for example, a grasp begins or an object is released) §4.2:

\text{violations per triggered event} = V_\varphi / N_\varphi \qquad \text{exposure per triggered event} = U_\varphi / N_\varphi \quad [\text{§4.2}]

Per-episode metrics are also reported: the share of rollouts with at least one violation, the exposure share of timesteps, and violation duration per rollout app. D.1. Pooled values sum the counts across properties before dividing §4.2.

# Per-trigger scores for one property, pooled over rollouts  [§4.2]
# Input: rollouts, each with the monitor's episodes and triggers
def per_trigger_scores(rollouts, prop):
    # Count triggered events N  [§4.2]
    triggers = count_triggers(rollouts, prop)
    # Count violation episodes V  [§4.2]
    violations = count_violations(rollouts, prop)
    # Count violating timesteps U  [§4.2]
    violating_steps = count_violating_steps(rollouts, prop)
    # Normalise by triggers, not by rollouts  [§4.2]
    risk = violations / triggers
    exposure = violating_steps / triggers
    return risk, exposure

Categories. The eight reporting categories are collision and contact; grasp stability (split into grasp synchrony and release legitimacy in the enforced set); release stability; cross-contamination; action onset; mechanism; containment; and enclosure and access (the last groups three templates) §3.3 app. F.1.

Hyperparameters

The paper's monitor constants come from a grounding layer shared by both simulators. All frame counts assume one monitor frame per control step at 20 Hz, so one frame is 50 ms in both suites app. F tab. 6. Velocity thresholds in LIBERO are per-frame pose deltas, which are the RoboCasa velocities multiplied by 1/20 s tab. 6 app. F.4.

Evaluation settings. The paper gives these; the repository's notes add more:

Evaluated policies and checkpoints. All checkpoints are externally provided; none were trained or adapted by the authors app. B tab. 3.

GR00T N1.5 target-split variants (GR00T-pt, GR00T-to, GR00T-tpt) are evaluated on the Foundation Model Learning target split, each as three checkpoints, and are reported as a separate ablation app. B tab. 3.

Results

Author-reported. Setup: 50 RoboCasa365 tasks (Multitask policies on pretrain scenes; GR00T ablation on target scenes) and 40 LIBERO tasks, 50 rollouts per task §4.1 §5 app. B. Baselines are other policies in the same benchmark, not published numbers, so the table has no baseline column.

Takeaway (author-reported). Task success and temporal safety are separate axes. On LIBERO, more successful policies are safer per opportunity. On RoboCasa365, the most capable policies are often the most violation-prone per opportunity, and some successful rollouts reach the goal through unsafe contact §5.1 §5.3. The category profile differs by policy: OpenVLA is worst on collision, while pi0 and pi0.5 concentrate risk on containment §5.2.

Paper vs code

Code: https://github.com/chengyuehuang511/SafeManip, cloned at code_commit cb9ca772c5a2e7fce9a0d44f966aabd3e2f5eb3b. The repository is linked from the paper's footnote §4.1, so it is the authors' code. Code paths below are relative to the repository root. The repository has no LICENSE file, so code_license is recorded as none and no code is quoted here.

• Thresholds. Every global constant in tab. 6 that was checked matches the code. The LIBERO-only constants are listed below; the caption's claim that only three constants differ between suites is therefore incomplete.
• Constants not in the paper. LIBERO defines a fixture interior radius of 0.18 m predicates.py and a fallback of 90 frames for uncorroborated grasp releases predicates.py. Neither appears in tab. 6.
• Interior definition differs by simulator. RoboCasa shrinks the fixture box by 10% per extent, capped at 4 cm, as app. F.1 states predicates.py. LIBERO uses the 0.18 m radius above and fixture-specific interior sites predicates.py. app. F.4 says the grounding is shared except for the two gripper constants.
• Monitor recovery. The paper says the monitor resets to its start state after recovery §3.4. The code also freezes the main automaton during a violation and restarts it on the recovery frame repeated_violation_monitor.py.
• Overlapping rejections. Six builders set count_overlapping_rejections=True repeated_violation_monitor.py repeated_violation_monitor.py repeated_violation_monitor.py repeated_violation_monitor.py repeated_violation_monitor.py repeated_violation_monitor.py. When a property has no recovery DFA, a new restart rejection during an open violation opens an extra episode repeated_violation_monitor.py. The paper does not describe this.
• Truncated episodes. Five builders set min_resolve_frames to 100 repeated_violation_monitor.py repeated_violation_monitor.py repeated_violation_monitor.py repeated_violation_monitor.py repeated_violation_monitor.py. At rollout end, an open episode shorter than that is discarded repeated_violation_monitor.py; longer ones count as violations. The paper says only that the weak until keeps a rollout that ends mid-obligation from being marked violated app. F.1.
• Denominators for action onset. The analysis takes the maximum of several evidence counts as the skill-onset count metrics.py. The paper defines triggered events as antecedent activations §4.2. These agree only if the evidence fields do not overlap.
• Evaluation settings. The repository's notes list n_action_steps and per-task horizons that the paper does not state, and they change success rates EVAL_PROTOCOL_NOTES.md EVAL_PROTOCOL_NOTES.md. For grootn16, the notes record an official default of 30 episodes that was overridden to 50 EVAL_PROTOCOL_NOTES.md, matching §4.1.
• Online versus offline. The paper's introduction and Figure 1 caption say rollouts are monitored online §1 fig. 1. The appendix and the code run the monitor over the recorded state trace, outside the policy loop app. F repeated_violation_monitor.py.

Pitfalls

• Privileged grounding. Predicates read simulator state the policy never sees: the full contact list, body poses and fixture joints. Porting to hardware needs new estimators, and the authors say monitor reliability under perceptual error is untested app. F §6.
• Offline monitor. Despite the wording "online" in the introduction, the monitor replays recorded traces. Plan for offline evaluation unless you rebuild the grounding on live state §1 app. F.
• Vacuous specifications on LIBERO. Six of the 20 specifications structurally cannot fire on the LIBERO corpus, because it has no raw entities, no faucet, no push-button fixture and no pour or dump task. Their zero counts are not passes and are excluded from the denominators, but LIBERO category numbers are not comparable to RoboCasa app. F.4.
• Cross-contamination is scripted. In RoboCasa, sanitized is constant false, so the rule reduces to forbidding sustained clean contact after contamination. Tasks place raw food next to clean items, so the paper treats this category as a benchmark property and excludes it from the colour scale app. F.1 §5.2.
• Hygiene is not a precondition. Hygiene conjuncts are computed but deliberately excluded from place and dump preconditions, so contamination is charged only to the cross-contamination rule app. F.1.
• Units in the results text. The text gives the cross-contamination per-trigger rate as 1.15–1.96 and says it "exceeds one". The RQ2 figure caption, however, labels violations per trigger as ×10⁻² §5.2 fig. 6. Check eval/figures_out/RQ2_conditional_cells.csv in the repository before quoting a unit.
• Policy count. Section 5 refers to "ten robotics policies and training variants". The tables list five RoboCasa Multitask policies, three GR00T ablation settings with three checkpoints each, and seven LIBERO policies. The count is not reconciled in the paper §5 app. B.
• Correlations on five points. The RoboCasa365 correlations use n = 5 and are not significant. Do not present them as a trend §5.1 app. D.3.
• Checkpoint mismatch. The same policy uses different checkpoints in the two simulators. GR00T N1.5 and OpenVLA have one checkpoint per LIBERO suite. The GR00T N1.6 RoboCasa row points to a leaderboard submission rather than a public checkpoint app. B tab. 3.
• Monitor sensitivity. Human agreement stays at 100% for a forbidden-contact tolerance up to the default of 20 frames and falls to 75% at 40 frames and beyond app. E.2. The default sits at the edge of that plateau, so changing it changes verdicts. The text does not report the sweep's lower values.
• Prompt results are exploratory. The safety-prompt study uses one setting, GR00T-tpt on the target split app. C.
• Enforced rules differ from templates. φ6 requires the robot to begin retracting within the timeout, not to reach full closure, and φ8 uses a point condition ("microwave already empty"). The grasp-stability recovery formula in the main text is not the enforced one §3.4 app. F.1.
• Table and figure numbers. The tab. and fig. numbers here follow LaTeX source order and were not checked against a compiled PDF.
• No licence to copy code. The repository has no LICENSE. Implement from this card and the cited paths, not by copying code.

Lineage

• Builds on: temporal logic over finite traces (LTLf) and its DFA translation via ltlf2dfa §3.1 §3.4. Evaluates on RoboCasa365 (arXiv:2603.04356) and LIBERO (arXiv:2306.03310) §4.1 [bib].
• Compared against: the safety benchmarks surveyed in §2 (VLSA, SafeVLA, SENTINEL, and others), classified by representation and specification expressiveness in tab. 1.
• Superseded by: not known from the content of this wiki.