Breaking Down the Barriers: Investigating Non-Expert User Experiences in Robotic Teleoperation in UK and Japan

In plain words

Controlling a robot arm by hand is hard for people who do not do it for a living. This paper looks at how people fare when they steer a robot arm through a computer copy of the robot, called a digital twin. The user moves a handheld or worn controller, the copy follows, and the real arm follows the copy.

The authors ran the same simple job on three arms: pick up three cubes and stack them into a tower, as many times as possible in ten minutes. Participants came from the United Kingdom and from Japan. The arms and their grippers were different, so the study compared the arms, the controllers and the two groups of people together.

Afterwards people filled in short questionnaires about how hard the task felt, how much effort and frustration they felt, and how much they trusted robots. The main result is that the arm with the longest reach and the most stable motion let people build the most towers, and it caused the least mental strain. The authors also found that the Japanese group reported more trust in robots, which goes against an earlier cross-country study. Because the countries and the robots differed at the same time, the authors do not claim that the robot alone explains every difference.

TL;DR

• User study with 74 participants (37 in the UK, 37 in Japan) teleoperating three arms through the TELESIM digital-twin framework §I §IV §IV-A.
• The task is stacking three cubes into a tower within 10 minutes; the outcomes are tower counts, a single-ease question, raw NASA-TLX and NARS S1 items §IV §V-A.
• The UR5e (Japan) produced the most towers, a mean of 10.50 per user, against 3.25 for Baxter and 1.03 for UR3 §V-A fig. 6.
• The UR5e had the lowest mental workload, and the UR3 with its Senseglove had the highest effort and frustration §V-B fig. 11.
• Author-reported. Robot, gripper, controller, country and recruitment all change together, so the results describe the setups, not the arms alone §IV-A §VI (see Pitfalls).

Problem

Teleoperating robot arms is hard for non-experts. Physical and mental fatigue are known risks in long sessions, and the skill needed for good teleoperation is not well defined §I. Earlier work by the same group showed that non-experts could operate arms through TELESIM, but it did not measure physical or cognitive strain or trust in robots §I §II-A.

The authors state that the literature mostly studies shared autonomy, uses small groups (8 to 23 people), and seldom compares direct teleoperation across several robots §II-A. They treat NASA-TLX, a six-dimension workload questionnaire, as the predominant instrument for teleoperation workload §II-B. For trust, they use the NARS questionnaire, a scale for negative attitudes toward robots, which they treat as a proxy for trust §II-C §IV §V-C.

Method

System (TELESIM). A controller produces a 3D pose. TELESIM moves a cube in a digital twin built in NVIDIA Isaac Sim, plans a real-time path for the arm with Riemannian Motion Policy (RMP) while avoiding collisions, and sends the resulting joint states to a real arm through ROS2 §III §III-B fig. 2. Robots are loaded from URDF files, so a new arm can be added without changing the core framework §III-B.

Controllers. Three control devices are used. A SteamVR controller drives the Baxter and the UR5e. A Senseglove with an attached Vive tracker drives the UR3, and the glove maps the index finger and thumb to the two fingers of a Yale T42 gripper §III-A §IV-A. A finger reading is taken from the joint angles of the fingers, aggregated and scaled into the open-to-closed range of the gripper motor §III-A. Haptic feedback was planned but not implemented for this paper §III-A.

Robots and grippers. Baxter (Rethink linear gripper), a UR3 with a modified Yale T42 gripper, and a UR5e with a Robotiq 2F gripper §III-C §IV-A tab. 1. The UK runs used Baxter and UR3; the Japan run used the UR5e, with the URDF changed to load the UR5e and the Robotiq gripper §IV §IV-A.

Task. Participants stand with their backs to the headset, which defines the world origin. Three cubes start in a triangle and must be moved to a central marked position to form a tower. The session lasts 10 minutes §IV fig. 1 fig. 4 fig. 5.

Protocol. Participants first complete a demographics questionnaire with 5-point Likert items (age, gender, and experience with VR, robots and wearable gaming devices). After each robot run they answer the Single Ease Question and the raw NASA-TLX, which uses a 7-point scale instead of the 21-point scale. At the end of the experiment they answer five items from the S1 subscale of NARS §IV.

# Session protocol  [§IV]
# Input:  one participant, one robot
# Output: tower count, SEQ score, raw NASA-TLX, NARS S1 answers

def run_session(participant, robot):
    # Brief demographics, 5-point Likert items  [§IV]
    demographics = ask_demographics(participant)
    # Stack the three cubes as many times as possible  [§IV] [§V-A]
    towers = teleoperate(robot, participant, minutes=10)
    # Single Ease Question, then raw NASA-TLX on a 7-point scale  [§IV]
    seq = ask_single_ease_question(participant)
    nasa_tlx = ask_raw_nasa_tlx(participant, scale_points=7)
    return towers, seq, nasa_tlx

# NARS S1 at the end of the experiment  [§IV] [§V-C]
# Input:  participant, after all robot runs
# Output: answers to five S1 items

def ask_nars_s1(participant):
    # Five items from the S1 subscale only; the other subscales are not used  [§IV]
    items = nars_s1_items()
    # Asked once per participant, not once per robot  [§V-C]
    answers = [ask(participant, item) for item in items]
    return answers

Tower and stability metrics. The per-user tower count is the main outcome. Three ratios are derived from the pick, place and collapse actions of each user fig. 8:

# Placing, collapse and still-in-place rates  [fig. 8]
# Input:  pick, place and collapse action counts for one user
# Output: three ratios, each divided by the number of picks

def stability_ratios(picks, places, collapses):
    # Share of picks that were followed by a place  [fig. 8]
    placing_rate = places / picks
    # Share of picks that led to a collapse  [fig. 8]
    collapse_rate = collapses / picks
    # Placements that did not collapse, per pick  [fig. 8]
    still_in_place_rate = (places - collapses) / picks
    return placing_rate, collapse_rate, still_in_place_rate

Analysis. Tests are reported with p-values and significance levels, but the paper does not name the tests, the correction for multiple comparisons, or the normality check §V-A §V-B §V-C.

Hyperparameters

The paper is a user study, so "hyperparameters" are protocol and setup values. Each row cites where the value is stated; where the code is the only source it is marked.

Results

Author-reported. Setup: 74 non-experts, UK (Baxter, UR3) and Japan (UR5e), 10-minute tower task §IV §V. The text gives the means and p-values below. NASA-TLX subscale values and the stability ratios are shown only in figures, so they are cited but not tabulated here.

Paper vs code

Official repository: cvas-ug/telesim_pnp at commit f237cc8 (MIT). The paper links no TELESIM repository. The search ran in this order: links in the paper (only third-party plugin repositories and the Universal Robots driver), the abstract, the IMMERTWIN project page (it links only its own repository), then a GitHub repository search for "telesim", which returned this repository under the lab's organisation. The LICENSE names both authors LICENSE.

• Only the master package is checked out. The tracked files are a README, LICENSE, .gitmodules, submodules.sh and a docs image. The ROS1 and ROS2 packages are empty submodule entries in this commit, so the Isaac Sim worlds, the controllers, the VR publisher, the T42 gripper code and the Senseglove integration cannot be read or cited from this clone .gitmodules .gitmodules. Those submodules point to a personal account (Ubb90) rather than the organisation .gitmodules .gitmodules.
• The paper's implementation claims are unverified. RMP settings, joint limits, the Senseglove scaling, gripper timing and the Isaac Sim worlds are described in the paper but live only in the missing submodules. The paper states no numeric RMP parameters, so none can be checked.
• No analysis code or survey data. The tracked file list has no NARS, NASA-TLX, SEQ or tower-count analysis and no survey data. The statistics in the paper cannot be reproduced from this repository.
• UR5e is a separate package. The README says the master package's UR setup needs ur_type set to ur3, and that UR5e with the Robotiq gripper needs a different package, ur_robotiq README.md .gitmodules. The Japan setup therefore relies on a submodule that is not in this commit.
• ROS versions. The README names ROS1 Noetic, ROS2 Galactic and an Isaac Sim 2022.2.0 path README.md README.md. The paper does not state versions, so this is an addition, not a discrepancy.
• Stale placeholders. The README still has a TODO where the paper link should be, and its clone URL uses the previous account README.md README.md.

No conflicts between the paper's stated values and the code could be checked, because the controller values are in the submodules.

Pitfalls

• Confounded comparison. The UR5e was tested only in Japan, while Baxter and UR3 were tested only in the UK §IV. Recruitment differed too: Japan used an agency and paid participants, while UK participants were drawn from academia and unpaid §IV. The arm, its gripper, its controller and the country all change together, so "the UR5e built more towers" cannot be separated from "Japan" or from the gripper §IV-A §VI. The authors say this themselves §VI.
• Within- versus between-subject design is not stated. The NARS was asked once and its results for Baxter and UR3 were combined, and the authors say a second NARS run after the first robot would have helped §V-C. This implies that UK participants used both robots, but the order, counterbalancing and per-robot N are not stated. Implement only the design the paper states.
• Statistics are under-specified. Tests, normality checks and multiple-comparison corrections are not named §V-A §V-B §V-C. Significance thresholds are inconsistent: some differences are called non-significant at P > 0.1 and others significant at P < 0.1, and SEQ is called non-significant at P > 0.01 §V-A. Treat P < 0.1 results as weak.
• Tower timing does not add up. The text says the UR5e participant with 25 towers averaged about 40 seconds per tower §V-A. Twenty-five towers at 40 seconds each would take 1000 seconds, but the run is 10 minutes (600 seconds) §IV §V-A. Either the average is over a different window or the number is wrong. The statement is not reconciled in the paper.
• Trust is a proxy. NARS measures negative attitudes toward robots, and the paper treats the S1 items as trust §II-C §IV. The direction is easy to get wrong: the caption says a low score means more trust, while the text says the Japanese group showed "higher trust" on Q.1, Q.2 and Q.5 §V-C fig. 12. Check the sign before comparing with other studies. NARS response options are not stated.
• Colour legends conflict. In Figure 11 blue is Japan (UR5e), but in Figure 12 blue is UK fig. 11 fig. 12. Confirm the legend in the rendered figure before using it.
• Dimension names differ. Section II lists Temporal Demand, while Section V reports Pace in its place §II-B §V-B.
• Hypothesis about demographics overstated. The authors say the demographics hypothesis "can be discarded" §V-B. The design cannot isolate demographics from robot and country, so treat it as weak.
• Sample size wording. The abstract says 37 UK participants, the introduction says "over 70", and the contributions say 74 [abstract] §I. Use 74 (37 + 37) §IV.
• Age wording. The paper says the two groups were recruited "within the same age range", but the mean ages are 25.32 and 27.81 with SDs of 6.26 and 7.93 §IV.
• Gripper spec mismatch. The text gives the Robotiq as 2F-140, while Table I gives 2F-185 §IV-A tab. 1.
• Unreported analysis details. The SEQ scale range, the NARS response scale and the training length are not stated §IV §VI. The overall NASA-TLX score is not reported, only subscales §V-B.
• Section and figure numbers. Citations here follow the IEEEtran layout (Roman section numbers, lettered subsections) and the order of figure environments in the LaTeX source. The compiled PDF was not read, so check numbering against it if exact page-level citations matter.
• Figure captions. The Figure 9 caption says "left" for both the Baxter and UR3 panels fig. 9. Use the figure, not the caption, for panel identity.

Lineage

• Builds on: TELESIM, Audonnet et al., ICRA 2024, DOI 10.1109/ICRA57147.2024.10610935, which introduced the framework and the UK setup with 37 participants §III §IV §I.
• Related, from the same group: IMMERTWIN (arXiv:2409.08964), cited for adding functionality on top of TELESIM §III-A. A later cross-country comparison by Bartneck et al. (2007) is cited for the NARS trust results and contradicted here §V-C.
• Compared against: the NASA-TLX literature for teleoperation (Lin et al.; Parsa et al.; Gottardi et al.) and the Single Ease Question §II-B §IV.
• Superseded by: not stated in this paper. The authors plan a deeper study of the Senseglove and T42 gripper on the UR3 §VI.