Human-in-the-loop

Overview

Human-in-the-loop (HITL) means a person is part of the cycle in which a system decides or learns. The Hybrid Intelligence framing describes two directions. In one, the AI supports a decision and a human evaluates its output, as when a physician reads a model's CT analysis 2105.00691 §3.1. In the other, humans teach the AI by tuning, debugging, labelling and giving feedback 2105.00691 §3.2. The authors present both as parts of one arrangement in which human and machine learn from each other 2105.00691 §4.

The variants below differ in where the person sits: as the teacher of the model, as the labeller of its uncertain cases, as the judge whose ratings train a reward model, as the one who picks from the model's suggestions, as the one who takes over a running robot policy and records corrections, as the one who decides when a robot in training may act alone, or as the one who steps in, reviews or sets the limits on what an agent may do without asking. Five more variants put a person in a shared-control blend, in the labelled trials that tune a device, in the review of a robot's unsupervised work, in the confirmation of a target before motion, and in the handover of authority at a predicted point. The survey by Vats et al. maps the same ground in build-time and run-time terms 2403.04931 §2 2403.04931 §3.2.

Variants

Human as teacher of the AI

Humans label data, set or tune hyperparameters, debug models and interpret unsupervised results such as clusters 2105.00691 §3.2. The paper says interactive learning often needs many labelled examples from people, and that human input is especially valuable when data is scarce 2105.00691 §4. It also names human game play as a training input for reinforcement-learning robots 2105.00691 §4. The paper measures none of this. Card: Hybrid Intelligence.

Human takes over a running robot policy and corrects it

A person watches a learned policy drive a robot, takes over when needed, and the takeover is recorded as correction data. HACTS is a leader arm whose joints are synchronised with the robot's: while a policy runs, the leader is driven to the robot's joint positions, and a foot pedal switches between autonomous and human control 2503.24070 §3.3. The authors name sudden movements and lost control at takeover as problems of one-way devices, which the two-way sync is meant to address 2503.24070 §1. Corrections are mixed with the original demonstrations and used to retrain the policy 2503.24070 §4.2.

On three real tasks with ACT, 50 base demonstrations plus 50 correction trajectories per setting were compared with 100 expert demonstrations. On the failure-correction setting, HACTS-ACT reached 50%, 80% and 70% on OpenBox, SteamBun and UprightMug, against 40%, 60% and 40% for the 100-demonstration baseline 2503.24070 tab. 2. The data sources are not equal in size, and the authors attribute the gain to correction data covering failed states 2503.24070 §4.2 2503.24070 §4.3. No controlled test isolates that cause. Each cell is 10 rollouts, so each result moves in 10-point steps and has no reported variance 2503.24070 §4.2.

For online reinforcement learning, short human corrections fine-tune a pretrained actor with RLPD. On CloseBin, success over 10 trials rose from 50% to 80% after 10 minutes of offline and 45 minutes of online training 2503.24070 tab. 4. The paper reports no behaviour-cloning success rate for the same task, so the gain is not compared with a baseline there 2503.24070 §4.5. Card: HACTS.

Human gates control while the robot learns (interactive imitation)

Here the robot drives during training, and the human expert takes over only when they judge the state unsafe. The expert's judgement, not a random draw, decides who acts 1810.02890 §II eq. 1. Labels are kept only for the states in which the expert holds control 1810.02890 eq. 2. DAgger, the baseline, mixes expert and learner control with a probability β that is set in advance. The paper says β is hard to pick beforehand and costly to tune on a physical system 1810.02890 §I.

The learner is an ensemble. At each takeover its doubt, the l2-norm of the diagonal of the output covariance, is logged. The threshold τ is the mean of the final quarter of these logged values 1810.02890 §II eq. 3 eq. 4. The paper uses τ only to estimate a permitted set for judging the trained policy, and it proposes but does not test using doubt as a run-time gate 1810.02890 §IV-B 1810.02890 §V eq. 5.

Author-reported results: in simulation, learning curves are said to be faster and more stable than DAgger and behavioral cloning 1810.02890 §IV-A fig. 3 fig. 4. Inside the estimated permitted set, the collision and road departure rates in the paper's table are about 12 and 20 times lower than outside it, as ratios of the reported rates 1810.02890 §IV-B 1810.02890 tab. 1. On a test vehicle with five fixed obstacle scenarios and 5,000 samples, HG-DAgger had no collisions and no road departures. Behavioral cloning had one collision and six departures, and DAgger had one of each 1810.02890 tab. 2. The authors call the test too limited for statistical significance 1810.02890 §IV-C.

The method needs a human who can spot danger and react in real time during training. The paper says it is unsuitable where that is not possible 1810.02890 §II footnote. The gate is a human judgement, so nothing in the paper automates it. The threshold formula in eq. 4 also does not match its own divisor as written, and the paper does not define the index base, so the convention must be chosen by the implementer 1810.02890 §II. Card: HG-DAgger: Interactive Imitation Learning with Human Experts.

Human labels the uncertain cases (active learning)

The survey describes an active-learning round in which the model predicts a label and a confidence for each unlabelled sample. Low-confidence samples go to human annotators, and the model is retrained on the enlarged labelled set 2403.04931 §2.1 fig. 5. The survey gives no confidence threshold and no stopping rule for the loop 2403.04931 §2.1 2403.04931 §2. The survey runs no experiment on this loop, so its benefit is not shown here. Card: A Survey on Human-AI Collaboration with Large Foundation Models.

Human judges outcomes to train a reward model

The person judges what the agent achieved, and a separate reward model learns from those judgements. An RL agent is then trained on the learned reward. The user judges outcomes and does not have to explain how to reach them, because credit assignment moves to the RL algorithm 1811.07871 §3.1. The paper says feedback is not needed on every interaction 1811.07871 §3.1. It leaves the feedback format open: preference labels, demonstrations and natural language are all discussed, and none is picked as the default 1811.07871 §2 1811.07871 §5.5. A reward model that is no longer updated is a known failure, because agents learn to exploit it 1811.07871 §5.1. The paper is a position paper with no experiments. Its two assumptions, that intentions can be learned accurately and that judging is easier than doing, are untested 1811.07871 §1. The authors concede the second fails for outcomes with few possible values, such as yes/no answers, unless an explanation is also wanted 1811.07871 §1. Card: Scalable agent alignment via reward modeling: a research direction.

AI suggests, human chooses (co-pilot)

Tutor CoPilot shows a tutor one suggested reply per teaching strategy. The replies come from GPT-4 with Bridge prompts. The tutor can edit a suggestion, regenerate it, or pick another strategy, and the final choice stays with the tutor 2410.03017 §Method fig. 1. Each activation sends only the 10 most recent messages to the external model 2410.03017 §Method. The survey names over-trust, and missed uncertainty, as the risk of this kind of pattern. Its suggested safeguards are showing confidence, explaining outputs and allowing override 2403.04931 tab. 2. The survey does not test them.

In a randomized trial the tutors were randomized, not the students, with 783 tutors analysed 2410.03017 §Methods. Access to the tool raised the unconditional exit-ticket pass rate by 4.0 percentage points, from 61.7% to 65.7% 2410.03017 tab. 1. The authors report a larger 14-point figure from an instrumental-variable estimate. That figure applies to tutors whose use changed with assignment, not to the tool overall [arxiv:2410.03017 SI: Treatment on the Treated Analysis]. The gain was larger for students of lower-rated tutors 2410.03017 §Results. The authors put the cost at about USD 20 per tutor per year 2410.03017 SI: Study Costs. Card: Tutor CoPilot.

Human sets the autonomy limit (agents)

This variant is a position, not a method. The authors propose a five-level scale of agent autonomy, from a program in which the model has no say over the flow to one in which the model creates and runs new code [arxiv:2502.02649 tab. 1]. They argue that fully autonomous agents should not be developed 2502.02649 §1 2502.02649 §6. They judge semi-autonomous systems to balance risks and benefits better, but name no threshold for where the line lies 2502.02649 §1. Their recommendations are a spectrum of autonomy levels, human-control mechanisms, and methods that keep agents within constraints set by humans 2502.02649 §6. Card: Fully Autonomous AI Agents Should Not be Developed.

The survey adds a run-time view of the same question. It names five collaboration patterns, among them dynamic delegation with a control handoff and supervisory orchestration with plan review and approval checkpoints. For each it lists the risk and a suggested safeguard, such as escalation rules, readiness checks and audit logs 2403.04931 tab. 2. The risks it names are a handoff that comes too late or too often, and errors that build up unseen in multi-step tasks 2403.04931 tab. 2. The survey proposes no thresholds for these, and it reports no tests of the safeguards 2403.04931 §3.2. Card: A Survey on Human-AI Collaboration with Large Foundation Models.

Human shares control of a robot (blended command)

A shared-control arm blends the human's command with the robot's own reference, and one weight sets how much the robot decides. The commanded position is a convex combination with weight gamma on the robot 2402.02758 §3.2 2402.02758 eq. 3. In a 24-person study of teleoperated 3-D tracking, the statistics compare a low level (gamma 0.2) with a high level (gamma 0.8) 2402.02758 §4.

Self-reported workload (NASA-TLX) fell and self-reported trust (MDMT) rose at the higher level, both with p < .001 2402.02758 tab. 1. The two objective load measures, tapping error and pupil index, showed no autonomy effect 2402.02758 tab. 1. So the self-report and the objective measures disagree in this study. Tracking error was lower at the high level, 0.83 cm against 2.16 cm in the units the paper gives 2402.02758 tab. 2. The high level is gamma 0.8, not full autonomy, which is gamma 1 2402.02758 §3.2. The paper notes that the robot gave no haptic feedback and that the design covers one task 2402.02758 §5.4. Card: Effects of Shared Control on Cognitive Load and Trust in Teleoperated Trajectory Tracking.

Human's labelled trials tune a device

A threshold-based state machine detects walk-to-sit and stair transitions on two lower-limb exoskeletons, and its thresholds were first set offline 2411.12573 §2.1 2411.12573 §2.2. For a user whose transitions are missed, the thresholds can be re-set with Bayesian optimization (BO) against that user's labelled transitions 2411.12573 §2.5.2 2411.12573 alg. 1. The BO cost is computed from ground-truth labels, so the user must supply labelled data before deployment 2411.12573 alg. 2. A statistics-based rescaling (SBA) also adapts the thresholds without a search 2411.12573 §2.5.1.

On eWalk, BO raised the W-SD and SD-W accuracies to 96% and 98% 2411.12573 §4.2 fig. 6. On autonomyo, BO raised four transitions from 77.5, 75, 77.5 and 75% to 90, 90, 87.5 and 82.5% 2411.12573 §4.2 fig. 6. SBA raised three eWalk transitions from 92, 86 and 80% to 94, 90 and 88%, and gave no improvement on autonomyo 2411.12573 §4.2. One subject's W-SD transition on eWalk went from 20% to 100% [arxiv:2411.12573 §4.2] fig. 7. The abstract's "up to 80%" figure is not tied to a table, and this single subject is the closest in-text case 2411.12573 §4.2.

BO was not run in real time 2411.12573 §5.5. The paper estimates about 2 minutes on a PC and does not measure an embedded run 2411.12573 §5.4. Ground truth was labelled live by an observer during zero-torque trials, and the tests gave no active assistance 2411.12573 §3 2411.12573 §5.5. Card: Personalising exoskeleton transition thresholds.

Human reviews a summary of unsupervised robot work

When a robot works without supervision, its operator must learn afterwards what it did. Pretrained video models can summarise a 40-minute robot video as a generic summary, chosen with no input, or as a query-driven summary, chosen after the operator asks a question. Each summary can be shown as a storyboard, a short video or text 2411.10016 §3. No model was fine-tuned 2411.10016 §3.4.3.

In a 30-person study with four 40-minute videos, query-driven summaries raised answer accuracy over raw video (coefficient 0.549, p = 0.005) and added task time (156.3, p = 0.002) 2411.10016 tab. 2. Generic summaries did not significantly change accuracy 2411.10016 tab. 2. Among modalities, storyboards were more accurate than video (0.905, p = 0.004) and took less time (-124.7, p = 0.028). Text was less accurate than video (-0.955, p = 0.005). The storyboard advantage did not hold for event questions (interaction -2.447, p < .001) 2411.10016 tab. 2. Time units are not stated 2411.10016 tab. 2.

The four videos were paired with tasks, but which video went with which modality is not stated, so video and modality effects may be confounded 2411.10016 §3.3. Per-condition accuracy percentages are not reported 2411.10016 §4.4. Card: Video foundation models for intermittent robot supervision.

Human confirms a target before the robot moves

An operator gives a spoken or written instruction, such as where a hand should rest, and the system shows a candidate point on the robot's camera image. Each later instruction is treated as a prediction, a correction or a confirmation, and only the confirmed point goes to the controller 2407.14229 §3 2407.14229 §3.2 fig. 2. A language model writes the arithmetic that places the point relative to detected objects, and a segmenter handles named objects 2407.14229 §3.3.

On 78 image, prompt and mask tuples, the best combination (GPT-3.5 with GroundingDINO and CLIPSeg) reached a median overall success of 0.61, with quartiles of 0.51 and 0.70, against 0.13 for random points [arxiv:2407.14229 tab. 1]. Cells use different models and the paper does not state the sample size per cell 2407.14229 §4.1. The masks mark acceptable contact areas, but the rule for a hit is not stated 2407.14229 §4.1 fig. 7. In a pilot with 11 volunteers, the median distance to the target at the fifth prompt was 21 px, against an 18 px target radius 2407.14229 §4.2 fig. 8. On the Talos humanoid, all four reaching settings succeeded, with no trial counts given 2407.14229 §4.3. The released code holds only the language and vision part, so the interaction loop must be built from the paper; see the card's Paper vs code section. Card: Words2Contact: Identifying Support Contacts from Verbal Instructions Using Foundation Models.

Human hands authority over at a predicted point

A planner models human reaching as a trade-off between speed and accuracy, and between effort and reward, then solves for a smooth motion 2412.13474 §2.1 2412.13474 §3.3. In co-manipulation, the robot estimates the human's goal from the first 0.2 s of motion and then runs the planned trajectory open-loop 2412.13474 §5.1. The solve takes 1.3 s for the seven-DoF arm 2412.13474 tab. 1.

For a task with a fast transport and a slow precise part, a Gaussian process fitted to earlier human data predicts where a person switches between them from the distance to the goal 2412.13474 §2.2 fig. 2. At that point the robot's stiffness drops from 400 N/m to 0, handing control over 2412.13474 §5.2.

The evidence is qualitative. The handover test used one operator and five trials per goal, and no baseline controller was run 2412.13474 §5.1 2412.13474 §5.2. In simulation, planned velocities follow width and distance trends by eye, with no fit reported fig. 3 2412.13474 §3.4. Card: Planning Human-Robot Co-manipulation with Human Motor Control Objectives and Multi-component Reaching Strategies.

When to use which

Implementation sketch

The papers give no interfaces, so the sketch below is our reading of the co-pilot loop plus a feedback log. The names propose, human.decide and log are placeholders.

# Human-in-the-loop step: the AI proposes, the human decides, feedback is kept
# [arxiv:2410.03017 §Method] [arxiv:2105.00691 §3.2]
# Input:  context, options (e.g. strategies), human, log
# Output: the final action, which the human chose or edited

def human_in_the_loop_step(context, options, human, log):
    # One candidate per option, built from the live context  [arxiv:2410.03017 §Method]
    candidates = {opt: propose(context, opt) for opt in options}

    # The human edits, regenerates or picks another option; their choice wins
    # [arxiv:2410.03017 §Method] [fig. 1]
    final = human.decide(candidates)

    # Keep the context, the candidates and the choice as labelled data
    # [arxiv:2105.00691 §3.2]
    log.append(context, candidates, final)
    return final

Contradictions

No direct numerical conflict. The papers mostly test different things, and the points of tension are these:

• Trust and autonomy. The shared-control study finds self-reported trust rises as the robot's share grows 2402.02758 §4.3. The position paper argues that fully autonomous agents should not be developed [arxiv:2502.02649 §6]. The study's high level is not full autonomy, and its robot is an arm, not an agent 2402.02758 §3.2. The two do not test the same claim.
• Load measures. In the shared-control study, self-reported workload fell with autonomy while tapping error and pupil index did not [arxiv:2402.02758 tab. 1]. The study does not say which measure is the better guide to load.

The Hybrid Intelligence claim that combined results are superior to either part 2105.00691 §4 is not tested. Tutor CoPilot measures one gain in one setting 2410.03017 tab. 1. Do not read one as confirming the other.

Papers

• Scalable agent alignment via reward modeling: a research direction: position paper proposing that a user's feedback trains a reward model that an RL agent then optimises; no experiments.
• Hybrid Intelligence: position paper defining humans and AI as complementary partners that learn from each other; no experiments.
• A Survey on Human-AI Collaboration with Large Foundation Models: survey of build-time guidance and run-time collaboration patterns for foundation models; no experiments.
• Tutor CoPilot: a tutor-facing assistant that suggests one reply per teaching strategy; a randomized trial with 783 tutors, author-reported.
• Fully Autonomous AI Agents Should Not be Developed: position paper proposing a five-level autonomy scale and arguing against full autonomy; no experiments.
• HACTS: a Human-As-Copilot Teleoperation System for Robot Learning: a leader-follower arm that lets a person take over a running policy, with corrections used for imitation learning and RLPD; author-reported results on a UR5 with 10 rollouts per cell.
• HG-DAgger: Interactive Imitation Learning with Human Experts: a DAgger variant in which the human expert decides when to take control and labels only those takeovers, with a doubt threshold from ensemble variance; author-reported simulation and five-scenario vehicle tests.
• Effects of Shared Control on Cognitive Load and Trust in Teleoperated Trajectory Tracking: a 24-person study of a blended-command robot arm; self-reported load and trust moved with autonomy, objective measures did not; author-reported.
• Personalising exoskeleton transition thresholds: a threshold state machine for two exoskeletons, personalised by statistics or by Bayesian optimization on labelled user data; author-reported.
• Video foundation models for intermittent robot supervision: pretrained video models summarise 40-minute robot runs for a supervisor; a 30-person study, author-reported.
• Words2Contact: Identifying Support Contacts from Verbal Instructions Using Foundation Models: an operator's instructions and corrections pick a contact point on the camera image before the robot moves; author-reported on 78 samples.
• Planning Human-Robot Co-manipulation with Human Motor Control Objectives and Multi-component Reaching Strategies: a planner built on human motor-control objectives, with goal estimation and a predicted handover point; qualitative, one operator.