SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysis

In plain words

Robotic surgery produces video in which instruments and tissue overlap and move quickly. Answering questions about a single frame, such as "which tool is this?" or "what is the surgeon about to do?", is hard. Most earlier systems train one model per question type, so each new question type needs new labelled data.

SurgRAW uses several AI helpers that work together without any training. A manager helper first decides whether a question is about what is visible or about background knowledge. Visual questions go to specialist helpers that write their reasoning step by step, and a moderator checks that the instrument answer and the action answer agree with a list of allowed tool-action pairs. Background questions go to helpers that first look up text from medical sources. The idea is that stepwise reasoning and cross-checking reduce confident wrong answers.

The authors also built a new benchmark of frames from surgery videos with multiple-choice questions, so that such systems can be compared on one set of tasks.

TL;DR

• Zero-shot multi-agent system for five multiple-choice tasks on robotic surgery frames, with a benchmark the authors built (SurgCoTBench) §3.1.
• An orchestrator splits questions into visual-semantic and cognitive-inference streams §3.2.
• Visual action answers go through a panel that checks a knowledge graph and two 1–5 rubrics, with up to 3 refinement runs §3.3.
• Knowledge questions use retrieved medical text, and every task uses its own chain-of-thought prompt §3.3 §3.4.
• Author-reported: SurgRAW-GPT4o reaches 61.73% overall, against 31.44% for GPT-4o without the workflow tab. 3. The released code does not match the paper in several places; see Paper vs code.

Problem

Surgical AI is usually built from separate models per task, with fragmented, labelled datasets and limited interpretability §1. General and medical vision-language models (VLMs) reason zero-shot, but they are trained mostly on natural images, are prone to hallucination in the surgical domain, and treat tasks independently instead of modelling how surgical steps depend on one another §1. The paper states that existing surgical datasets rarely carry multi-task reasoning annotations, so such systems cannot be evaluated on that basis §1 §2.

Method

The workflow has three parts: a two-stream orchestrator and task agents §3.2, supplementary modules (a panel discussion for visual-semantic queries and retrieval for cognitive-inference queries) §3.3, and task-specific chain-of-thought prompts §3.4. The paper's overview figure shows these parts fig. 1.

Steps, following §3.2 to §3.4:

1. A top-level orchestrator labels each question as visual-semantic (VS: action recognition, instrument recognition) or cognitive-inference (CI: action prediction, surgical outcome, patient detail) §3.2.
2. VS questions go to task agents. The Instrument Specialist answers the instrument question. The Action Interpreter answers the action question with its chain-of-thought prompt §3.2 §3.4.
3. For VS questions, an Action Evaluator runs a panel check §3.3:
   • Parse both answers to a label.
   • Check whether the action is allowed for the instrument in a knowledge graph of instrument-action pairs §3.3.
   • Rate the pair on two rubrics, Inter-Chain Coherence and Inter-Agents Collaborative Synergy, each an integer from 1 to 5 §3.3.
4. If the knowledge-graph check is true and both ratings are higher than 3, the answer is accepted. Otherwise the evaluator passes information between the agents and reruns, for at most 3 runs §3.3.
5. CI questions are routed to the Action Predictor, Outcome Analyst or Patient Advocate §3.2. Each receives text retrieved from MedlinePlus for the question §3.3, and writes its answer with the CI chain-of-thought prompt §3.4.
6. The final response is produced after these intermediate steps §3.2.

# SurgRAW answer routine  [§3.2] [fig. 1]
# Input:  question (str, multiple-choice text); frame (image)
# Output: answer text, ending "The answer is: Option ()"

def answer_question(question, frame):
    # Top orchestrator: visual-semantic or cognitive-inference  [§3.2]
    if classify_stream(question) == 'VS':
        return visual_semantic_answer(question, frame)
    # Retrieve medical text for the question  [§3.3]
    knowledge = retrieve_knowledge(question)
    # Action Predictor, Outcome Analyst or Patient Advocate  [§3.2]
    agent = ci_agent_for(question)
    return agent(question, frame, knowledge)  # CI prompt  [§3.4]

# Panel discussion for VS questions  [§3.3]
# Threshold: rating above 3 on a 1-5 scale; at most 3 refinement runs
# [§3.3]. Code runs this panel only for action questions
# [code: Orchestrators.py#L97-L102].

def visual_semantic_answer(question, frame):
    # Instrument Specialist answers the instrument form of the question
    # [§3.2] [code: Utils/Debate_utils.py#L78-L103]
    instrument = instrument_agent(instrument_form(question), frame)
    # Action Interpreter answers with its chain-of-thought prompt  [§3.4]
    action = action_agent(question, frame)
    first = make_candidate(instrument, action)  # parse, KG, rate
    if passes(first):  # KG true and both ratings > 3  [§3.3]
        return first['action']
    candidates = [first]
    for run in range(3):  # at most 3 refinement runs  [§3.3]
        # Code writes the instrument name into the action prompt
        # [code: Agents/GP_Moderator.py#L108-L116]; the paper does
        # not say how the refinement is prompted
        instrument = instrument_agent(instrument_form(question), frame)
        action = action_agent(
            with_instrument(question, instrument),
            frame,
        )
        candidates.append(make_candidate(instrument, action))
        if passes(candidates[-1]):
            break
    # Paper: "the conversation will be interrupted". Code instead asks an
    # LLM to pick one candidate  [code: Utils/Debate_utils.py#L295-L349]
    return select_best(candidates)['action']

# Retrieval for CI questions  [§3.3]
# Code searches three pages, each indexed separately on every call
# [code: Agents/RAG_module.py#L88-L133]

def retrieve_knowledge(question):
    # Source pages; the paper names only MedlinePlus  [§3.3]
    passages = []
    for page in source_pages:
        # Fetched live on every call, no cache  [code: RAG_module.py#L77]
        text = fetch_page_text(page)
        # Chunks of 400 characters with 50 overlap (code only)
        # [code: Agents/RAG_module.py#L102]
        chunks = split_text(text, size=400, overlap=50)
        # Keep chunks with similarity score at least 0.50 (code only)
        # [code: Agents/RAG_module.py#L113]
        hits = most_similar(chunks, question, threshold=0.50)
        if not hits:
            passages.append('No relevant data found.')
            continue
        passages.append(summarise(hits, question))
    return join(passages)

Chain-of-thought prompts §3.4:

• VS, shared structure: five stages: question analysis, image feature extraction, cross-reference of features with the question's expectations, elimination of options that conflict with the visual cues, and a consistency check against the scene §3.4.
• Action Recognition: five chains (question analysis; visual cues of ongoing action; match with candidates; eliminate conflicting options; verify against scene and phase) §3.4.
• Instrument Recognition: chains 1 to 5, with sub-chains 2.1–2.3 and 3.1–3.3 for localisation, feature extraction, functional context and ambiguity between similar tools §3.4.
• CI, shared structure: decompose the question into clinical parts, extract visual cues, answer the sub-questions, refine with procedural norms or retrieved knowledge, and eliminate conflicting options §3.4.
• Action Prediction: four task chains (decompose into current and next phase; identify current phase; answer sub-questions; refine with knowledge), followed by the shared chains 5 and 6 §3.4.
• Surgical Outcome and Patient Detail: four task chains each, followed by the shared chains 5 and 6 §3.4.

Dataset, SurgCoTBench §3.1:

• Built from 12 patient videos of prostatectomy and lobectomy procedures, taken from public da Vinci videos §3.1.
• Videos are sampled at 1 frame per second; frames are chosen at clinically meaningful phases §3.1.
• Questions come from GPT-4o, written for each selected frame from surgeon narration transcribed with Whisper §3.1.2.
• Up to seven QA pairs per frame, across five tasks: action recognition, instrument recognition, action prediction, surgical outcome, patient detail §3.1 §3.1.2.
• Action and instrument recognition are split into left- and right-arm questions §3.1.1.
• Two clinicians reviewed the data §3.1.2. The paper gives no review error rate.
• Supervised baseline split: 8 videos for training, 4 for testing, by patient §3.1.

Counts tab. 1:

Hyperparameters

Results

Author-reported. Setup: zero-shot on SurgCoTBench frames, accuracy as mean of three runs ± standard deviation, one NVIDIA A6000 for open-source models §4.1. The supervised baseline is Surgical-VQA, trained as in its original paper §4.1. Baselines are the general VLMs GPT-4o and Qwen3VL-8B, the medical VLMs MedGemma-4B and LLaVA-Med-7B, LLaVA-CoT, MDAgents and MedAgents §4.2 tab. 3. The paper's accuracy numbers are not reproducible from the repository, which has no evaluation script (see Paper vs code).

Table 3, comparison on SurgCoTBench, all values are % accuracy tab. 3. CI is cognitive-inference, VS is visual-semantic. Act. Pred. = action prediction, Out. = surgical outcome, Pat. = patient detail, Act. Rec. = action recognition, Inst. Rec. = instrument recognition.

Headline numbers, from the paper's text §4.2 §4.3:

• Surgical-VQA: 47.12 overall, the supervised baseline. SurgRAW-GPT4o is 14.61 points above it tab. 3.
• SurgRAW improves over its matched backbone by 29.32 points on average, across GPT-4o and Qwen3VL-8B §4.2. This is the mean of the two overall differences, 30.29 and 28.34, derived from tab. 3.
• SurgRAW-GPT4o gains 37.50 points on action prediction and 86.77 points on patient detail over GPT-4o §4.2. These are differences of table cells, not relative gains.
• Standard deviation for SurgRAW-Qwen3VL-8B is ±1.51 overall, against ±5.46 for Surgical-VQA §4.2 tab. 3.

Table 4, ablation on SurgRAW-GPT4o, % accuracy tab. 4. CoT is the chain-of-thought prompts, RAG is retrieval, PD is the panel discussion.

The paper reads these as: CoT is the largest contributor (CI average +38.86 points, VS average +6.33 points), RAG adds to CoT on CI tasks, and the panel matters for action recognition §4.3. These gains are also differences of table cells.

Paper vs code

Official repository: github.com/jinlab-imvr/SurgRAW, linked from the abstract §abstract. It belongs to the authors' lab (organisation jinlab-imvr). Commit 7302fa0, shallow clone. There is no LICENSE file, so the code is unlicensed. This card paraphrases the code only and quotes none of it.

Checked by reading the code. Nothing was executed.

• Import errors. Agents/GP_Moderator.py line 6 and Utils/Debate_utils.py line 6 import Agents.Agent4_InstrumentIdentification, which is not in the repository; the instrument agent is in Agents/Agent2_InstrumentIdentification.py. Agents 1 to 5 import gemini_vision_caption from Utils/API_utils.py, which that file does not define Agent1_ActionRecognition.py GP_Moderator.py Debate_utils.py. Importing Orchestrators.py therefore fails.
• No Qwen3VL-8B path. The paper reports SurgRAW-Qwen3VL-8B §4.2 and sampling settings for open-source VLMs §4.1. The code calls only gpt4_vision_caption, which uses the OpenAI model gpt-4o-latest with no sampling parameters API_utils.py. The Qwen run and its settings are not in the repository.
• Panel only for action questions. The paper says VS tasks, including instrument recognition, go through the Action Evaluator panel §3.3. The code sends instrument questions straight to the instrument agent and only action questions to the panel Orchestrators.py. The ablation's instrument column is unchanged when the panel is added (43.57 both ways) tab. 4, which fits the code.
• Refinement and final selection. The paper says the loop stops when the knowledge-graph check and both ratings pass, or is "interrupted" after 3 runs §3.3. The code, after any refinement, asks GPT-3.5 to pick one of the initial and refined candidates, and returns it even if no candidate passed GP_Moderator.py Debate_utils.py. The paper does not describe that selection.
• Rubric scope. The paper's Inter-Chain Coherence rubric judges the logic within one task's chain of thought §3.3. The code's coherence rubric judges whether the instrument enables the predicted action and whether the two answers contradict each other Debate_utils.py.
• Failed ratings count as failures. A rating that cannot be parsed after three attempts is set to 3 Debate_utils.py. The acceptance test is strictly above 3 GP_Moderator.py, so a parse failure always triggers refinement. The paper says nothing about this default.
• Knowledge graph source. The paper says the knowledge graph is derived from the Da Vinci specifications §3.3. The code has a hard-coded table of seven instruments and their allowed actions, with no source citation beyond letter comments Debate_utils.py.
• Retrieval sources. The paper says the repository is MedlinePlus §3.3. The code fetches one MedlinePlus page and two SEER cancer statistics pages, and its comment says the list is non-exhaustive RAG_module.py.
• Surgical plan. The paper lists surgical plan among the CI chains §3.4, but the five tasks in its benchmark and results have no surgical plan task tab. 2 tab. 3. The code's classifier prompt mentions "surgical plan" Orchestrators.py, and any unrecognised answer falls back to action prediction Orchestrators.py.
• Model names. The paper uses "GPT-4o" §4.2. The code uses the alias gpt-4o-latest API_utils.py, which can change without notice. The orchestrator, parser, rater and RAG answer step use gpt-3.5-turbo API_utils.py RAG_module.py. The paper does not name the LLM for these steps.
• Dataset count. The paper gives 14,256 QA pairs §3.1 and its table sums to that tab. 1. The README says 14,176 README.md.
• Dataset and prompts. The README says the dataset and the full CoT prompts "will follow publication" and links a Hugging Face dataset README.md. The agent prompts are in the repository Agent1_ActionRecognition.py. The dataset is not in the repository, and this card does not use it.
• Evaluation code. The repository has no evaluation or scoring script. Main.py reads the ground-truth column and never uses it Main.py. The accuracy numbers cannot be checked from the repository.

Pitfalls

• "%" numbers are point differences. 14.61, 29.32, 37.50, 86.77, 38.86 and 6.33 are differences between table cells, in percentage points, not relative gains tab. 3 tab. 4 §4.2 §4.3.
• Averages do not always follow from their columns. The paper does not say how averages are formed. For SurgRAW-GPT4o, the mean of its three CI cells is 69.85, but the table reports 70.39; for its VS cells, the mean is 49.33, against 49.81. In the ablation, CoT-only VS averages to 47.69 against 46.72 reported tab. 3 tab. 4. The overall column also differs from the mean of the five task cells (61.64 for GPT-4o, 63.68 for Qwen). Use the cells, not the averages, for your own comparisons.
• Small evaluation. Each cell is a mean of three runs §4.1. The supervised baseline is trained on eight videos and tested on four §3.1. The paper gives no significance tests.
• Labels are model-generated. Questions are written by GPT-4o from surgeon narration §3.1.2. Two clinicians reviewed them, but the paper gives no error rate or count of corrections. Treat the labels as noisy.
• Frames, not video. The system answers one frame at a time, and the paper lists temporal reasoning as future work §5. Frame context such as the preceding steps is not used, though the paper says frames are "interpreted within procedural workflow context" §5. Decide how you supply that context before using the workflow for video.
• Routing depends on exact strings. Orchestrator and router outputs are matched against exact labels, e.g. vision_class == "instrument recognition" Orchestrators.py. A reply such as Instrument recognition. with a full stop fails that test and goes to the action panel. Unmatched knowledge labels go to action prediction Orchestrators.py.
• Option count in the prompts. The action and instrument prompts say "choose one of the 4 options" Agent1_ActionRecognition.py Agent2_InstrumentIdentification.py, but the instrument question has seven options (A–G) Debate_utils.py and the action map also has seven Debate_utils.py. Keep the prompt count consistent with your questions.
• Fixed option letters. The code assumes action option A is retraction through G is applying clip, and the instrument options are fixed in one hard-coded list Debate_utils.py. The paper does not state the option order. Any dataset with a different order will be misread.
• Brittle question rewriting. The instrument question is made by replacing one exact phrase and keeping only the text before the first question mark. If the phrase is absent, the text still asks about the action while the options ask about instruments. The original options are dropped Debate_utils.py.
• Parse failures cost calls. An answer the parser cannot match becomes "unknown", which fails the knowledge-graph check and triggers refinement Debate_utils.py Debate_utils.py.
• Cost per question. Counted from the code, a visual action question can take up to 8 vision-model calls (2 first-round plus 2 per refinement) and about 35 GPT-3.5 calls: 2 routing calls, up to 8 parsing calls, up to 24 rating calls (two ratings with up to 3 tries each, in up to 4 rounds), and 1 final selection. The paper says only that the workflow has "additional inference overhead" §4.1.
• Live retrieval breaks reproducibility. Each RAG call fetches the pages live, rebuilds the embeddings and makes a new index RAG_module.py. Page content and the embedding model can change between runs. requests.get has no timeout RAG_module.py.
• Keys are blank. API keys are empty strings in the code API_utils.py RAG_module.py. Set them in your own environment, and do not commit them.
• Sampling is not reported for GPT-4o. The paper's sampling settings apply to open-source models only §4.1. The GPT-4o runs' temperature and other settings are not stated, and the code sets none.
• Version. This card is compiled from the arXiv v3 source [raw/arxiv/2503.10265.json]. The IEEE Robotics and Automation Letters version may differ.
• Figure cross-reference. The paper labels two figures with the same key fig:example. The case study cited in the text as the qualitative example is the second one fig. 3; the chat board is fig. 2. The paper has no numbered equations and no algorithm environments; all pseudocode above is ours, from the prose.

Lineage

• Builds on: LLaVA-CoT for chain-of-thought prompting of VLMs, which the paper replaces with task-specific prompts §3.2 §3.4. MDAgents and MedAgents as multi-agent medical baselines §2. Retrieval-augmented generation §3.3.
• Compared against: GPT-4o, Qwen3VL-8B, LLaVA-OV-7B, MedGemma-4B, LLaVA-Med-7B, LLaVA-CoT, MDAgents, MedAgents and the supervised Surgical-VQA §4.2 tab. 3.
• Superseded by: none known from content in this wiki.