Information-Theoretic Graph Fusion with Vision-Language-Action Model for Policy Reasoning and Dual Robotic Control

In plain words

To teach a two-armed robot a building task, the usual trick is to copy the exact movements of a person. That breaks as soon as the blocks move or the arms are placed differently. This paper instead watches a person's hands and the blocks in a video and works out which objects matter and how they touch each other, such as "hand holds block A" or "block A rests on block B". It writes that down as a graph of steps.

A language model then reads the graph and writes a step-by-step plan with its reasons and a check for each step. A second output of the same model turns the current step into arm motions. The robot can therefore follow the plan from one human demonstration, and it can try again when a step fails.

TL;DR

• Learns a dual-arm block-assembly policy from one human RGB-D video; hand and object interactions are found with entropy and mutual information §3, §4.
• A LLaMA 2 7B model with two LoRA-tuned heads writes chain-of-thought plans and emits Cartesian motion commands §4.2, §4.4.
• Author-reported: 94% grasp success, 89% placement accuracy and 90% task success on 120 held-out block trials and six transfer variants tab. 1, tab. 3.
• No official code was found. Many values the implementer needs are missing or inconsistent (see Pitfalls).

Problem

Trajectory imitation from human video does not generalise across object types, spatial layouts and arm configurations §1. Vision-language-action (VLA) models can follow language, but the authors say they lack explicit models of how objects and agents interact over time, and they need large pretraining sets §1. Pure information-theoretic methods model interactions but do not connect to high-level planning §1. GF-VLA targets the gap between these two §1.

Method

Overview. The system has two stages. Stage 1 turns a human demonstration into a policy: it extracts information-theoretic scene graphs and keyframes, and a policy agent turns them into subtasks with reasoning traces §4.1 fig. 1 fig. 7. Stage 2 executes the plan on the robot with a dual-head VLA that outputs both a plan (LLM head) and motion commands (action head), with self-verification and local replanning §4.1 fig. 5.

Stage 1a: perception. A segmentation model (SAM 2) and a hand-pose estimator (cited as FingerPoseNet) give a 6-D pose for each hand and each object in every frame §3. The rest of the pipeline works on the position signals §3.1.

Stage 1b: entropy of a position signal. For each position signal, the method histograms the positions inside a sliding window with a fixed bin width, then takes the Shannon entropy of the bin probabilities eq. 1 §3.1. The authors argue that a static object has zero entropy, so entropy rises when an object starts to move eq. 2 §3.1 fig. 2. The sign of its slope marks the start (positive) or the settling (negative) of a movement §3.1.

Stage 1c: mutual information of a hand and an object. Mutual information (MI) is computed per window from the entropies of the hand and object signals and their joint histogram eq. 3 eq. 4 §3.1. The paper writes it as the reduction of uncertainty in one signal given the other eq. 5. For a rigid grasp, MI approaches the object's own entropy eq. 6 §3.1 fig. 3. MI is summed over the three spatial axes §3.1.

Stage 1d: hand-object (HO) edge. Only objects within a distance gate of the hand are considered, nearest first; the search stops at the first detection §3.2.1. The steps are:

1. If the mean hand-object distance is over the gate, no edge is made §3.2.1.
2. If MI is above a near-zero threshold, the edge is Coupled-Motion, meaning the hand is moving the object §3.2.1.
3. Otherwise the edge is Docked when MI is falling and below a second threshold, or when the previous frame was already Docked §3.2.1.
4. A Docked state may only follow a Coupled-Motion phase §3.2.1.
5. An edge ends when the distance exceeds the gate §3.2.1.

# Hand-object (HO) state for one hand and one object  [§3.2.1]
# Input:  mi (summed over x, y, z), its slope, distance, previous state
# Output: "Coupled-Motion", "Docked" or None

def hand_object_state(mi, mi_slope, distance, previous):
    # Proximity gate on the mean distance  [§3.2.1] [§5.1]
    if distance >= r_th_ho:
        return None
    # Active coupling: MI above the near-zero threshold  [§3.2.1]
    if mi > alpha:
        return "Coupled-Motion"
    # Docked may only follow a Coupled-Motion phase: one reading of
    # the text, not the only one (see Pitfalls)  [§3.2.1]
    if previous == "Coupled-Motion" and mi_slope < 0 and mi < gamma:
        return "Docked"
    # A Docked state is kept while the hand stays near  [§3.2.1]
    if previous == "Docked":
        return "Docked"
    return None

Stage 1e: object-object (OO) edge. An OO edge links the manipulated object to one background object, nearest first, and only if an HO edge exists §3.2 §3.2.2. Its type depends on the distance between the two objects.

1. If the mean distance is over the OO gate, there is no edge §3.2.2.
2. An E-OO (efficient, task-relevant) edge is kept while the distance stays under the gate, once it has been set §3.2.2.
3. If the hand docks the manipulated object, the pair becomes E-OO §3.2.2.
4. While the hand is moving the object, the sign of the slope of the distance entropy decides: falling gives E-OO, otherwise T-OO (transitory, dropped from the plan) §3.2.2.

# Object-object (OO) state, manipulated vs background object  [§3.2.2]
# Input:  mean distance, slope of distance entropy, hand state, previous
# Output: "E-OO", "T-OO" or None

def object_object_state(distance, entropy_slope, hand_state, previous):
    # Proximity gate on the mean distance  [§3.2.2] [§5.1]
    if distance >= r_th_oo:
        return None
    # Once E-OO is set it stays until the objects separate  [§3.2.2]
    if previous == "E-OO" or hand_state == "Docked":
        return "E-OO"
    if hand_state == "Coupled-Motion":
        # Falling distance entropy means a stable, intended pair  [§3.2.2]
        if entropy_slope < 0:
            return "E-OO"
        return "T-OO"
    return None

Stage 1f: the scene graph. Nodes are the hands and objects; directed edges are HO and OO relations, each labelled with its type and, for HO, its MI §3.2 §3.2.1. Four topologies occur: hand-object only, hand-object plus one OO edge, and the variants shown in fig. 4. Graphs are stored per frame §3.

Stage 1g: cross-hand selection. For a grasp that places an object at a target pose, the state uses four Euclidean distances: each hand to its own source block and each hand to the target §3.2.3. If the target is nearer the right hand, the left hand grasps the block from the left-side source; otherwise the right hand grasps eq. 7 §3.2.3.

# Pick the gripper for a grasp at the target pose  [§3.2.3] [eq. 7]
# Input:  Euclidean distances from each hand to the target pose
# Output: "use_left_hand" or "use_right_hand"

def select_hand(dist_left_to_target, dist_right_to_target):
    # Target nearer the right hand: grasp from the left-side source
    if dist_right_to_target < dist_left_to_target:
        return "use_left_hand"
    # Left nearer, or a tie: right hand  [eq. 7]
    return "use_right_hand"

The paper also defines a reward of a bonus for agreeing with the human demonstrator and a penalty otherwise eq. 8 §3.2.3. It is described as a learning signal, but no update rule is given (see Pitfalls).

Stage 2a: the dual-head VLA. Top-down RGB-D patch embeddings and binary object masks pass through a two-layer MLP and are fused with prompt tokens. The language model is a pretrained 7B LLaMA 2 §4.2. Spoken commands go through an ASR module named Gladia and become text prompts §4.2.

Stage 2b: LLM head (planning). The head writes a behaviour tree once per task. Each node has an action type (for example a dual-arm pick or place), its parameters, a natural-language reason and a verification condition §4.3 fig. 6. Verification uses gripper status, object pose and scene stability §4.3. The head also judges whether each subtask succeeded §5.2.

Stage 2c: action head (control). The action head outputs Cartesian poses and gripper states as a continuous stream at 5 Hz, which is sent to the two arms §4.2.

Stage 2d: training with LoRA. Two separate LoRA adapters sit in the shared LLaMA blocks, one per head §4.4. The visual encoder and projector are frozen §4.4. The LLM head is trained by next-token prediction on plan steps and reasoning traces from 125 annotated videos §4.4. The action head is trained with regression losses on Cartesian poses and gripper states, using 120 robot trials §4.4 §5.3. The two heads are trained by alternating batches §4.4.

Metric definitions are given in eq. 9 to eq. 17 in Results.

# Window mutual information, summed over the three axes  [§3.1] [eq. 3]
# Input:  hand and object positions for one window, shape (N, 3)
# Output: scalar MI

def window_mutual_information(hand_xyz, object_xyz):
    # Accumulate the per-axis MI  [§3.1]
    total = 0.0
    for axis in range(3):
        # Marginal and joint entropies of the window  [eq. 3] [eq. 4]
        h_hand = window_entropy(hand_xyz[:, axis], bin_width, eps)
        h_object = window_entropy(object_xyz[:, axis], bin_width, eps)
        h_joint = joint_entropy(hand_xyz[:, axis], object_xyz[:, axis])
        total = total + h_hand + h_object - h_joint
    return total

# Shannon entropy of one window of positions  [§3.1] [eq. 1]
# Input:  positions (N,) in the window; bin width (quantisation)
# Output: scalar entropy (natural log, see Pitfalls)

def window_entropy(positions, bin_width, eps):
    # Histogram with a fixed bin width, so bin count varies  [§3.1]
    counts = histogram(positions, bin_width)
    probs = counts / sum(counts)
    # Entropy over non-empty bins  [eq. 1]
    return -eps * sum(p * ln(p) for p in probs if p > 0)

Equations

Equations are reproduced from the paper's numbering (in order of appearance).

Entropy eq. 1:

\mathcal{H}^{\mathcal{X}}(p) = -\epsilon \sum_{i=1}^{N_x} p(x_i)\, \ln p(x_i)

Static window eq. 2:

\mathcal{H}_{static} = -\sum_{i} p(x_i) \ln p(x_i) = -(1 \cdot \ln 1) = 0

Mutual information from entropies eq. 3:

\varpi_{X|Y} = \mathcal{H}^X + \mathcal{H}^Y - \mathcal{H}^{X,Y}

The paper calls the joint term in eq. 4 "joint entropy":

\varpi_{X|Y} = -\sum_{i}\sum_{j} p(x_i, y_j) \ln p(x_i, y_j)

Conditional form eq. 5:

\varpi_{X|Y} = \mathcal{H}^X - \mathcal{H}^{X|Y}

Rigid coupling limit eq. 6:

\lim_{\text{coupling}\to\text{rigid}} \varpi_{X|Y} \approx \mathcal{H}^X

Hand selection rule eq. 7:

a_t = \begin{cases} \text{use\_left\_hand} & r_{h_\triangleright, o_i} < r_{h_\triangleleft, o_i} \\ \text{use\_right\_hand} & r_{h_\triangleleft, o_i} \le r_{h_\triangleright, o_i} \end{cases}

Reward eq. 8:

R(a_t, a_t^*) = \begin{cases} +R_{bonus}, & a_t = a_t^* \\ -R_{penalty}, & a_t \neq a_t^* \end{cases}

Hyperparameters

Results

Author-reported. Setup: four dual-arm block-assembly tasks (shape generalization, ambiguous spatial relation, absolute 6-D pose, relative pose), each with three block shapes and 10 trials per shape for the 120 evaluated trials §5.3 tab. 1. Transfer experiments reuse one policy from a single "R" demonstration for letter and tower variants, 10 trials each §5.4. Baselines are GraspVLA, OpenHelix, GR00T N1, ChatVLA and DiffusionVLA §5.4 tab. 4.

Task representation (video pipeline, §6.1 fig. 10):

LLM planning (§6.2 fig. 11, normalised scores; the text gives the percentages and 5-point CI scores):

Block manipulation (§6.3 tab. 1; 120 evaluated trials):

Ablation (§6.4 tab. 2; TSR on the two transfer scenarios):

Transfer from one demonstration (§6.5 tab. 3; 10 trials each §5.4):

Comparison with baselines (§6.5 tab. 4, TSR only):

Metric definitions used in Results:

Graph representation accuracy eq. 9:

\mathrm{GRA} = \frac{1}{T}\sum_{t=1}^{T} \mathbb{I}[\mathcal{G}_t = \mathcal{G}^*_t]

Task segmentation accuracy eq. 10:

\mathrm{TSA} = \frac{1}{|B^*|}\sum_{i=1}^{|B^*|} \mathbb{I}\left[\min_j |\hat{t}_j - t^*_i| \le \tau\right]

Plan coverage eq. 11:

PC = \frac{|\mathcal{T}_{gt} \cap \mathcal{T}_{llm}|}{|\mathcal{T}_{gt}|}

Ordering accuracy, from Kendall's tau eq. 12:

OA = \frac{1 + \tau(\mathcal{T}_{gt}, \mathcal{T}_{llm})}{2}

Chain-of-thought interpretability eq. 13:

CI = \frac{1}{3N}\sum_{i=1}^{N}\sum_{j=1}^{3} r_{ij}

Verification correctness eq. 14:

VC = \frac{1}{|\mathcal{T}_{gt}|}\sum_{i=1}^{|\mathcal{T}_{gt}|} \mathbb{1}(v_i = \hat{v}_i)

Placement error eq. 15:

\mathrm{6DPE} = \sqrt{\lVert \mathbf{p}_{est} - \mathbf{p}_{gt} \rVert^2} + \lambda\,\theta

Instruction compliance eq. 16:

\mathrm{ICS} = \frac{1}{kN}\sum_{i=1}^{N}\sum_{j=1}^{k} r_{ij}

Bimanual coordination eq. 17:

\mathrm{BCS} = \frac{1}{N}\sum_{i=1}^{N} s_i

Grasp success (GSR), place success (PSR) and plan transferability (PTR) are ratios of successes to attempts §5.3 §5.4 and have no numbered equation.

Paper vs code

No official code.

What was searched:

• The paper text, its arXiv comments and the arXiv metadata contain no code link. A grep of the LaTeX source for "github" and "available" found none.
• GitHub repository search for "GF-VLA" returned three repositories: liampaull/gfn-vla-toy (a toy GFlowNet/diffusion VLA), ara31750-lab/vlad-gft (no description) and gaolongsen/GF_VLA_Robot_Manipulation. The last is owned by co-author Longsen Gao, but its contents are a static academic project-page template with no model, training or evaluation code, so it is not used as code. The first two are unrelated.
• A GitHub user search for "shunlei li" returned one account; its only repository is a fork of a GitHub Pages site.
• A repository search for "entropy scene graph dual arm VLA" returned nothing.

Pitfalls

• Entropy units. Equation 1 uses the natural log, but the text says entropy is measured in bits §3.1. The constant ε is "in R+" in eq. 1 and "in (0,1)" in the text §3.1. The unit choice changes the scale of the thresholds, and the paper does not settle it.
• Equation 4 is mislabelled. It is called "joint entropy" §3.1, but it is the term subtracted in eq. 3. How the joint histogram is binned in two dimensions is not stated.
• Frame count. A 1 s window at 30 Hz is 30 frames, but the text says 40 frames §5.1. The paper does not say which one was used; state the choice in your implementation.
• Dimensions. The method sums MI over x, y and z §3.1. The experiments compute entropy and distances from x and y only §5.1. Pick one and say which.
• Symbol mismatch. The quantisation width is ζ in §3.1 and ξ in §5.1. The Coupled-Motion threshold (α), the Docked threshold (γ), the hand gate (r_th,ho vs d_th) and the object gate (r_th,oo vs d_oo) are used under different names. Only ε_MI = 0.05, d_th = 0.15 m and d_oo = 0.2 m are given numerically.
• Docked rule. The text gives both "Docked if MI is falling and below γ, or if Docked before" and "Docked can only follow Coupled-Motion" §3.2.1. The pseudocode above follows both. Test the variant without the extra condition.
• Cross-hand policy. eq. 7 uses only the hand-to-target distances. The hand-to-source distances defined in §3.2.3 do not enter the rule. The reward in eq. 8 has no stated values and no stated update rule, so the "learning" policy cannot be reproduced from the text.
• LoRA scope. The text says the LLaMA backbone and the LoRA adapters are updated §4.4, which differs from the usual LoRA setup of a frozen backbone. The rank, targets, optimiser and schedule are missing.
• Architecture wording. The model is called a "transformer encoder" §4.2, but the backbone is a 7B LLaMA 2 model. How the action head attaches to the model and its output dimensions are not stated.
• Placement error. eq. 15 needs λ, which is not stated. The table reports position and angle separately, so 6DPE cannot be recomputed from the paper.
• Trial counts. Each transfer variant has 10 trials §5.4, so TSR values such as 0.87 or 0.93 are not possible as counts of 10. The paper does not say how the reported values were averaged. The "±" spread in the tables is not defined.
• Data overlap. The representation assessment uses "the 250 videos" §5.1, which include the videos used to train the LLM head §4.4. The "single human demonstration" transfer claim §5.4 sits beside the 125 training demonstrations used for that head §4.4.
• Baselines. The paper does not say whether the five baselines were re-run, nor with what settings. Table 4 reports TSR only. The conclusion says the method halves the 6D pose error compared with imitation-learning and language-only baselines §8, but no table reports those baselines.
• Unsupported claims. The discussion mentions UMAP projections and a stability analysis across participants §7. No figure or number for these appears in the text.
• Costs and assumptions. Instruction inference takes 7.56 s on average §4.4, so the 5 Hz action stream is separate from planning latency. The paper assumes pre-calibrated camera-arm alignment and a static workspace §7.
• Repeated headings. "Task representation assessment" appears as §5.1 and as §6.1. Cite by number to avoid confusion.

Lineage

• Builds on: the VLA line of OpenVLA and pi0 §2.1 (no cards yet); LoRA for parameter-efficient tuning §4.4; chain-of-thought prompting §4.3. The data protocol follows Merlo et al. §4.4 (cited by name only).
• Compared against: GraspVLA, OpenHelix, GR00T N1, ChatVLA and DiffusionVLA §5.4 tab. 4.
• Superseded by: not known from this wiki.