Reward Machine Inference for Robotic Manipulation

In plain words

A robot that learns a task from demonstrations is often given a task plan written by hand: which steps must happen, in which order, and what counts as a step being done. This paper removes that hand-written part. The robot is shown several videos of a task, such as stacking blocks. It groups the frames that look alike, because the moments when the scene pauses at a useful point (a block placed, a stack started) show up in many demonstrations. Each group gives one "milestone" picture, and the order in which the demonstrations move between milestones gives a small graph of the task.

A reinforcement learning agent then uses that graph to learn the task. Its reward also reflects how far it is along the graph, not only the final success, which makes learning easier. In the simulated block tasks the agent learned about as well as one given the true milestones, with one exception noted in the results.

TL;DR

• Infers a reward machine (a finite-state graph of sub-goals with a reward structure) from unlabelled RGB demonstrations, with no predefined propositions §1 §4.
• Pipeline: pretrained ResNet-50 features, DBSCAN clusters, one prototype frame per cluster, and distance thresholds as propositions §4.2–§4.4.
• Five simulated Franka Panda block tasks; the author-reported learning curves of the inferred-machine agent are similar to a ground-truth-machine agent's, except on Stack-3 §6 fig. 5.
• No numeric results table is published: the evidence is in figures and prose. The code is not released.

Problem

Reward machines help reinforcement learning over long horizons, but most approaches assume an expert writes the machine, or assume predefined propositions and a labelling function that maps states to them §1 §2. The paper targets vision-based manipulation, where the propositions are not given and the sparse task reward is hard to learn from §1 §3.2 §4.4.

Method

The section numbers below count the paper's headings in order. The compiled paper prints no section numbers (secnumdepth is 0 in the source), so §4.3 means the third subsection of the fourth heading, "Inferring Sub-Goals through Clustering". Figure numbers follow source order.

Step 1, demonstrations. A front-view camera records 256 × 256 × 3 frames of an expert executing the task; each frame is a state §4.1. A top-down camera records the same demonstrations, used only for policy training §4.1.

Step 2, features. Each frame is cropped to the tabletop, then mapped to a feature vector φ(s) by a ResNet-50 pretrained on ImageNet §4.2 §6.

Step 3, sub-goal clusters. All feature vectors from all demonstrations are clustered with DBSCAN, which needs no preset cluster count and tolerates noise §4.3. The paper's reasoning is that sub-goals are visited more often than other states §4.3 §2. Each cluster C_i has a mean μ_i eq. 3.

Step 4, prototypes. The prototype s_i* of cluster C_i is the frame whose feature is closest to μ_i in Euclidean distance eq. 4 §4.3. Prototypes are shown as images, which makes the sub-goals readable §4.3 fig. 3 fig. 4.

Step 5, propositions and RM states. Each prototype gives one RM state u_i and one proposition p_i. p_i is true for state s when the feature distance to s_i* is below a threshold κ eq. 5 §4.4. κ must be set so that at most one proposition is true per frame, since the machine can occupy only one state §4.4.

Step 6, transitions. Each demonstration becomes a sequence of sets of true propositions. Algorithm 1 walks through these sequences and records, for each (state, proposition) pair, the RM state it leads to alg. 1 §4.4.

# Infer transitions from abstract demonstrations  [alg. 1]
# Input:  demos: list of sequences; each step is the set of true propositions
# Output: delta_u: dict (state, proposition) -> state

def infer_transitions(demos, initial_state, state_of):
    # Keep only the first observed transition for each pair  [alg. 1]
    delta_u = {}
    for demo in demos:
        # Every demonstration starts in the initial RM state u_0  [alg. 1]
        state = initial_state
        for true_props in demo:
            for prop in true_props:
                if (state, prop) not in delta_u:
                    # The target is the RM state of this proposition  [alg. 1]
                    delta_u[(state, prop)] = state_of(prop)
                # Move along the transition; with several true propositions
                # the last one in the loop sets the state  [alg. 1]
                state = delta_u[(state, prop)]
    return delta_u

Step 7, reward machine and shaping. The sparse reward δ'_r is the base reward. The shaped reward adds a potential Ψ(u) = γ^{d_goal(u)}, where d_goal is the number of edges from u to the goal eq. 6 §4.4. The shaped transition reward is δ_r(u, u') = δ'_r(u, u') + γΨ(u') − Ψ(u) eq. 7 §4.4. The paper states that this is zero when the agent moves one step closer to the goal, slightly negative when it stays, more negative when it moves away, and 1 at the goal §4.4. Substituting Ψ = γ^d gives those values for transitions that move by one edge, and 1 for arrival from distance 1 when δ'_r = 1 at the goal.

# Potential-based shaped reward  [eq. 6] [eq. 7]
# Input:  sparse_reward, u, u_next (RM states), gamma
# Output: shaped reward (float)

def shaped_reward(sparse_reward, u, u_next, gamma, d_goal):
    # Potential of each RM state: gamma to the edge distance to goal  [eq. 6]
    potential = gamma ** d_goal(u)
    potential_next = gamma ** d_goal(u_next)
    # Potential-based shaping keeps the optimal policy  [eq. 7] [§4.4]
    return sparse_reward + gamma * potential_next - potential

Step 8, DQN on the inferred machine. The paper uses a deep Q-learning agent with two fully convolutional networks, ψ_pick and ψ_place, which output pixel-wise Q-value maps over pick and place locations §5 fig. 2. Both share one architecture: a ResNet-50 pretrained on ImageNet, an intermediate feature map from its second residual block, two convolutions that reduce it to one channel, and bilinear upsampling to the input size §5. Only Q-values inside the table rectangle count; the paper gives no coordinates §5. The pick network takes the current top-down observation. The place network also takes a top-down prototype for each valid transition out of the current RM state; how the prototype enters the network is not stated §5 fig. 2. Training follows the DQRM approach: one Q-function per RM state, each with its own experience buffer that also stores demonstrations, all updated at the end of each episode from sampled batches §5. The loss is the Huber TD loss, with targets as in the background section eq. 1 eq. 2 §3.1. Gradients flow only to the Q-value at the executed action's pixel; all other pixels get zero loss, and all parameters, including the ResNet layers, are updated §5.

The Q-function is Q_θ(s, a) for a pick-and-place action, a = (pick, place) in pixel coordinates, mapped to robot coordinates via calibration and depth §5.

Hyperparameters

Per-task values from tab. 1:

Results

Author-reported. Setup: five simulated block tasks in robosuite with a Franka Emika Panda, demonstrations from an algorithmic expert policy, 10 runs each, compared with a DQN agent given the ground-truth machine and propositions §6 fig. 5. The paper publishes no numeric table of final rewards, so values are given as the paper states them.

Learned behaviour: one policy path is learned even when several valid paths exist in the machine §7.

Paper vs code

No official code.

The paper has no code link in its text, abstract or LaTeX source, and no project page is linked. Searches run: GitHub repositories for "reward machine inference robotic manipulation", "reward machine inference", "baert reward machine", "reward machine robosuite", and "IDLab ghent reward machine LfD"; a GitHub user search for "mattijs baert". None matched the paper.

Pitfalls

• Demonstrations are contradictory. The Results section says "unstructured human demonstrations" in one sentence and "generated by an algorithmically defined expert policy" in the next §6. The method is only tested with scripted demonstrations.
• Feature extractor is underspecified. The ResNet-50 layer, pooling, output dimension, normalisation and crop box are not stated §4.2 §6. Because κ and ε_cluster are Euclidean distances in this feature space, the tuned values are not portable to a different extractor, crop or image size.
• Tuning is undocumented. κ, ε_cluster and min points are "tuned for each scenario" §6. The tuning data, objective and search are not stated. The per-task min-points values range from 15 to 110 tab. 1.
• κ must make propositions mutually exclusive. The paper requires that at most one proposition holds per frame §4.4 but gives no check or fallback for frames where none or several hold. Algorithm 1 records only the first observed transition for each (state, proposition) pair alg. 1, so conflicting later transitions are dropped silently.
• Algorithm 1 is ambiguous on multiple true propositions. Its inner loop moves the state for each true proposition in turn. The paper does not say which proposition wins when several hold. The card's pseudocode follows the listing literally and flags this.
• Initial state is not defined by a rule. The "0-prototype" is the initial RM state fig. 1 §4.4, but the paper does not say how the cluster that starts a demonstration is chosen.
• DBSCAN noise is unaddressed. Points labelled as noise by DBSCAN are not mentioned. It is not stated whether they are dropped, or whether they affect the cluster means μ_i §4.3.
• Cameras and calibration. Inference uses the front camera and control uses the top-down camera §5. Calibration and depth are needed to map pixels to the robot §5. The ground-truth baseline needs object positions at every step §6; this is available only in simulation.
• Shaping depends on γ. The potential Ψ = γ^d uses γ, and the discount value is not stated §4.4 §6. The paper assumes the same γ is used in the TD target, which is not stated either.
• Q-network input for the place action. How a transition prototype is combined with the current top-down image is not stated §5 fig. 2.
• Training run length and buffer behaviour are missing. Number of episodes, total steps, replay size, target-network use and the exponential ε schedule length are not stated §6. Results are shown as curves in figures only fig. 5.
• Stalling. The paper says prototype-matching errors can stall the agent in one RM state until the episode ends §6. No mitigation is described.
• Single path. Only one path between the start and goal is learned even when several are valid §7. Place-3 needed six demonstrations to cover the orderings §6 tab. 1.
• Simulation only. The evaluation is in robosuite with a Franka Panda and block tasks §6. Nothing is shown on a real robot or on other domains §7.

Lineage

• Builds on: reward machines (Icarte et al., 2022) for the machine structure §3.2; the DQRM approach (Icarte et al.) for one Q-function per machine state §5; potential-based reward shaping (Ng et al., 1999) §4.4; and the vision-based RM work of Camacho et al. (2021), which this paper extends by removing its predefined propositions §2 §4.4.
• Compared against: an agent with a ground-truth RM and predefined propositions, following the setup of Camacho et al. §6.
• Related prior work by the authors: learning temporal task specifications from demonstrations, cited for the sub-goal visitation assumption §2 §4.3.
• Superseded by: not known from content in this wiki.