Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper

In plain words

Robot arms that learn from people often use a handheld gripper with a camera, so the robot sees what the human saw. Most of these grippers have no sense of touch, which matters when a fingertip has to feel whether a test tube is seated or how hard a pipette is squeezed. The camera can also be blocked or dimly lit, so touch is a useful second signal.

This work adds thin, flexible pressure pads to the fingers of such a gripper. The pads record where and how hard they press, while the camera films the same moment. The team gathered a large set of these recordings in grocery stores, outdoor markets and other messy places, then trained a model to predict the touch readings from the camera image and the other touch readings. To do that well, the model has to learn how looking and touching relate to each other.

That model is then plugged into a robot policy that learns from demonstrations. On four fine-grained tasks, such as inserting a test tube into a rack and moving liquid with a pipette, the authors report higher success than a vision-only policy, and than the same model without this pretraining.

TL;DR

• Hardware: a handheld gripper whose two soft fingers each carry a flexible 12×32 tactile pad, synced with a fisheye camera. Total weight about 962 g §3.1 §3.2.
• Method: Stage 1 pretrains a visuo-tactile encoder by reconstructing masked tactile images. Stage 2 feeds that encoder and proprioception into a diffusion policy §4.2 §4.3.
• Data: over 2,700 demonstrations, 2.6 million visuo-tactile pairs, 43 tasks, 12 environments §1 §5.1.
• Result (author-reported, 20 trials per cell): test-tube whole-task success 0.85 with pretraining, 0.70 without, 0.25 vision-only tab. 1.
• Use it if you need contact-rich manipulation with touch, collected in the wild. Read the pitfalls first: sample sizes are small and several implementation details differ between the paper and the code.

Problem

• Most tactile sensors are bulky, rigid or not robust enough for handheld or outdoor use. Soft bubble sensors are too large for a handheld gripper, and optical sensors are often bulky or fragile over long use §1.
• Vision alone degrades under poor lighting, background clutter and occlusion, while touch gives local contact information that vision often cannot see §1 §4.1.
• In-the-wild gripper systems so far are mostly vision-only, and no large public in-the-wild dataset with synchronized RGB and tactile signals existed before this work §2.
• Learning from multimodal data is hard: the two signals differ in scale and in what they describe, and the representation has to keep both §1.
• Contrastive alignment of modalities tends to suppress the fine-grained, geometry-sensitive tactile signal, so the authors choose masked reconstruction instead §4.2.

Method

Hardware and data collection §3

• Each tactile pad is a piezoresistive layer between two flexible printed circuits. The design builds on 3D-ViTac's triple-layer design, with FPC electrodes replacing stainless steel for uniform trace pitch §3.1.
• Each pad has 12×32 taxels, each covering 2×2 mm². The authors report a fabrication time of under five minutes per pad and two minutes to mount it §3.1.
• Two custom Arduino-based boards sit under the palm. The pad streams at 23 Hz over USB to a laptop, and frames are timestamped on the microcontroller §3.2.
• A fisheye camera is mounted on the gripper. A GoPro Hero 9 at 60 Hz is used in the data pipeline §3.2.
• Synchronization is hardware-free: a QR code showing host time is shown to the camera at 30 Hz before each demonstration, tactile packets carry host-clock timestamps via ROS2, and the QR sequence is decoded offline §3.2.
• The whole handheld unit with batteries weighs about 962 g §3.2.
• The system is built for use in grocery stores, outdoor markets and other unstructured environments §3.2.

Stage 1: visuo-tactile representation §4.2

The encoder E and decoder D are trained jointly to reconstruct the full tactile image from a partially masked tactile input and the image eq. 1.

• Tactile input. The two fingertip arrays, each 1×12×32, are stacked vertically into a 1×24×32 image §4.1. A fixed colormap turns this into a 3-channel image. The image is cut into non-overlapping 4×4 patches, giving a 6×8 grid §4.2.
• Masking. In 95% of samples, 60–80% of patches are replaced by a learnable mask token. The other 5% are shown unmasked §4.2.
• Tactile embedding. A 3-layer CNN maps the masked tactile image to a d-dimensional vector, with d = 768 §4.2.
• Vision embedding. A ViT-B/16 initialised from CLIP processes the RGB image at 224×224. All layers are fine-tuned at learning rate 3×10⁻⁵, and the final [CLS] token is the visual embedding §4.1 §4.2.
• Cross-modal fusion. Two rounds of multi-head cross-attention with LayerNorm update each embedding from the other eq. 2 eq. 3. The fused representation is the concatenation of the two updated embeddings, of size 2d eq. 4.
• Decoder. A two-layer MLP with a sigmoid output reconstructs the tactile image of size 1×24×32 §4.2.
• Target stabilisation. A target encoder is kept as an exponential moving average of the online weights with decay 0.9995. The EMA encoder is used for checkpointing and attention visualisation §4.2.

The Stage 1 training step, written from the paper's equations and checked against the code:

# Stage 1: one training step of masked tactile reconstruction  [§4.2]
# Input:  rgb (B, 3, 224, 224); tactile (B, 12, 64), values in [0, 1]
# Output: scalar loss

def stage1_loss(rgb, tactile):
    # 1. Colour-map the two fingertip arrays, stacked to 24x32  [§4.1]
    colour = colour_map(stack_fingertips(tactile))   # (B, 3, 24, 32)
    # 2. Hide 60-80% of the 48 patches (4x4) in most samples  [§4.2]
    masked = hide_patches(colour, mask_token)        # (B, 3, 24, 32)
    # 3. Keep the [CLS] token of a CLIP ViT-B/16 on the image  [§4.2]
    image_cls = clip_vit_b16(rgb)[:, 0]              # (B, 768)
    # 4. Embed the masked tactile image with a 3-layer CNN  [§4.2]
    tactile_emb = tactile_cnn(masked)                # (B, 768)
    # 5. Tactile attends to image, then image to tactile  [eq. 2] [eq. 3]
    tactile_emb = layer_norm(tactile_emb + attend(tactile_emb, image_cls))
    image_cls = layer_norm(image_cls + attend(image_cls, tactile_emb))
    # 6. Concatenate both embeddings into the fused representation  [eq. 4]
    fused = concat(tactile_emb, image_cls)           # (B, 1536)
    # 7. Decode the full tactile image, sigmoid output  [§4.2]
    prediction = sigmoid(decoder(fused))             # (B, 1, 24, 32)
    # 8. Squared error over the full image against the target  [eq. 1]
    return mean_squared_error(prediction, stack_fingertips(tactile))

Implementation: pretrain_mae_model.py (encoder), pretrain_mae_model.py (forward), pretrain_mae.py (loss). The code adds residual connections inside attend and uses a single-token key and value; see Paper vs code.

Stage 2: policy learning by behaviour cloning §4.3

• Observation. At time t the robot receives an RGB image I_t, a tactile image T_t and proprioception p_t (end-effector pose and gripper width). The pretrained encoder gives z_t = E(I_t, T_t), and the policy is conditioned on o_t = (z_t, p_t) §4.3.
• Diffusion policy. A conditional diffusion policy from Chi et al. predicts noise rather than actions. The training loss is the MSE between the added noise and the predicted noise eq. 5. Inference starts from Gaussian noise and iterates a denoising update with hyperparameters α, γ and σ eq. 6. The paper does not give their values §4.3.
• Network and fine-tuning. The denoiser is Diffusion Policy's convolutional U-Net, sampled with DDIM. The CLIP backbone, the tactile CNN and the cross-attention layers are all fine-tuned at learning rate 3×10⁻⁵ §4.3.
• Proprioception. The conditioning includes two consecutive proprioceptive observations §4.3.

# Stage 2: one behaviour-cloning step with a diffusion policy  [§4.3]
# Input:  two frames of rgb and tactile, proprioception, expert actions
#         actions (B, 16, 10)
# Output: scalar loss

def stage2_loss(frames, proprio, actions):
    # 1. Encode each frame with the pretrained encoder, no masking  [§4.3]
    embeddings = encode_unmasked(frames)             # (B, 2, 1536)
    # 2. Condition on the flattened embeddings and proprioception  [§4.3]
    condition = concat(flatten(embeddings), proprio)
    # 3. Sample a diffusion step and the added noise  [§4.3]
    timestep = random_int(0, 50)                     # 50 train steps
    noise = normal(0, 1, shape=actions.shape)
    # 4. Extra input noise, code only: see Paper vs code
    noisy_input = noise + 0.1 * normal(0, 1, shape=actions.shape)
    # 5. Forward diffusion of the actions  [§4.3]
    noisy_actions = add_noise(actions, noisy_input, timestep)
    # 6. Predict the noise; loss is the MSE to the added noise  [eq. 5]
    predicted = unet(noisy_actions, timestep, condition)
    return mean_squared_error(predicted, noise)

# Inference: sample an action chunk with DDIM  [§4.3] [eq. 6]
# Input:  frames, proprioception
# Output: actions (B, 16, 10)

def sample_actions(frames, proprio):
    # 1. Start from Gaussian noise  [eq. 6]
    actions = normal(0, 1, shape=(B, 16, 10))
    # 2. Same conditioning as in training  [§4.3]
    condition = build_condition(frames, proprio)
    # 3. Run 16 denoising steps  [§4.3] [tab. 5]
    for step in ddim_steps(num_steps=16):
        # 4. Predict the noise at this step  [eq. 6]
        noise_hat = unet(actions, step, condition)
        # 5. Denoise; alpha, gamma, sigma are not stated  [eq. 6]
        actions = ddim_update(actions, noise_hat, step)
    return actions

Implementation: diffusion_unet_timm_policy.py (loss), timm_obs_encoder.py (tactile branch), train_diffusion_unet_timm_umi_workspace.yaml (DDIM scheduler).

Hyperparameters

Stage 1: pretraining

Augmentation during pretraining is not stated in the paper. The code uses a random crop to 95% of the side length, resized to 224, followed by colour jitter (brightness 0.3, contrast 0.4, saturation 0.5, hue 0.08) pretrain_mae.yaml.

Stage 2: policy

Results

Author-reported. The setup is four real-world tasks, 20 trials per task with moderate randomisation of the initial robot pose and environment, all policies trained for 60 epochs §5.2 tab. 1. The robot is an XArm 850 with the same sensor configuration fig. 2, with a parallel gripper in the experiments §6.

Baselines §5.2:

• Vision-only: one RGB image through CLIP (768-d) plus proprioception, following Chi et al.
• Ours w/o cross-attention: two tactile images through a 3-layer CNN (512-d), concatenated with the visual embedding and proprioception.
• Ours w/o pretraining: the same encoder with CLIP initialisation and the rest from scratch.
• Ours w/ pretraining: the same encoder, with visuo-tactile pretraining.

Table 1: real-world success rates, 20 episodes per cell tab. 1

Tasks requiring in-hand state information:

Demonstrations: test tube 197, pencil 170 tab. 1.

Tasks requiring fine-grained force information:

Demonstrations: fluid transfer 160, whiteboard erasing 105 tab. 1.

Sensor replacement tab. 2. Test-tube policy with pretraining, original sensors vs newly fabricated sensors: whole task 0.85 in both cases (grasp 1.00, reorient 1.00, insert 0.85 in both) app. A.

Held-out tasks tab. 3. Pretraining with all demonstrations of a task removed:

The pretraining set is about 2,500 demonstration videos in this check, against 2,700 in the full set app. B.

Data scaling and epochs. Pretraining ablations on test-tube collection, varying the number of demonstrations and training epochs, are shown only as a plot, with no numbers in the text fig. 7 §5.3. The text reports that pretraining helps most with fewer than 60 demonstrations and fewer than 60 epochs §5.3. Pretraining loss falls with more pretraining data fig. 8 app. B.

Robustness. The abstract's claim of robustness to external disturbances rests on qualitative observations in the text, not on a separate disturbance experiment in a table §5.3 §1.

Representation. ViT self-attention maps are reported to focus on the gripper–contact region for pretrained policies and on background edges for policies without pretraining, shown qualitatively fig. 5 fig. 10 app. C.

Paper vs code

The official repository is https://github.com/YolandaXinyueZhu/touch_in_the_wild, linked from the project page and owned by the first author, at commit 2a946261d34bee5ac363c5dc696810273a0b20d8 under the MIT licence. The README points to the Universal Manipulation Interface setup guide for system dependencies README.md. The README links a Hugging Face dataset and a pretrained MAE checkpoint; we did not check the licence of either.

Differences between the paper and the code:

• Masking is per batch, not per sample. The paper says 95% of samples are masked §4.2. The code draws the 5% unmasked decision once per call, so a whole batch is either masked or not, and the masking ratio is drawn once per batch pretrain_mae_model.py.
• Validation and evaluation are masked too. The masking is not tied to training mode. Evaluation calls the same forward pass, so validation loss uses random masks pretrain_mae_model.py pretrain_mae.py.
• Cross-attention has a residual connection and LayerNorm after the sum. The paper's equations show attention followed by LayerNorm, with no residual eq. 2 eq. 3 pretrain_mae_model.py.
• Attention keys and values are single tokens. In the default cls mode the key and value are the single [CLS] token pretrain_mae_model.py. The paper does not state the token count.
• Two directions, each applied once. The text says "two rounds" of cross-attention §4.2. The code applies one tactile-to-image step and one image-to-tactile step pretrain_mae_model.py. The equations match the code here.
• Positional embedding is unused in the default mode. The code adds a learned positional embedding to patch tokens, but the default cls mode never passes patch tokens to the output pretrain_mae_model.py pretrain_mae_model.py.
• Tactile CNN details not in the paper. The code uses three stride-2 convolutions with batch normalisation and ReLU, global average pooling and a linear layer to 768 pretrain_mae_model.py.
• Decoder dropout. The decoder has dropout 0.1 after its hidden layer pretrain_mae_model.py.
• Loss scale. The paper writes a squared L2 norm eq. 1. The code uses mean squared error over the batch pretrain_mae.py.
• Gradient clipping happens after optimizer.step(), so it does not affect the update it follows pretrain_mae.py.
• Stage 2 encoder learning rates differ. The paper says all encoder components use 3×10⁻⁵ §4.3 tab. 5. The workspace code sets the CLIP backbone to pretrain_clip_lr = 2×10⁻⁵, and the tactile CNN and cross-attention to 0.1 × the UNet rate = 3×10⁻⁵ train_diffusion_unet_image_workspace.py train_diffusion_unet_timm_umi_workspace.yaml.
• Training length. The paper says 60 epochs §5.2. The default config is 200 epochs train_diffusion_unet_timm_umi_workspace.yaml, and the README's training command does not override it README.md.
• Pretrained weights are off by default. The default pretrain_ckpt_path is the string None, not YAML null. The loader tests for None and silently skips loading if the path does not exist train_diffusion_unet_timm_umi_workspace.yaml timm_obs_encoder.py. The README passes the checkpoint path explicitly README.md.
• Policy conditioning covers two image frames. The code concatenates the encoder output of both frames, so the conditioning is 2 × 2d plus proprioception timm_obs_encoder.py umi.yaml. The paper text mentions two proprioceptive observations §4.3, while its table also lists an image horizon of 2 tab. 5.
• Executed actions. The policy config sets n_action_steps to 8 train_diffusion_unet_timm_umi_workspace.yaml while the action horizon is 16 umi.yaml. The paper does not state how many predicted actions are executed.
• Extra training settings not in the paper: weight decay 10⁻⁶, input perturbation of 0.1, an EMA of the policy with maximum decay 0.9999, and a 2000-step warm-up train_diffusion_unet_timm_umi_workspace.yaml diffusion_unet_timm_policy.py.

The following match: Stage 1 batch size 128, 5 epochs, seed 42, CLIP rate 3×10⁻⁵, encoder rate 10⁻⁴, AdamW with decay 2×10⁻³, 10% warm-up with cosine decay, EMA decay 0.9995, cross-attention dropout 0.2, 8 heads, d = 768, 60–80% masking of 4×4 patches, 10% validation split. Stage 2 matches the batch size of 64, 50 training and 16 inference diffusion steps, action horizon 16, AdamW betas, and 224×224 observations.

Pitfalls

• Masking granularity. Implement the decision per sample, as the paper says, or per batch, as the code does. The two give different gradient statistics, and the paper does not say which one its results used.
• Validation is masked. Validation loss in the code uses random masks, so it changes between evaluations even with the same weights.
• Single-token attention. With one key and one value, softmax weights are identically 1. Each attention step then reduces to a learned value-and-output projection of the other token, plus dropout on that weight. The paper's equations do not expose this, and the paper gives no token count. The code's patch mode attends over the 196 spatial patch tokens and excludes twelve corner patches with a key-padding mask pretrain_mae_model.py; the paper does not describe or evaluate it.
• Unstated units for taxel values. The paper does not state the scale of the raw sensor readings. The code clamps them to [0, 1] and indexes the colormap at 0–255 pretrain_mae_model.py. Readings outside that range are silently clipped.
• No CLIP normalisation. The code feeds RGB in [0, 1] to CLIP without mean and standard-deviation normalisation. The imagenet_norm flag is accepted but never used umi_pretrain_mae.py timm_obs_encoder.py. The paper does not say how the image is normalised.
• Frame subsampling. The pretraining sampler takes one sample every five frames umi_pretrain_mae.py. The paper reports 2.6 million pairs and does not state a sampling step, so the pair count is not the number of training samples.
• EMA and batch-norm statistics (code reading, not run). The EMA copy is built with AveragedModel and updated only through its parameters. The default in PyTorch's AveragedModel does not average buffers, so the batch-norm running statistics in the saved EMA weights may stay at their initial values pretrain_mae.py pretrain_mae.py. Stage 2 trains the same layers and should update those statistics, but we did not verify the effect. Treat the EMA checkpoint's batch-norm state with care.
• Small samples. Each cell is 20 trials, so one trial moves a success rate by 0.05. Several differences are one or two trials: whiteboard whole task 0.70 vs 0.60 without pretraining, and held-out 0.70 vs 0.75. No error bars are given.
• Held-out result is not a clear win. The policy without task data scores higher on whiteboard whole task (0.75 vs 0.70). The paper calls the two comparable tab. 3 app. B.
• Vision-only baseline details. The paper does not state whether CLIP is frozen in this baseline.
• Compute. Stage 2 runs the CLIP ViT-B/16 on both frames of every sample at every step, which is the main cost timm_obs_encoder.py. The paper does not report GPU hours or hardware for training.
• Hardware. The method needs the custom gripper with FPC pads, which are fabricated per unit, plus the QR-based sync. The paper gives fabrication times but no per-unit cost.

Lineage

• Builds on: Diffusion Policy (Chi et al.) for the policy §4.3; masked autoencoders (He et al.) for the reconstruction objective §4.2; CLIP (Radford et al.) for the vision backbone §4.2; 3D-ViTac (Huang et al.) for the tactile pad design §3.1; the adaptive fin-shaped gripper (Chi et al.) for the finger geometry §3.1. The README points to the Universal Manipulation Interface setup guide README.md.
• Compared against: a vision-only policy with a CLIP embedding, following Chi et al. §5.2; and "w/o cross-attention", an ablation of this paper's own encoder that uses concatenation instead of cross-attention §5.2.
• Related, concurrent: ViTaMIn, a portable visuo-tactile data collection system with a cross-modal learning framework, which the paper calls a concurrent project §2.
• Superseded by: not known from this wiki.