ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data

In plain words

A camera often cannot tell whether a tool is touching a surface, or whether a cup holds anything. Sound from contact carries that information. A small contact microphone, pressed against one gripper finger, picks up the scrapes, taps and rattles that happen when the finger touches something.

The paper has two parts. The first is a handheld gripper, based on the UMI data-collection device, with that microphone built in. People can record demonstrations anywhere, and the sound and video are saved together. The second is a learned policy that reads the camera images and the sound, and outputs the robot's next motions. Sound recorded by hand and sound recorded on the robot sound different, so the training sound is mixed with background noise and with recordings of the robot's motors.

The authors test four tasks that need contact: wiping a drawn shape off a whiteboard, flipping a bagel in a pan, pouring dice out of a cup, and picking the right velcro tape. They also report that demonstrations collected in varied places help the policy work on surfaces it was not trained on.

TL;DR

• A UMI-style handheld gripper with a contact microphone on one finger records synchronised video and audio from human demonstrations §3.1.
• The policy encodes two camera frames with a CLIP ViT-B/16 and the audio with an audio spectrogram transformer (AST), fuses them with a transformer layer, and predicts 16 steps of 10-D actions with a diffusion head §3.2.
• Background sounds and robot-motor noise are overlaid on the training audio, each with probability 0.5 §3.2.
• Author-reported: higher overall success than the vision-only policy and the listed ablations on wiping, flipping and pouring, and better material sensing on taping §4. One appendix success table does not add up (see Pitfalls).

Problem

Vision alone is ambiguous about contact. The wrist camera often cannot show whether a spatula touches the pan, or whether a finger is in contact with a board §4.1 §4.2 app. B.2.3. Earlier audio-based policies needed microphones attached to the robot or to objects, and teleoperated data, which the authors say limits how much data can be collected §2. Their premise is that a contact microphone can be carried by a human-held gripper, so audio can be recorded in the wild with the same video pipeline §1 §3.1.

Audio collected by a human-held device differs from audio at deployment. The robot's own motor noise and sounds from unexpected collisions are absent from the human data §3.2. This is the domain gap the method targets.

Method

Hardware §3.1. The UMI parallel-jaw gripper is modified: one finger has a piezoelectric contact microphone under high-friction grip tape. The microphone plugs into the external mic port of a GoPro media mod. Audio is recorded at 48 kHz and stored with 60 Hz video in MP4 files §3.1. At deployment the same gripper is mounted on a UR5 arm, and images and audio are streamed through a capture card to a desktop §3.1.

Training-time audio augmentation §3.2. Each training audio window gets two independent overlays, each applied with probability 0.5:

1. a background clip sampled from ESC-50 §3.2;
2. a recording of robot-motor noise taken with the same microphone placement as at deployment §3.2. The paper says the overlay sounds are scaled to the level of the training audio §3.2. The authors argue this forces the model to use task-relevant sound and to ignore unpredictable noise §3.2.

Vision encoder §3.2 app. A.2. Each frame is randomly cropped to 95% and resized to 224 x 224, then colour-jittered. The encoder is a CLIP-pretrained ViT-B/16 and is trained end-to-end. The frames are sampled at 20 Hz, and the two most recent frames are used. Each frame's classification-token feature is kept.

Audio encoder §3.2 app. A.2. The most recent 2 to 3 seconds of audio, depending on the task §3.2, are resampled from 48 kHz to 16 kHz. A log-mel spectrogram with 64 bins, a 400-sample window and a 160-sample hop is computed, then linearly scaled to [-1, 1]. An AST encoder, trained from scratch rather than pretrained, produces the audio feature from the last hidden layer's classification token. The authors prefer a transformer to a CNN because shifts in time and frequency change the meaning of a spectrogram §3.2. They report that a from-scratch transformer beats both a pretrained and a from-scratch CNN encoder §3.2.

Sensory fusion §3.2 app. A.2. The vision feature of each frame and the audio feature are the inputs to one transformer encoder layer with 8 heads, a 2048-unit feed-forward block and no dropout. The output is concatenated and projected to 768 dimensions. The end-effector poses of the two most recent timesteps (20 Hz) are then appended §3.2.

Policy §3.2. The fused feature conditions a diffusion head, which is the UNet design of Chi et al. §3.2. The head is applied at every denoising step. The whole model is trained end to end with a noise-prediction MSE loss on future trajectories of 16 steps §3.2.

Latency matching §3.2 app. A.1. The audio stream lags the image stream by about 0.06 s, and the image latency is 0.17 s. The total audio latency used at deployment is 0.23 s. The authors compensate for it the way UMI does §3.2. The calibration is done by tapping the microphone while the finger is seen touching it app. A.1.

# Training step for one batch  [§3.2]
# frames: (B, 2, 3, 224, 224)  audio: (B, 96000) at 48 kHz
# poses: (B, 2, 10)  actions: (B, 16, 10)
def training_step(frames, audio, poses, actions):
    # 1. Overlay ESC-50 background noise with p = 0.5  [§3.2]
    audio = maybe_add_background_clip(audio, probability=0.5)
    # 2. Overlay robot-motor noise with p = 0.5  [§3.2]
    audio = maybe_add_robot_noise(audio, probability=0.5)

    # 3. Resample to 16 kHz, take log-mel (64 bins), scale to [-1, 1]  [§3.2]
    spectrogram = log_mel(resample(audio, from_hz=48000, to_hz=16000))
    spectrogram = scale_to_unit_range(spectrogram)

    # 4. Encode each frame with CLIP ViT-B/16, keep the CLS token  [§3.2]
    vision_tokens = [clip_cls(crop_and_jitter(f)) for f in frames]

    # 5. Encode the spectrogram with an AST trained from scratch  [§3.2]
    audio_token = ast_cls(spectrogram)

    # 6. Fuse vision and audio with one transformer layer  [§3.2] [app. A.2]
    fused = transformer_layer(stack(vision_tokens, audio_token))

    # 7. Project to 768 and append the poses of the last 2 steps  [§3.2]
    observation = concat(linear(concat(fused)), concat(poses))

    # 8. Noise a 16-step action chunk at a random diffusion step  [§3.2]
    noise = random_normal(actions.shape)
    step = random_integer(num_train_steps)
    noisy_actions = add_noise(actions, noise, step)
    prediction = unet(noisy_actions, step, condition=observation)

    # 9. Noise-prediction MSE, trained end to end  [§3.2]
    return mean_squared_error(prediction, noise)

The pseudocode condenses the paper's description. Several choices that change the result are only in the code (see Paper vs code).

Hyperparameters

Results

Author-reported, on the robot, with 20 rollouts for wiping and flipping, 12 for pouring and 10 for taping §4.1 §4.2 §4.3 §4.4. Baselines are ablations of the same system or earlier designs, such as AVID audio pretraining §4.1.

The wiping gain is stated in the text as 40% to 85% §4.2. Overall pouring and taping rates, and the in-the-wild flipping rates, are shown only in charts fig. 5 fig. 7 fig. 8, which this card did not extract. Limitations named by the authors are objects that do not make salient sound, quasi-static tasks, and loss of the sound signal under robot motor noise. They also list the lack of use of the higher audio rate for reactive control §4.5.

Paper vs code

The official repository is real-stanford/maniwav (MIT), linked from the project page LICENSE. The code checked is at commit dc7d0ab6fd2c8528b0de92b5fc9458a038900653. The repository also contains the UMI and Diffusion Policy code and other tasks; the ManiWAV path is the umi_maniwav task with train_diffusion_unet_maniwav_workspace.py.

• Audio window. Paper: last 2 to 3 s, depending on task §3.2. Code: one value, 120 frames of 800 samples, which is 2 s at 48 kHz umi_maniwav.yaml umi_maniwav.yaml. The 1 s and 10 s pouring ablations §4.3 have no config in the repository.
• Noise augmentation scale. Paper: the sounds are normalised to the level of the training audio §3.2. Code: the ESC-50 clip and the robot noise are added sample by sample, with no gain matching or resampling noise_aug.py noise_aug.py.
• Robot-noise samples. Paper: 10 recordings §3.2. Code: one file, data/robot-noise-calib/robot.wav, which is in the repository umi_maniwav.yaml.
• Latency during training. The camera and audio latency offsets are used at deployment eval_real_umi.py. In the training config, dataset_frequeny is 0, so every latency_steps value is 0 umi_maniwav.yaml. The training config also keeps a camera latency of 0.125 s, the same value as the UMI environment default umi_maniwav.yaml umi_env.py. Training therefore does not apply the 0.17 s and 0.23 s offsets.
• Spectrogram. Appendix: STFT and mel parameters from torchaudio app. A.2. Code: the active audio path uses transformers' ASTFeatureExtractor with 64 mel bins and max_length 200 [code: diffusion_policy/model/vision/maniwav_obs_encoder.py#L108-L118] maniwav_obs_encoder.py. The frame length and shift come from that library, not from this repository. The commented-out torchaudio MelSpectrogram block uses n_fft 400, hop 160 and 64 bins train_diffusion_unet_maniwav_workspace.yaml.
• Scaling range. Paper: linear scaling to [-1, 1] §3.2. Code: uses fixed min -16.0 and max 10.6 from the config, but replaces them with statistics from spec_stats.pkl when the file exists in the stats directory maniwav_obs_encoder.py train_diffusion_unet_maniwav_workspace.yaml. The workspace writes that file while computing the normalizer umi_maniwav_dataset.py, after the model is built train_diffusion_unet_maniwav_workspace.py.
• Loss. Paper: noise-prediction MSE §3.2. Code: the noisy input gets an extra Gaussian perturbation of scale 0.1, but the regression target is the clean noise diffusion_unet_maniwav_policy.py train_diffusion_unet_maniwav_workspace.yaml.
• Low-dimensional inputs. Paper: end-effector poses for the last 2 timesteps §3.2. Code: also the gripper width, 2 timesteps, which adds 2 values umi_maniwav.yaml. The head therefore sees 768 + 20 values (inferred from the shapes).
• Fusion. Paper: one transformer layer, 8 heads, 2048 feed-forward units, dropout 0.0 app. A.2. Code: the same, plus a linear layer that maps the 768-d AST feature to the 1536-d token size before fusion maniwav_obs_encoder.py maniwav_obs_encoder.py. The code sets no norm or activation, so PyTorch's defaults apply.
• Action frame. Paper: not specified. Code: poses for both observations and actions are converted relative to the last observed pose, and the action conversion uses the observation representation setting umi_maniwav_dataset.py. Both settings are relative in the released config, so this has no effect there umi_maniwav.yaml.
• Action normalisation. Not in the paper. Code: position and gripper are range-normalised; rotation is left as is umi_maniwav_dataset.py.
• Masking baseline. Paper: mask audio below 500 Hz, the UR5 control frequency §4.2. Code: the mask_robot option zeroes the first 8 mel bins after normalisation maniwav_obs_encoder.py. The released config sets mask_robot: False train_diffusion_unet_maniwav_workspace.yaml. No code converts the 500 Hz cutoff into a bin count.
• Not in code or paper. The paper gives no learning-rate schedule, EMA parameters, UNet size or diffusion schedule. These values come from the released config only.

Pitfalls

• Flipping table, ResNet column. Counting the 20 rows of the appendix table app. B.2.4 gives 8 successes, which is 40%. The table and the stated headline say 35% app. B.2.4. Use the row-level values, not the headline.
• Pouring percentages do not share a denominator. The text gives 12 rollouts for pouring §4.3. With 12 trials, possible rates are multiples of 8.3%. The table values 90, 30 and 60 cannot come from 12 trials app. B.2.1. Treat the pouring table as author-reported, not as exact counts.
• Audio window depends on the task. The paper says 2 to 3 s §3.2. The released config is 2 s for every task. A re-implementation has to choose.
• Audio spectrogram. The appendix STFT settings do not describe the transformers feature extractor used by the released code. Matching the appendix therefore gives a different input.
• Scaling range depends on the output directory. The config's -16.0 and 10.6 are used unless spec_stats.pkl already exists in the stats directory when the model is built. In that case the data statistics are used instead maniwav_obs_encoder.py. Record which range a run used.
• Noise overlay has no level control. Without gain matching, the overlaid clips keep the level of their files, so the effective augmentation strength depends on those recordings. The ESC-50 files are not in the repository; only the path is set umi_maniwav.yaml.
• Latency offsets are applied only at deployment. The paper describes the compensation at deployment §3.2. The training config sets every latency step to zero, and the deployment environment takes the audio offset as its receive latency umi_env.py. A re-implementation must choose where to apply the offsets.
• Input perturbation. The 0.1 perturbation on the noisy input is not in the paper. It changes the training distribution diffusion_unet_maniwav_policy.py.
• Transformer defaults. PyTorch's default post-norm layout and ReLU activation are used, because the repository sets only width, heads, feed-forward size, batch-first and dropout maniwav_obs_encoder.py.
• Environment count. Section 4.1 says the in-the-wild flipping data came from seven environments §4.1. Appendix B.1 lists six groups of locations app. B.1.

Lineage

• Builds on UMI, the handheld data-collection device and its gripper (arXiv:2402.10329) §3.1.
• Uses the diffusion policy head of Chi et al. [arXiv:2303.04137] §3.2.
• Uses the AST audio encoder [arXiv:2104.01778] §3.2.
• Compared against: the vision-only diffusion policy §4.1, the MLP fusion of "Play it by ear" [arXiv:2205.14850] §4.2, the ResNet audio encoder with CoordConv from See, Hear and Feel [arXiv:2212.03858] §4.1, and AVID audio pretraining from Hearing Touch [arXiv:2405.08576] §4.1.
• Superseded by: not known from this wiki.