STEVE-Audio: Expanding the Goal Conditioning Modalities of Embodied Agents in Minecraft

In plain words

STEVE-1 is an agent that plays Minecraft by following short instructions. Its instructions are CLIP-style vectors, so it can only take the kinds of prompts that CLIP understands: text and pictures. This paper asks whether the agent can also take a sound as its instruction, such as the sound of digging dirt.

The authors do this without retraining the agent. They train a new model that places Minecraft audio and gameplay video in one shared space, then learn a mapping from that space into the goal space the STEVE-1 agent already uses. An audio prompt is turned into a goal, and the existing agent acts on it.

On short item-collection tasks, the audio-prompted agent did better than the picture-prompted agent on four of six tasks, and better than the text-prompted agent on five of six. It did poorly on "place" tasks, where the sound of placing a block is very close to the sound of digging it. These are the authors' own results.

TL;DR

Audio prompts for the Minecraft agent STEVE-1, added without retraining its policy. A new Audio-Video CLIP model and a CVAE prior map audio into STEVE-1's goal space §3. Author-reported: on six item-collection tasks, audio beats the visual prompt on four and the text prompt on five §4.1. Audio is weak on "place" tasks §4.2.1. The code is public but has no license file, and it has several gaps listed under Pitfalls.

Problem

STEVE-1 follows instructions only in the modalities of the CLIP model it was trained with, which for STEVE-1 are text and video §1 §2. The authors argue that audio and video occur together naturally, so audio-video pairs can be collected at internet scale with little effort §1. The goal is to add audio as a prompt modality without retraining the policy §1 §3.

Method

The paper adds audio conditioning in three parts. No equations or algorithms are numbered in the paper, so the loss and the prior are described in prose and cited by section.

1. Audio-Video CLIP model §3.1. A frozen MineCLIP video encoder (attention pooling, 16 frames) §3.1 fig. 2 and a frozen Audio Spectrogram Transformer (AST) §3.1 app. A feed two trainable mapping networks. The AST contributes its 527 output logits §3.1. Each mapping network projects its input to a shared 512-d space §3.1. Training is contrastive, as in CLIP: matching audio-video pairs get high cosine similarity and non-matching pairs in the batch get low similarity §3.1.
2. Training §3.3 tab. 2. Only the two mapping networks are trained, for 100 epochs, with no cosine learning-rate schedule §3.3.
3. Audio prior §3.4 fig. 3. A conditional VAE maps the 512-d audio embedding to a 512-d MineCLIP visual embedding, which is the goal space of the STEVE-1 policy §3.4. The policy itself is not changed §3.4.

Data: YouTube Minecraft gameplay without commentary, with the first two minutes of each video cut §3.2. Audio is resampled to 16 kHz. Video is downscaled to 160 × 256 at 32 fps and 16 frames are sampled per sample. Samples are 1-second windows with 75% overlap app. A tab. 1. The paper uses 25 hours of such data §3.2 plus about two hours of task-specific clips app. A. A 600-hour release is announced §1. The training data is not in the repository (see Pitfalls).

# Audio-Video CLIP contrastive step  [§3.1] [§3.3]
# Input:  audio_logits (B, 527); video_embeds (B, 512), both frozen outputs
# Output: scalar loss

def clip_loss(audio_logits, video_embeds):
    # Trainable mapping networks into the shared 512-d space  [§3.1]
    audio_feats = normalize(audio_mapping(audio_logits))  # (B, 512)
    video_feats = normalize(video_mapping(video_embeds))  # (B, 512)
    # Learned temperature, initialised to 1/0.07  [code: model/clip.py#L164]
    scale = exp(logit_scale)
    logits = scale * video_feats @ audio_feats.T  # (B, B)
    # Matching pairs lie on the diagonal  [§3.1]
    labels = arange(B)
    loss_video = cross_entropy(logits, labels)
    loss_audio = cross_entropy(logits.T, labels)
    return (loss_video + loss_audio) / 2

# Mapping network: input to 512-d  [§3.1] [code: model/clip.py#L48-L84]
# Input: x (B, in_dim), in_dim = 527 (audio) or 512 (video)
# Output: (B, 512)

def mapping_network(x):
    # BatchNorm on the raw input  [code: model/clip.py#L73, #L79]
    x = batch_norm(x)
    # 10 layers, hidden width 1024  [§3.1] [code: configs/avclip.yaml#L5-L7]
    for layer in layers:
        # Equalised learning rate: weights scaled by
        # lr_multiplier / sqrt(fan_in), with lr_multiplier = 0.01
        # [code: model/clip.py#L27-L41] [code: model/clip.py#L55]
        x = leaky_relu(linear(x), slope=0.2)  # every layer activated
    return x

# Audio features for the AST encoder  [app. A]
# Input:  waveform (channels, samples) at its own sample rate
# Output: features (1024, 128)

def audio_features(waveform, sample_rate):
    # 128 log-mel bins, 10 ms frame shift  [code: preprocess.py#L8-L11]
    fbank = kaldi_fbank(waveform, sample_rate, mel_bins=128)
    # Pad or cut to 1024 frames, before normalising
    # [code: preprocess.py#L13-L20]
    fbank = pad_or_cut(fbank, target_frames=1024)
    # Normalise with fixed constants  [code: preprocess.py#L22]
    return (fbank + 4.2677393) / (2 * 4.5689974)

# Audio prior: CVAE from audio to MineCLIP goal  [§3.4]
# Training input: audio (B, 512), visual (B, 512), both precomputed
# Output: scalar loss (the paper does not state the loss)

def prior_loss(audio, visual, beta):
    # Encoder reads the visual target and the audio condition
    # [code: cvae/cvae.py#L57-L59]
    mu, logvar = split(encoder(concat(visual, audio)))
    # Reparameterisation trick  [code: cvae/cvae.py#L61-L64]
    latent = mu + exp(0.5 * logvar) * random_normal()
    # Decoder reads latent and audio condition  [code: cvae/cvae.py#L66-L68]
    recon = decoder(concat(latent, audio))
    # KL summed over batch and dims, MSE averaged
    # [code: train_audio_prior.py#L19-L21]
    kl = -0.5 * sum(1 + logvar - mu**2 - exp(logvar))
    mse = mean((recon - visual)**2)
    return beta * kl + mse

# Inference: wav file to a goal for the STEVE-1 policy  [§3.4] [fig. 3]
# Input:  path to a .wav file
# Output: goal vector (1, 512), MineCLIP space

def audio_goal(path):
    waveform, rate = load_wav(path)  # no resampling in code
    features = audio_features(waveform, rate)
    # Frozen AST logits, then the trained audio mapping network
    # [code: steve_interface/interface_util.py#L105-L109]
    audio_embed = audio_mapping(ast_logits(features))
    # The prior samples a fresh latent on every call
    # [code: cvae/cvae.py#L70-L73]
    # [code: steve_interface/interface_util.py#L111]
    latent = random_normal(256)
    return cvae_decode(latent, audio_embed)

Hyperparameters

Values in the code that the paper does not state are marked as code-only.

Results

Author-reported. Setup: six short item-collection tasks (dirt, wood, seeds, sand, cobblestone, leaves), each run on 10 seeds for 2 minutes, with minimal prompt engineering §3.5. Baselines are the original text-prompted and visual-prompted STEVE-1 §4.1. Model size, hardware and the statistic behind the ratios are not stated. Results are shown only in fig. 4 as bars at the 10th, 50th and 90th percentiles. The text gives ratios, not absolute counts §4.1.

The paper also argues that audio prompts may need less prompt engineering than text or visual prompts. This is a suggestion, and the authors say it needs further study §4.2.2.

Paper vs code

Repository found via the project page [sites.google.com/itbhu.ac.in/steve-audio], which links github.com/nlenzen/steve_multimodal. The owner is an author (Nicholas Lenzen). No other search was needed. The README links the checkpoints on Google Drive. Code cites use paths in that repository.

• Learning rate. Paper and CLI agree on 0.001 tab. 2 train_avclip.py. The default argument of train_model is 0.00025 train_avclip.py, which only applies when called without the CLI.
• Optimiser. Paper Table 2 lists Adam hyperparameters and weight decay tab. 2. The code uses AdamW, which decouples weight decay train_avclip.py.
• Temperature. The paper says the loss follows CLIP §3.1. The code has a trainable logit scale, initialised to 1/0.07 clip.py, and uses it in the logits clip.py.
• Mapping network. The paper says "upscaled StyleGAN 3 mapping network" and gives width and depth §3.1. The code also adds BatchNorm on the input clip.py, a 0.01 equalised learning-rate multiplier clip.py, and LeakyReLU on every layer including the last clip.py.
• Prior size. The paper says the encoder and decoder are "two-layer MLPs" with hidden width 256 and layer normalisation §3.4. The code uses three linear layers per side, with two hidden layers that have LayerNorm and ReLU cvae.py.
• Prior input. The paper says the audio embedding comes from the "audio encoder" of the Audio-Video CLIP model §3.4. At inference the code uses the 512-d output after the trained audio mapping network interface_util.py.
• Prior loss. The paper does not describe the training loss or its conditioning. The code uses KL plus reconstruction MSE train_audio_prior.py.
• Gameplay length. The paper uses 2400 steps (2 min at 20 fps) §3.5. The code default is 3000 steps interface_run_agent.py. FPS is 20 interface_run_agent.py.
• Audio resampling. The paper resamples to 16 kHz app. A. Inference reads the file at its own rate and does not resample interface_util.py.
• Evaluation code. The paper says it follows Baker et al. and Lifshitz et al. §3.5. The evaluator and the agent come from the external STEVE-1 repository interface_util.py interface_run_agent.py.
• Dataset. The paper says the data will be released §1. The repository's datasets/ directory is empty [datasets/README.txt].

Pitfalls

• Training inputs are not in the repository. EmbeddingLoader loads precomputed arrays from .npz files dataloader.py. No script in the repository computes MineCLIP or AST embeddings from video or audio, and the frame sampling and resampling exist only in the paper app. A. Because the encoders run only at inference, the requires_grad lines in the training loop do nothing for the released path train_avclip.py.
• The starting checkpoint is missing. The default --model_path is checkpoints/base_avclip.pth train_avclip.py, and the checkpoints directory is empty [checkpoints/README.txt]. init_model in load.py could build the model from MineCLIP and AST weights, but no script calls it load.py. ASTModel(pretrain=True) builds the network with random weights and does not load AudioSet weights audio_encoder.py. The AST weights therefore come only from an external checkpoint.
• Two optimiser steps per batch. train_batch calls zero_grad, backward and step train_avclip.py. The epoch loop then calls optimizer.step() again on the same gradients train_avclip.py. Each batch therefore gets two AdamW updates. The paper does not mention this, and the authors do not say whether it is intended.
• Import errors (by reading, not run). train_avclip.py uses a relative import from .load import ... train_avclip.py, which fails under python train_avclip.py, the command the README gives. train_audio_prior.py imports dataloading/datasets.py train_audio_prior.py, which imports src.source.preprocess datasets.py. That package is not in the repository.
• Padding before normalisation, not masked. make_features zero-pads the filterbank before normalising preprocess.py. A 1-second clip is about 100 real frames followed by about 924 padded frames, and each padded value becomes (0 + 4.2677393) / 9.138 ≈ 0.467. The AST forward pass has no attention mask, so the padding is attended to audio_encoder.py. If the training embeddings were made with this function (the dataset loader imports it datasets.py), the same padding applies there.
• Sample rate at inference. make_features uses the file's own sample rate preprocess.py, and get_prior_embed does not resample interface_util.py. The paper trains at 16 kHz app. A. At 44.1 kHz, the 128 mel bins span 0 to 22 kHz instead of 0 to 8 kHz. Resample WAV inputs to 16 kHz before use.
• Prior sampling. The prior draws fresh noise on every call cvae.py. The interface calls it once per WAV file and reuses the goal for all 10 seeds interface_run_agent.py. The paper does not say whether the sample or the mean is used.
• Prior loss scale. The KL term is summed over the batch and the latent dims, while the MSE is averaged over elements train_audio_prior.py. The relative weight of the two terms therefore depends on batch size. The paper gives no loss.
• Equalised learning rate. The mapping networks store unscaled weights and multiply by 0.01 / sqrt(fan_in) at each forward pass clip.py. Adam normalises gradient scale, so the weights move by roughly lr × 0.01 / sqrt(fan_in) per step. A plain nn.Linear with the same learning rate would not match this behaviour.
• Train-accuracy sampling. get_random_train_samples draws indices with the test-set length dataloader.py. This affects only the logged train accuracy, but if the train set is larger, only a prefix is used.
• Conditioning scale. The audio conditioning scale 6.0 is a code default interface_run_agent.py. The paper does not state the scale used for its results, so the reported numbers cannot be reproduced exactly.
• Paper's conclusion overstates its results. The conclusion says the audio agent "outperforms" the text and visual versions on item-collection tasks §5. Section 4.1 reports lower counts on seeds and cobblestone against visual, and on cobblestone against text §4.1. The abstract says "comparable" [abstract].
• Dataset size is unclear. The paper uses 25 hours §3.2 and about two more hours of task-specific clips app. A. It does not say whether the two hours are part of the 25. The test-sample count in Table 1 is blank tab. 1.
• Licensing. The repository has no LICENSE file, so its code may not be quoted. Only path citations are allowed. The checkpoints are on Google Drive [README.md#L31], with no license stated. The paper's weights and data are therefore not known to be redistributable.
• Dependencies. ASTModel asserts timm == 0.4.5 audio_encoder.py. The agent and evaluator need the external STEVE-1 repository and MineRL interface_util.py.

Lineage

• Builds on: STEVE-1 [Lifshitz et al., §1, §3]. The paper gives no arXiv id for it. MineCLIP and MineDojo [see /papers/2206.08853] supply the visual video encoder §3.1. The AST encoder [Gong et al., app. A] supplies the audio encoder. The CLIP training scheme is reused for the audio-video model §3.1.
• Compared against: STEVE-1 with its original text and visual prompts §4.1.
• Superseded by: not known.