Audio-Visual Scene-Aware Dialog

In plain words

Picture a short video of someone doing everyday things indoors, and a chat about it. One person asks a question such as what happens next, and another answers using what they saw and heard. The paper builds a benchmark for this: more than 11,000 such videos, each with a ten-round question and answer chat and a closing summary written by the questioner.

The baseline is deliberately simple. It turns the video, the audio and the earlier chat into fixed-size vectors, joins them, and then picks the best reply from a list of 100 candidate answers instead of writing one. The list mixes the correct answer with plausible wrong ones, so the model has to use the evidence.

The main result is that the chat history matters most, and adding the video helps further. Adding the sound does not improve the main score on top of the video. The authors also report that shuffling the order of the questions and answers lowers the scores, which suggests the order carries information.

TL;DR

• AVSD is a benchmark of 11,816 dialogs about Charades videos, with 10 question-answer rounds each, split 7,985 / 1,863 / 1,968 for train / validation / test §3 §5.
• The task is to rank 100 candidate answers for a question, given the video, its audio, a script and the earlier rounds §4.
• The baseline is a late-fusion model with a 4096-d I3D video vector and a 4096-d AENet audio vector §4 tab. 3.
• Best reported test MRR is 53.41 (Q+DH+V) against 50.40 for dialog history alone (Q+DH). Adding audio gives 53.03 MRR, so it does not beat video alone on MRR tab. 3 (author-reported).
• The paper does not state the training hyperparameters. The official code uses Adam, learning rate 1e-3 and 20 epochs, and differs from the paper in how it builds candidates train.py makejson_with_options.py.

Problem

Earlier visual dialog work asks questions about a still image. The authors argue that a scene changes over time and has sound, so a system needs the video, its audio and the dialog history to answer follow-up questions such as "what does he do after that?" §1. Existing video question answering sets use TV or film clips, or multiple-choice answers; AVSD uses crowd-sourced clips and free-form answers §3 tab. 1.

Assumptions: answers are scored by ranking a fixed set of 100 candidates, not by generation, which the authors chose because discriminative ranking was stronger than generation in visual dialog §4 Decoder Model. Scores are retrieval metrics, not BLEU or similar, because those metrics correlate poorly with human judgment §5 Evaluation Metrics.

Method

Dataset (AVSD v1.0)

• Videos: 11,816 Charades clips of everyday indoor activity, about 30 seconds each, with at least two actions per video §3.1 fig. 2.
• Each dialog starts from the Charades ground-truth script, a short description of the video §4 Input Representations.
• Two crowd workers chat in real time. The Answerer sees the video and the script and answers. The Questioner sees only three frames (beginning, middle and end) and asks questions. Workers were told to "investigate what is happening" §3.1 fig. 4.
• A dialog has 10 question-answer pairs. Then the Questioner writes a summary of the video §3.1 Downstream Task.
• Worker rules: at least 95% task acceptance, located in North America, at least 500 completed tasks, at most 200 tasks per worker. In total 1,553 workers took part §3.1 Worker Qualifications.
• The dataset has 118,160 question-answer pairs §3.2. Questions average 7.9 words and answers 9.4 words, compared with 5.1 and 2.9 in VisDial §3.2.
• Audio is asked about in 57% of conversations and "what happened next" questions appear in more than 70% §3.2. In 55.2% of questions, the text contains co-reference words such as "her", "they" or "it" §6.
• Summary quality: against three video-watcher summaries, a watcher summary scores CIDEr 0.429 and a Questioner summary 0.297 tab. 2. The authors call the two numbers close §3.2.

Model

The model takes the video, its audio, the dialog history (script plus the first t-1 question-answer pairs) and the t-th question §4. It is a discriminative late-fusion network:

1. Encode the dialog history with an LSTM. The history is the script followed by each earlier question-answer pair §4 Input Representations.
2. Encode the question with an LSTM, trained together with the model §4 Input Representations.
3. Encode the video as one 4096-d vector from I3D, using 40 frames sampled evenly in time §4 Implementation Details.
4. Encode the audio as one 4096-d vector from AENet, after resampling the track to 16 kHz, 16-bit, mono §4 Implementation Details.
5. Concatenate the four vectors and apply one fully connected layer with a tanh activation to get a joint embedding §4 Encoder Network. The equations for this step are unnumbered in the source.
6. Embed each candidate answer with an LSTM and score it by the inner product with the joint embedding eq. 1.
7. Apply a softmax over the 100 scores. Train by maximising the log-likelihood of the true answer. At test time, rank the candidates by probability §4 Decoder Model eq. 1.

For ablations, the network is trained without the removed input and the fusion layer keeps its output size §4 Encoder Network §6.

The middle-frame variant encodes one frame with a VGG-16 network pretrained on ImageNet, to remove temporal information §4 Input Representations.

# Score the 100 candidate answers for one question  [§4] [eq. 1]
# Inputs:  question tokens, history tokens (script and earlier QA),
#          video (4096,), audio (4096,), candidates (100 token lists)
# Output:  scores (100,)

def score_candidates(question, history, video, audio, candidates):
    # Encode the question with an LSTM  [§4 Input Representations]
    question_state = question_lstm(embed(question))         # (d_h,)
    # Encode the script and all earlier QA pairs  [§4]
    history_state = history_lstm(embed(history))            # (d_h,)
    # Video and audio are already fixed-size vectors  [§4]
    # Concatenate the four inputs and fuse them  [§4 Encoder Network]
    fused_input = concat(history_state, question_state, video, audio)
    joint = tanh(fusion_linear(fused_input))                # (d_h,)
    scores = []
    for candidate in candidates:
        # Encode each candidate answer with an LSTM  [§4 Decoder Model]
        answer_state = answer_lstm(embed(candidate))        # (d_h,)
        # Score is the inner product with the joint embedding  [eq. 1]
        scores.append(dot(answer_state, joint))
    return scores

# Loss and rank metric  [§4 Decoder Model] [§5 Evaluation Metrics]
# Inputs: scores (100,), gt_index (position of the true answer)

def training_loss(scores, gt_index):
    # Softmax over the 100 candidates  [§4 Decoder Model]
    probs = softmax(scores)                                 # (100,)
    # Negative log-likelihood of the true answer  [§4 Decoder Model]
    return -log(probs[gt_index])

def ground_truth_rank(scores, gt_index):
    # Rank 1 means the true answer has the highest score  [§5]
    higher = count(s > scores[gt_index] for s in scores)
    return 1 + higher

Candidate answers (evaluation set construction)

Each test question is ranked against 100 candidates, built from four groups §4 Selecting Candidate Answers:

• 50 hard negatives: the true answers to the 50 training questions most similar to the query. Similarity uses Euclidean distance between question vectors. A question vector is the concatenation of the first three word vectors and the average of the remaining word vectors §4 Selecting Candidate Answers.
• 30 popular answers: the 30 most frequent answers in the dataset §4 Selecting Candidate Answers.
• 19 answers to random questions §4 Selecting Candidate Answers.
• 1 true answer.

The paper does not say how duplicates are handled or what is drawn from which split. The code does not follow these counts; see Paper vs code.

# Build 100 candidates for one question  [§4 Selecting Candidate Answers]
# Inputs: the question, its true answer, the pool of all answers

def build_candidates(question, true_answer):
    # 50 hard negatives: answers of the 50 most similar questions  [§4]
    similar = nearest_questions(question, k=50)  # Euclidean distance
    hard = [answer_of(q) for q in similar]
    # 30 most popular answers in the dataset  [§4]
    popular = most_frequent_answers(k=30)
    # 19 answers to random questions  [§4]
    random_answers = sample_answers(k=19)
    # One candidate is the true answer  [§4]
    return hard + popular + random_answers + [true_answer]

Training and implementation details

The paper gives the feature sizes, the candidate design and the dataset split. It does not give the optimizer, learning rate, batch size, epochs, embedding size, LSTM size, dropout or initialisation. Its supplementary section on implementation training details is a TODO in the source and is not in the compiled paper. The values below come from the official code and are not confirmed by the paper.

Hyperparameters

Results

Author-reported. Setup: AVSD v1.0 test split of 1,968 dialogs, 10 rounds each, ranking 100 candidates per question. Metrics are MRR, recall at 1, 5 and 10, and mean rank §5 tab. 3. The paper does not report hardware, seeds, or variance. Units are not stated; the code prints recall as fractions, so the table values look like percentages (see Pitfalls).

Table 3 (test split, model ablations) tab. 3

Key abbreviations: Q = question, DH = dialog history (script plus earlier rounds), C = caption, I = middle frame, V = video, A = audio §5 §6.

Derived from the table: adding dialog history to the question raises MRR from 36.12 to 50.40. Adding video to Q+DH raises MRR from 50.40 to 53.41. Adding audio to Q+DH+V lowers MRR by 0.38 and raises R@10 by 0.13 and lowers mean rank by 0.02.

Table 4 (shuffled question-answer order) tab. 4

The authors state the model was tested on shuffled dialogs, not retrained on them. A "retrained and shuffled" block is commented out in the source and is not in the compiled paper tab. 4. The authors report an average difference of about 1.87 in recall and about 0.33 in mean rank between original and shuffled order §6.

Table 5 (mean rank on question subsets) tab. 5

Audio-related questions are selected by the words talk, hear, sound, audio, music and noise. Temporal questions by after, before, beginning, then, end and start §6.

Figure 7 shows the answer probabilities for two examples, with the true answer highlighted fig. 7. Appendix B has more qualitative examples app. B.

Paper vs code

The official repository is batra-mlp-lab/avsd at commit e795d578a83c98463dbef18152c63bee0b0a45e2. The README names the paper and its authors README.md. The owner, batra-mlp-lab, is the organisation that also hosts the VisDial starter code the README builds on README.md; we treat it as an author's lab. The repository has no LICENSE file. The README says "BSD" README.md, so the license is not verifiable from the repository. This card cites the code and quotes none of it.

How the code was found: the arXiv metadata and the paper text link no code. The project page given in the paper (video-dialog.com) no longer shows the authors' work; it now shows unrelated content, so it was not used. A GitHub repository search for "avsd dialog" found batra-mlp-lab/avsd [GitHub search, 2026-11-06].

Differences between the paper and the code:

• Candidate composition. The paper uses 50 hard negatives, 30 popular answers, 19 random answers and the true answer §4 Selecting Candidate Answers. The code's defaults are 15 nearest answers and 50 popular answers makejson_with_options.py. Nearest answers that are also popular are dropped makejson_with_options.py. The rest of the 100 is filled with random non-popular answers makejson_with_options.py makejson_with_options.py. The composition therefore differs from the paper's.
• Hard negatives. The paper uses the true answers to the 50 most similar questions §4. The code walks the sorted distances and keeps up to 15 distinct answers, and it searches only a random subset of 4,000 questions for each query makejson_with_options.py makejson_with_options.py.
• Question vector. The paper says the remaining words are averaged §4 Selecting Candidate Answers. The code sums them into the last 300 dimensions svqa_utils.py. The word vectors come from the spaCy model en_vectors_web_lg makejson_with_options.py. The paper says GloVe.
• Candidate pool. Popularity counts, the random distractors and the nearest-question search all use questions and answers from train, validation and test together makejson_with_options.py makejson_with_options.py. The paper does not say which split the pool comes from.
• Feature normalisation. Audio features are L2-normalised dataloader.py. Video features are not dataloader.py. The paper says nothing about either.
• Dropout. Dropout 0.5 is applied to the LSTM stacks and to the fused vector lf.py lf.py. The paper gives no dropout.
• Decoder LSTM depth. The answer LSTM has one layer, while the encoder LSTMs have two disc.py lf.py.
• Word embeddings. The model's word embeddings are randomly initialised with no pretrained vectors lf.py. The retrieval vectors are a separate spaCy lookup.
• Input choices. The paper's Q+C ablation has no matching input option. The only option that includes the caption also includes the image, so question plus caption alone cannot be run with the default script train.py.
• Answer Prior. The Answer Prior baseline in Table 3 has no implementation in the repository. A search for "prior" in the Python code found nothing.
• External features. The I3D, AENet and VGG features are read from HDF5 files that the code does not produce dataloader.py README.md. The 40-frame and 16 kHz settings cannot be checked here.
• Checkpoint. The evaluation script defaults to checkpoints/13-Jun-2019-16:22:48/model_epoch_14.pth evaluate.py. The paper does not say which checkpoint produced Table 3.
• Consistent. The metric definitions match: rank 1 is the top score; recall at k, mean rank and mean reciprocal rank are computed as in the paper eval_utils.py eval_utils.py. The loss is cross-entropy over the 100 raw scores, which equals softmax then negative log-likelihood train.py train.py disc.py.

Pitfalls

• Units. Table values such as R@1 36.22 look like percentages. The code prints recall as a fraction in [0, 1] eval_utils.py. The paper does not state units. Compare like with like.
• No variance. Every number is a single run. The differences between Q+DH+V and Q+DH+V+A (0.38 MRR) are small, and no seeds or intervals are given. The code fixes seed 1234 train.py.
• Candidate sets depend on sampling. The code's candidates come from a script with fixed seeds makejson_with_options.py. The paper gives the recipe but not the sampled candidates, and the released candidate files are not in this repository, so the exact test candidates cannot be checked.
• Split mixing. Distractors and popularity are drawn from all three splits, as described above. If you build your own candidates, use the training split only, or state that you did not.
• Checkpoint choice. The code logs validation loss every 100 iterations but does not pick a checkpoint by it train.py. Save every 2 epochs train.py. The paper does not say which epoch it reports.
• History length. History is capped at 400 tokens dataloader.py. The script is cut to 40 tokens and each earlier round adds up to 41 (1 end token, 20 question, 20 answer) dataloader.py dataloader.py. By the tenth round the history can reach 40 + 9 × 41 = 409 tokens, above the cap. The code has no guard for this. This is a derived bound, not an observed failure.
• Casing. Tokenisation uses NLTK word tokenisation without lowercasing prepro.py prepro.py prepro.py. "The" and "the" are separate vocabulary entries.
• Vocabulary. Word counts come from all splits, and words under 5 occurrences map to UNK prepro.py. Indices start at 1, with 0 for padding prepro.py.
• Ties. Ranks come from sorting scores in descending order. Ties are broken by the sort, not specified in the paper eval_utils.py.
• Answer duplicates. The paper does not say how duplicate answer strings are removed from the candidate list. The code uses exact string matching makejson_with_options.py.
• Missing training details. The paper has no training section in the compiled version (the source file is a TODO and is not included). Every row marked "code" in the Hyperparameters table comes from the repository, not from the paper.
• Dataset claims. The authors' claims about dialog order and about summaries are based on their own analyses §3.2 §6. They are not independently checked here.
• Learning-rate floor. The schedule stops decaying at 5e-5 and is never raised again train.py. Derived from the decay factor, the floor is reached after about 12,440 iterations, which is about 19 epochs at 666 iterations per epoch (7,985 training dialogs at batch size 12) train.py.

Lineage

• Builds on: visual dialog with a discriminative late-fusion encoder (Das et al., 2017, and Jain et al., 2018, as cited in §4 Encoder Network and §4 Decoder Model). Also Charades videos (Sigurdsson et al., 2016) §3.1. I3D video features (Carreira and Zisserman, 2017) and AENet audio features (Takahashi et al., 2018) are used as pretrained encoders §4 Input Representations.
• Compared against: question-only and dialog-history-only baselines and the Answer Prior baseline, all inside this paper tab. 3.
• Superseded by: none in this wiki yet.