What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessment

In plain words

Judging a dive is more than one number. A judge has to see which dive was performed (its position, whether the diver does an armstand, how many somersaults and twists) and how well it was carried out. This paper trains one neural network to predict the quality score and, at the same time, to recognise those parts of the dive and to write a short commentary on its good and bad points. The extra targets act as a hint: the network learns features that describe the dive, not only features that fit one number.

The network reads a video of the dive in one of two ways. One version cuts the video into short chunks, learns from each chunk and averages the results. The other squeezes the whole dive into 16 frames and processes them in one pass. Both are trained end to end with a single summed loss.

The authors also built a dataset of 1412 diving videos from competitions, labelled with scores, dive parts and TV commentary. On that data, the chunk-averaging version beats the same network trained on the score alone. These are the authors' own numbers.

TL;DR

• One 3D CNN is trained jointly on AQA score regression, five dive-part classifiers and commentary captioning. The total loss weights are 1, 1 and 0.01 §4 §5 eq. 4.
• Two backbones: C3D-AVG (96 frames as six 16-frame clips, averaged) and MSCADC (16 frames, dilated convolutions) §4.1 §4.2.
• Author-reported: C3D-AVG-MTL reaches 90.44% Spearman versus 89.60% for C3D-AVG trained on the score alone tab. 4 tab. 5.
• Use this card if you need the exact training recipe. The official code is public but has no LICENSE file, and it has test-time and config traps described under Pitfalls.

Problem

Action quality assessment (AQA) asks how well an action was performed, for example a dive, a gymnastics vault or a surgical skill §1. Existing AQA and skill-assessment methods learn features for a single label, the final score, with a regression or ranking loss §1 §2. The authors argue that one score does not capture a complicated action, because the score depends both on what was done (difficulty) and on how it was done (execution quality) §1.

Diving was the test domain. The two earlier diving datasets, MIT Dive and UNLV Dive, contain 159 and 370 samples, all from one event with male 10 m platform divers, and carry only the AQA score tab. 1 §3. The new MTL-AQA dataset has 1412 samples from 16 events, with 3 m springboard and 10 m platform, male and female divers, synchronised pairs, and labels for score, dive class and commentary tab. 1 §3.

The 3D CNN has a practical limit. Spatio-temporal 3D CNNs need a lot of memory, which restricts them to short clips, yet an AQA decision can depend on any part of the whole action §4.

Method

Data. Each sample is a televised dive with three labels. The AQA score is the judges' total, taken after the dive is finished §3. The dive is factorised into five sub-labels: position (free, tuck, pike), armstand (no, yes), rotation type (inward, reverse, backward, forward), number of somersaults and number of twists tab. 2 §3. The commentary, recorded from former athletes, is converted to text with a speech-to-text service §3. Captions are capped at 100 words and the vocabulary has 5779 words §5.

Multitask objective. The AQA score is a regression target. The authors use an L2 term plus an L1 term, because L1 added to L2 gave better results in initial tests §4 eq. 1. Each dive sub-label is a cross-entropy classification target, and the five losses are summed §4 eq. 2. Captioning uses negative log-likelihood over the caption tokens §4 eq. 3. The total loss is a weighted sum with α, β and γ §4 eq. 4.

C3D-AVG (main model). The backbone is C3D up to pool-5 §4.1. The 96-frame video is split into 16-frame clips, each clip goes through the backbone, and the clip features are averaged element-wise. The authors motivate the averaging as a linear aggregation that encodes points gained along the action §4.1. The score and dive-class heads read the averaged vector. The caption branch reads the individual clip features, which the authors found worked better than the average §4.1. The architecture figure is fig. 2.

MSCADC. This variant follows Nibali et al.'s dilated multiscale design §4.2. The shared body is a C3D-style stack with batch normalisation, dropout at the end, and dilated convolutions in the last group tab. 3. The video is reduced to 16 frames, which the authors describe as key action snapshots §4.2. Each task has its own head: a 1×1×1 reduction to 12 channels, a context net that aggregates features at several dilation rates, a pooling step, and a final 3D convolution. The AQA head ends in average pooling over a 2×11×11 window, and the caption head feeds a GRU tab. 3.

Training. Both backbones are trained end to end, not frozen, with Adam at learning rate 1e-4 for 100 epochs and batch size 3 §5. Augmentation is centre cropping, temporal jitter and random horizontal flips §5.

Pseudocode for one C3D-AVG training step, reconstructed from the paper and the code:

# One joint training step, C3D-AVG variant  [§4.1] [eq. 4]
# Input:  video (B, 3, 96, 112, 112); score (B,); dive (B, 5); caption (B, L)
# Output: scalar loss

def training_step(video, score, dive_labels, caption):
    # Run C3D on six 16-frame clips of the video  [§4.1]
    # [code: MTL-AQA_code_release/train_test_C3DAVG.py#L70-L75]
    clip_feats = backbone(video)               # (B, 6, 8192)
    # Average the clip features into one video vector  [§4.1]
    video_feats = mean(clip_feats, axis=1)     # (B, 8192)
    # The code adds an fc6 layer (8192 -> 4096, ReLU) that the paper does
    # not describe  [code: MTL-AQA_code_release/models/C3DAVG/my_fc6.py#L99]
    hidden = relu(fc6(video_feats))            # (B, 4096)
    # AQA loss: L2 plus L1 on the score divided by 17  [§4] [eq. 1]
    # [code: MTL-AQA_code_release/train_test_C3DAVG.py#L87-L88]
    score_pred = score_head(hidden)            # (B, 1)
    loss_aqa = mse(score_pred, score / 17) + l1(score_pred, score / 17)
    # Five dive sub-heads, cross-entropy each, summed  [§4] [eq. 2]
    loss_cls = sum(cross_entropy(head(hidden), dive_labels[:, k])
                   for k, head in enumerate(dive_heads))
    # Captioning reads the per-clip features, not the fc6 output  [§4.1]
    # [code: MTL-AQA_code_release/train_test_C3DAVG.py#L85]
    loss_cap = masked_nll(caption_decoder(clip_feats, caption), caption)
    # Weights alpha = 1, beta = 1, gamma = 0.01  [§5] [eq. 4]
    return 1.0 * loss_aqa + 1.0 * loss_cls + 0.01 * loss_cap

Hyperparameters

Values from the paper unless the "Where" column says code. "not stated" means neither the paper nor the code gives a value.

Results

Author-reported. Dataset: MTL-AQA, 1412 diving samples §3 tab. 1. Metric: Spearman rank correlation for the score, accuracy for dive parts, and BLEU, METEOR, ROUGE and CIDEr for commentary §5. Hardware: not stated. Each result is one run on the authors' split; no variance is reported.

Main comparison, MTL against single-task learning (STL) tab. 4:

Comparison with existing AQA methods on MTL-AQA tab. 5:

On the older UNLV-Dive split, with segment-specific baselines tab. 5: S3D 86.00, Li et al. 80.09, MSCADC-STL 79.79, C3D-AVG-STL 83.83, MSCADC-MTL 80.60, C3D-AVG-MTL 88.08. The text notes the S3D method needs segment annotations that MTL-AQA lacks, so the authors trained on UNLV-Dive for that comparison §5.

Dive-part accuracy tab. 6, with the classification-only baseline of Nibali et al.:

Commentary metrics tab. 6: C3D-AVG BLEU-1 0.26, BLEU-4 0.02, METEOR 0.11, ROUGE 0.14, CIDEr 0.06. MSCADC BLEU-1 0.25, BLEU-4 0.01, METEOR 0.11, ROUGE 0.13, CIDEr 0.05. The paper has no captioning baseline §5.

Training-set size tab. 8, Spearman:

Linear-probe test: linear regressors fitted on the activations of each convolutional layer, c1–c5 tab. 9 §5.2. For diving: Baseline-1 (C3D on UCF-101) 71.01 / 71.39 / 73.13 / 76.34 / 73.69; Baseline-2 (MTL-AQA model trained on dive-class recognition only) 72.43 / 70.15 / 70.35 / 57.20 / 37.63; C3D-AVG-MTL 74.26 / 77.95 / 82.78 / 86.18 / 85.75. On gymnastic vault, a class not in training, the within-dataset and out-of-dataset results are in tab. 10. The authors report that the out-of-dataset drop is more graceful for Baseline-2 and C3D-AVG-MTL than for Baseline-1 §5.2.

Qualitative examples with ground-truth and predicted labels are in tab. 7; they illustrate, and are not a benchmark.

Paper vs code

Official repository: ParitoshParmar/MTL-AQA, linked from the paper's abstract, at commit bf7dcb2. It contains MTL-AQA_code_release/ and MTL-AQA_dataset_release/. The repository has no LICENSE file, so the code is cited, not copied.

• Captioning dropout. Paper: 0.2 in the encoder and decoder §5. Code: caption_lstm_dropout = 0.5 opts.py, passed to the caption model in both scripts train_test_C3DAVG.py train_test_MSCADC.py. The caption module's own default is 0.2 S2VTModel.py, but the scripts override it.
• Clip length. Paper: each dive is "temporally normalized to a length of 96 frames" §5. Code: each sample loads a fixed 103-frame window ending at the annotated end frame, with no resampling opts.py dataloader_C3DAVG.py. The clip loop uses np.arange(0, frames − 17, 16), which gives six clips covering frames 0–95 and leaves frames 96–102 unused train_test_C3DAVG.py.
• Pretraining. Paper: backbones pretrained on UCF101 §5. Code README: "Sports-1M pretrained C3D weights" ReadMe.txt. The C3D-AVG script loads c3d.pickle train_test_C3DAVG.py. The MSCADC script loads C3D_small_PyTorch_Trained_12.pth train_test_MSCADC.py, which ships only as split 7z archive parts whose origin is not documented. The paper does not say which weights were used.
• fc6 layer. The paper describes averaging followed by task heads §4.1. The code adds a 8192 → 4096 ReLU layer, randomly initialised, before the score and classification heads my_fc6.py train_test_C3DAVG.py.
• Dilation in MSCADC. The text says pooling is removed from "the last two convolutional groups" and dilation 2 is used §4.2. In the code, conv4 has neither pooling nor dilation, and only conv5 is dilated body.py. The architecture table matches the code tab. 3.
• MSCADC test-time resize is discarded. In the training path, load_image_train resizes to the configured size and returns the resized image dataloader_MSCADC.py. In the test path, load_image calls image.resize(size, interpolator) without assigning the result dataloader_MSCADC.py. Test frames are therefore centre-cropped from their native resolution, while training frames are first resized to 640×360. The paper describes the resize for both. Native frame size is not stated in the paper, and the frame extraction script does not rescale frame_extractor.sh. If native frames are not 640×360, train and test inputs differ in scale. The C3D-AVG loader assigns the resized image in both paths dataloader_C3DAVG.py.
• Caption head. The paper's table shows the caption head as a shared context net, a pooling step and a convolution, then GRUs tab. 3. The code's caption head adds a 3×3×3 convolution and a stride-1 max-pool, then flattens to 1200 features per step head_captions.py head_captions.py.
• MSCADC frames. The paper says the action is downsampled to 16 frames §4.2. The code samples 16 frames at a stride of 6.5 over a 103-frame window dataloader_MSCADC.py.
• Active config. The shipped opts.py is set to MSCADC: 3×180×180 input, resize to 640×360, caption dim 1200 opts.py opts.py opts.py. The C3D-AVG values are commented out on the same lines. Running train_test_C3DAVG.py as shipped uses MSCADC sizes.
• Unused settings. caption_lstm_lr is defined but never read opts.py; the optimizer uses base_learning_rate or a hard-coded 1e-4. temporal_stride is defined and not read opts.py.
• Captioning metrics. The paper reports BLEU, METEOR, ROUGE and CIDEr §5. The release has no metric code; the test loop prints only dive accuracies and Spearman train_test_C3DAVG.py.
• Loss sign. Eq. 1 as printed has a leading minus sign over both terms eq. 1. The code adds MSE and L1 as positive losses and minimises them train_test_C3DAVG.py. This is the intended reading.
• Score scale. Labels are divided by 17 before the loss dataloader_C3DAVG.py. The paper does not mention this. The MSCADC script multiplies the error by 17 when printing MSE train_test_MSCADC.py; the C3D-AVG script prints no MSE.
• Checkpoint choice. The test split is scored every epoch and checkpoints are saved every epoch train_test_C3DAVG.py. The paper does not say which epoch is reported.

Pitfalls

• Test-time resize. See the MSCADC resize bug above. Fix it before comparing MSCADC numbers.
• Random resampling filter. Both loaders draw one of four PIL filters (nearest, bilinear, bicubic, Lanczos) at random for each frame dataloader_C3DAVG.py. This happens in test as well as training, so test inputs vary with the random state. The paper does not mention it.
• Config defaults. Running either script without editing opts.py uses MSCADC sizes. Several values are commented alternatives on the same line opts.py opts.py.
• Caption truncation. Captions longer than 100 tokens are cut, then the last remaining token is overwritten with <eos> dataloader_C3DAVG.py dataloader_MSCADC.py. The cut word is lost, not just the tail.
• Caption loss scale. The caption loss is a masked NLL summed over tokens and divided by batch size, not token count utils_1.py. With γ = 0.01, the caption term is small in both scripts train_test_C3DAVG.py.
• Evaluation on the test split. The authors evaluate on the test split at every epoch and report no separate validation split train_test_C3DAVG.py. Treat the reported numbers as optimistic unless you train with a validation split.
• Single split, single seed, no variance. Results come from one split (*_split_0.pkl) and seed 0 opts.py. Tables give no spread tab. 4 tab. 5.
• Memory and batch size. The paper says 3D CNNs need large memory §4 and uses batch size 3 §5. It does not say how much memory a run needs; hardware is not stated.
• Data licence and source. Videos are linked on YouTube through a spreadsheet Readme.txt. No licence for the dataset is stated in the repository or the paper. Do not republish frames or annotations without checking.
• Generalisation claims. The 140-sample MTL gain is 4.0 points over STL tab. 8; with one split and no variance, treat it as indicative.
• Vault transfer. The gymnastic-vault experiment uses 125 train and 56 test samples §5.2. Its within-dataset rows use train and test events that overlap, and its out-of-dataset rows use non-overlapping events; the authors call the latter harder tab. 10 §5.2.

Lineage

• Builds on: C3D §4.1 for the backbone, and the authors' earlier AQA work on UNLV-Dive with C3D features §2 §3.
• Architecture from: MSCADC, a dilated multiscale context design §4.2, and Nibali et al.'s dive classifier §3 §4.2.
• Compared against: Pose + DCT and C3D-SVR/C3D-LSTM tab. 5, S3D tab. 5, and Li et al. tab. 5.
• Superseded by: not stated in this wiki.