TransFER: Learning Relation-aware Facial Expression Representations with Transformers

In plain words

Reading a face's expression, such as happy, sad or angry, is harder than it looks. Two emotions can look alike around the mouth, and the same emotion can look quite different from one person to the next. Older methods often looked at fixed spots on the face, or at a single attention spot, which can miss the useful clues when the face is turned or partly hidden.

TransFER takes a different route. It lets the model look at several regions of the face at once, and then uses a Transformer, a neural network design built on attention in which every part of the input can weigh every other part, to work out how those regions relate to each other. To stop the model from always leaning on the same obvious region, two dropout-style tricks are used only during training. One switches off one region's attention map at random. The other switches off one attention block inside the Transformer at random. The model then has to find useful clues in the regions that are left.

The authors report better accuracy than earlier methods on three public expression datasets. Those numbers are their own, measured under their setup.

TL;DR

TransFER is a facial expression recognition model made of a CNN stem, several learned attention maps over face regions, and a Vision Transformer over the resulting tokens §3. Two random dropping modules keep it from relying on one region or one self-attention module: MAD on the region maps §3.2 and MSAD on the Transformer's self-attention §3.3. The authors report 90.91% on RAF-DB, 66.23% mean class accuracy on the AffectNet validation set and 90.83% on FERPlus tab. 5 tab. 6. No official code was found, so the card is based on the paper alone. Transformer width and MLP size are not stated.

Problem

Expression classes differ only slightly in some regions, and the same class can look very different across people §1. Methods that crop fixed facial parts depend on landmark detection, which can fail under strong illumination, large pose or heavy occlusion §1. Attention-based methods can put their attention on similar parts and neglect other discriminative ones §1. The paper argues that diverse local patches are needed, and that the relations between them should be modelled over the whole image §1. The authors state that they are the first to apply Transformers to FER and to study relation-aware local patches for it §1 §2.2; this is their own claim.

Method

The model has four parts (overview in §3, fig. 2):

1. Stem CNN. An IR-50 network, pretrained on MS-Celeb-1M, is used with only its first three stages. Inputs are 112 × 112 faces §4.2. The stage-3 output is 14 × 14 §4.3.
2. Local CNNs. B local branches each apply a LANet block to the stem features §3.1. A LANet block is a 1 × 1 convolution to c/r maps, a ReLU, a second 1 × 1 convolution to one map and a sigmoid, giving an attention map M_i §3.1. The reduction ratio r is not stated.
3. Multi-Attention Dropping (MAD). With probability p1, one of the B maps, picked uniformly, is set to zero during training §3.2 §3.1. The zeroed branch is not passed on, so it gets no gradient through that path (the paper calls this a stop-gradient) §3.2.
4. Aggregation. The B maps are combined by element-wise maximum eq. 1 §3.1. The result multiplies the stem features element-wise §3.1.
5. Projection and tokens. A 1 × 1 convolution maps the features to c2 channels (c2 is not stated). The h × w grid is flattened into a sequence, a learnable class token is prepended, and learnable 1-D position embeddings are added §3.3.
6. Transformer encoder. M = 8 identical blocks, each with 8 self-attention modules whose outputs are concatenated and projected, followed by a two-layer GELU MLP, with skip connections §3.3 §4.2. Self-attention is defined by eq. 2 eq. 3 eq. 4. Layer normalisation is not mentioned; the block layout is said to follow ViT §3.3.
7. Multi-head Self-Attention Dropping (MSAD). In every block and for every sample, with probability p2 one of the k = 8 self-attention modules, picked uniformly, is set to zero during training §3.3. The paper describes MAD as "injected behind every Multi-head Self Attention module" §3.3 fig. 2.
8. Head and loss. A single MLP layer gives class scores §3.3. The loss is plain cross-entropy §4.2 §4.4.

Pseudocode of the training forward pass, written from the text. It is not from the authors' code.

# TransFER training forward pass  [§3] [eq. 1-4]
# Input:  images (B_img, 3, 112, 112); labels (B_img,)
# Output: scalar cross-entropy loss
def transfer_loss(images, labels, training):
    # Stem: IR-50, stages 1-3, pretrained  [§4.2] [§4.3]
    stem = stem_cnn(images)
    # (B_img, c, 14, 14)

    # One attention map per local branch, sigmoid output  [§3.1]
    maps = []
    for each branch b in 1..B:
        maps.append(lanet_branch_b(stem))
        # (B_img, 1, 14, 14)

    # MAD: in training, with prob. p1, zero one map picked uniformly
    # [§3.2] [§3.1]
    if training and random() < p1:
        dropped = uniform_index(B)
        maps[dropped] = zeros_like(maps[dropped])

    # Element-wise maximum over the maps  [eq. 1] [§3.1]
    attention = elementwise_max(maps)
    # (B_img, 1, 14, 14)

    # Re-weight stem features by the aggregated map  [§3.1]
    local = stem * attention
    # (B_img, c, 14, 14)

    # 1x1 projection to c2 channels, then flatten to tokens  [§3.3]
    proj = conv1x1(local)
    # (B_img, c2, 14, 14)
    tokens = flatten_to_sequence(proj)
    # (B_img, 196, c2)
    tokens = prepend_class_token(tokens)
    # (B_img, 197, c2)
    tokens = tokens + position_embedding
    # learnable 1-D position embedding  [§3.3]

    # Transformer encoder, M = 8 blocks  [§3.3] [§4.2]
    for each block m in 1..M:
        # Self-attention heads, eq. 2-4  [§3.3]
        heads = []
        for each head j in 1..k:
            # Eq. 2: queries, keys and values  [eq. 2]
            queries = tokens @ w_q[j]
            keys = tokens @ w_k[j]
            values = tokens @ w_v[j]
            # Eq. 3: attention weights  [eq. 3]
            weights = softmax(queries @ keys.T / sqrt(d_k))
            # Eq. 4: weighted sum of values  [eq. 4]
            heads.append(weights @ values)

        # MSAD: in training, per sample and per block, with prob. p2,
        # zero one head picked uniformly  [§3.3]
        if training and random() < p2:
            dropped = uniform_index(k)
            heads[dropped] = zeros_like(heads[dropped])

        # Concatenate, project, residual, then GELU MLP  [§3.3]
        tokens = tokens + project(concat(heads))
        tokens = tokens + mlp_gelu(tokens)

    # Head reads the class token; input choice is ViT-style  [§3.3]
    logits = mlp_head(tokens[:, 0])
    # (B_img, 7 or 8)

    # Plain cross-entropy, no other loss terms  [§4.2] [§4.4]
    return cross_entropy(logits, labels)

Key equations (reproduced from the paper):

M_{out}(x, y) = \max\{M_1(x, y), M_2(x, y), \ldots, M_B(x, y)\} eq. 1

[q, k, v] = x\,[w_q, w_k, w_v] eq. 2

A = \mathrm{Softmax}\left(\frac{q\,k^{T}}{\sqrt{d_k}}\right) eq. 3

O = A\,v eq. 4

Hyperparameters

Results

Author-reported. Setup: three public FER datasets (RAF-DB single-label, FERPlus, AffectNet) with the splits in §4.1. Metric: overall test accuracy for RAF-DB and FERPlus, and mean class accuracy on the validation set for AffectNet §4.1. Ablation on RAF-DB and AffectNet tab. 1:

Stage of the stem on RAF-DB tab. 2: stage 2 84.94, stage 3 90.91, stage 4 90.32.

Drop rates on RAF-DB tab. 3: with p2 = 0.3, p1 = 0.4 / 0.5 / 0.6 / 0.7 / 0.8 gives 89.28 / 89.24 / 90.91 / 89.89 / 89.89. With p1 = 0.6, p2 = 0.1 / 0.2 / 0.3 / 0.4 / 0.5 gives 89.80 / 89.83 / 90.91 / 89.93 / 89.24.

Comparison with other regularisers, same model with MAD replaced tab. 4: RAF-DB, MAD 90.91, Dropout 90.35, DropBlock 90.25, Spatial Dropout 89.99. AffectNet, MAD 66.23, Dropout 66.06, DropBlock 66.03, Spatial Dropout 65.54.

State of the art, RAF-DB and AffectNet tab. 5:

State of the art, FERPlus tab. 6: PLD 85.10, RAN 88.55, SeNet50 88.80, RAN-VGG16 89.16, SCN 89.35, KTN 90.49, TransFER (this paper) 90.83.

Attention visualisations of the four training variants on AffectNet are shown in fig. 4 and discussed in §4.5. The authors read them qualitatively; no numbers are given.

Paper vs code

No official code.

Searched: the paper, its arXiv metadata and the LaTeX source have no repository or project-page link (the only URL is a footnote to the pytorch-image-models weights §4.2). GitHub search for "TransFER" (also with "facial expression" and in:name), "ViT-FER", "multi-attention dropping" and "transfer facial expression recognition" found no repository from the authors. The first author's GitHub account (FangleiXue) holds two forks (alphafold3, biotite) and no matching repository. User searches for the other two authors did not turn up their accounts.

Pitfalls

• Equation 2 writes the three projections as one product. Read it as three separate projections with the shapes given (w_q and w_k in d × d_k, w_v in d × d_v). Biases are not stated eq. 2 §3.3.
• The letter k is used for the number of self-attention modules §3.3 and for the key matrix in eq. 2 and eq. 3. Do not conflate them.
• The name MAD is used for two different operations. In §3.2 and §3.1 it drops one local attention map. In §3.3 it is used for dropping one self-attention module, and that operation is named MSAD elsewhere §3.3 §1. Implement by location, not by name.
• MAD on the local maps: the paper says one map is selected uniformly and set to zero, with probability p1 §3.2. Whether the choice is per sample or per batch is not stated. The MSAD text is explicit that selection is per sample and per block §3.3.
• Because the maps are sigmoid outputs §3.1, zeroing one map changes the element-wise maximum only where that map held the maximum. This is an inference from eq. 1 and §3.1, not a stated fact.
• MSAD zeroing point: the paper does not say whether the chosen head is zeroed before the concatenation and output projection, or after it. Both give different gradients.
• No rescaling at inference: the paper says MAD does not rescale weights at inference, unlike Dropout §3.3. The text is about MAD in general, so it is read as covering the local-map version too.
• The paper says dropping two or more self-attention modules at once did not help §3.3. This is an observation without a table.
• The B sweep is inconsistent. The text says performance "first increases and starts to decrease after B = 5", but also that the best is at B = 2 §4.3 fig. 3. Read the figure values yourself if you need the curve. The value for B = 0 is not stated.
• Ablation rows do not state B. The "local CNNs only" row (90.03) is assumed to use the default B = 2, which the paper does not say.
• Text vs table: the text gives the MAD gain on AffectNet as 0.30 over the baseline §4.3. The table gives 65.94 − 65.63 = 0.31 tab. 1. This is a rounding discrepancy.
• Table 4's Dropout value on RAF-DB (90.35) equals the "local CNNs + MAD" row of Table 1, which has no MSAD tab. 1 tab. 4. The paper says MAD is "replaced" in that comparison §4.3, but does not say whether MSAD stays in the model. Check before comparing these rows.
• Hyperparameter selection: p1, p2 and B are chosen by grid search on RAF-DB §4.2 §4.3. The same RAF-DB numbers are reported as the result. The split used for the search is not stated, so the reported gain may be optimistic.
• Face alignment for FERPlus is described two ways: provided landmarks, and MTCNN §4.2. The text names MTCNN for FERPlus specifically, so the card reads MTCNN as the FERPlus choice (inference).
• AffectNet's learning rate schedule is given in iterations, not epochs §4.2. "Divided by 10 at 9.6K and 19.2K" is read as two decays by 10 each (inference).
• Warmup, gradient clipping, label smoothing, and the exact upsampling method for class balance are all not stated §4.2.
• The ViT variant and patch size are not stated. The paper gives 8 heads and 8 blocks §4.2, but not the embedding width, so a standard ViT size cannot be assumed from the text.
• The AffectNet metric is mean class accuracy on the validation set, which the paper uses in place of a test set §4.1. Whether the baseline numbers in Table 5 were re-run or copied from the original papers is not stated.
• The "first to achieve over 90% on RAF-DB" claim is made "to the best of our knowledge" by the authors §4.4.
• All results are single numbers with no variance or seeds reported §4.4.
• Cost: the paper reports the hardware (two V100s) but not training time or parameter count §4.2.

Lineage

• Builds on: ViT, which the Transformer encoder and its pretraining follow §3.3 §4.2; LANet, for the local attention blocks §3.1; Dropout, DropBlock and Cutout as regularisation context §2.3; IR-50 (ArcFace), as the stem §3 §4.2; MTCNN, for face detection §4.2; Chefer et al., for attention visualisations §4.5.
• Compared against: DLP-CNN, gACNN, IPA2LT, RAN, CovPool, SCN, DACL and KTN on RAF-DB and AffectNet tab. 5; PLD, RAN-VGG16 and SeNet50 on FERPlus tab. 6.
• Superseded by: not stated.