Angular margin losses (ArcFace and relatives)

Overview

Many embedding models (faces, re-identification, retrieval) are trained as ordinary classifiers over the training identities, after which the classifier is thrown away and only the embedding is kept. Plain softmax cross-entropy only has to separate the training classes; it does not make them compact or far apart, so features of unseen identities at test time are not discriminative enough 1801.07698 §1. Triplet-style losses avoid a classifier but need a combinatorial number of triplets and careful mining 1801.07698 §1.

Angular margin losses keep the classifier but change its geometry. Biases are dropped, every class weight W_j is L2-normalized, and the feature is L2-normalized and rescaled to a fixed norm s, so the logit for class j becomes s\cos\theta_j, where \theta_j is the angle between feature and class weight 1801.07698 §3.1 / eq. 2. A margin then makes the target class harder to win during training. The shared rule across variants: the margin touches only the ground-truth logit, the other logits stay s\cos\theta_j, and at test time there is no margin and no classifier, only a comparison of normalized embeddings 1801.07698 eq. 3 / §4.1.

All three classic margins fit a single formula, with a multiplicative angular margin m_1, an additive angular margin m_2 and an additive cosine margin m_3 1801.07698 eq. 4:

L_4=-\log\frac{e^{s(\cos(m_1\theta_{y_i}+m_2)-m_3)}}{e^{s(\cos(m_1\theta_{y_i}+m_2)-m_3)}+\sum_{j=1,j\neq y_i}^{N}e^{s\cos\theta_{j}}}

This page draws on two cards: ArcFace, which carries most of the numbers, and a forgery detector that uses an AM-Softmax loss as one component. SphereFace (arXiv:1704.08063) and CosFace (arXiv:1801.09414) have no card on wikillm, so they are described here only as ArcFace reimplements and reports them.

Variants

Normalized softmax (no margin)

Only the normalization and the scale s, with no margin (m_1=1, m_2=m_3=0) 1801.07698 eq. 2. It is sensitive to s: with ResNet50 on CASIA, s=20 reaches 94.61 CFP-FP, while s=64 falls to 89.79, below plain softmax at 94.39 1801.07698 tab. 2 / §4.2. Adding the auxiliary intra- and inter-class angle penalties of 1801.07698 eq. 5, eq. 6 helps it a bit (94.88 CFP-FP) but does not reach margin losses 1801.07698 tab. 2.

Multiplicative angular margin (SphereFace-style, m_1)

The target angle is multiplied, \cos(m_1\theta_{y_i}). In the original formulation the integer margin makes the target logit curve steep, needs approximations, and needs an auxiliary softmax loss to converge 1801.07698 §1 / §3.1. ArcFace's authors reimplement it through arccos with a non-integer m_1=1.35, which they report trains without the auxiliary loss, and measure 99.11 / 94.38 / 91.70 on LFW / CFP-FP / AgeDB, the weakest of the margin variants in their table 1801.07698 §3.1 / tab. 2.

Additive cosine margin (CosFace-style, m_3)

The target logit becomes s(\cos\theta_{y_i}-m_3). ArcFace's reimplementation with m_3=0.35 reaches 99.51 / 95.44 / 94.56 1801.07698 tab. 2. ArcFace takes its s=64 from this line of work 1801.07698 §4.1.

Additive angular margin (ArcFace, m_2)

The margin is added to the angle itself, s\cos(\theta_{y_i}+m) 1801.07698 eq. 3. The authors argue it corresponds exactly to geodesic distance on the hypersphere and gives a constant, linear angular decision margin, where the multiplicative and cosine margins give nonlinear ones 1801.07698 §3.1 / fig. 5. With s=64, m=0.5 and 512-d features it scores 99.53 / 95.56 / 95.15 on LFW / CFP-FP / AgeDB, the best row of the ablation; margins 0.4 to 0.55 are all within about 0.25 points 1801.07698 tab. 2. Card: ArcFace: Additive Angular Margin Loss for Deep Face Recognition.

AM-Softmax in a forgery detector (margin and scale not stated)

A face forgery detector uses AM-Softmax as its classification loss. The authors choose it because, in their view, it gives smaller intra-class and larger inter-class variation than cross-entropy 2103.12376 §4.3. The paper does not give the margin or the scale, so neither value can be taken from it 2103.12376 §4.3, hyperparameter table. Its ablation adds architecture modules and does not compare loss functions, so the card does not show what the loss contributes on its own 2103.12376 §5.2 / tab. 3. Card: Generalizing Face Forgery Detection with High-frequency Features.

Combined margins

Using several margins at once, e.g. (m_1, m_2, m_3) = (1, 0.3, 0.2) or (0.9, 0.4, 0.15), scores 95.12 and 95.24 CFP-FP, between CosFace and ArcFace alone 1801.07698 tab. 2.

Sub-center ArcFace (noisy labels)

Each class gets K normalized weight vectors; the N\times K cosines are max-pooled over k to one per class, and the ArcFace margin is applied on top 1801.07698 §3.2 / eq. 7. A mislabeled sample only needs to be close to one of its class's sub-centers, so noise tends to gather in non-dominant sub-centers instead of dragging the main one 1801.07698 §3.2. The full recipe uses it as a cleaning step: train with K=3, keep only the dominant sub-center per class, drop samples more than 75° from it, and retrain a K=1 ArcFace from scratch 1801.07698 §3.2 / §4.1. On noisy MS1MV0 with ResNet50 this lifts IJB-C TPR@FPR=1e-4 from 90.27 (K=1) to 93.72 (K=3) and 95.92 (clean and retrain) 1801.07698 tab. 3. Card: ArcFace.

When to use which

Implementation sketch

Input: embeddings x[B, d], class weights W[d, N] (or W[d, N, K] for sub-centers),
       labels y[B], scale s, margins (m1, m2, m3)
1. x_hat = x / ||x||;  W_hat = W / ||W|| along d
2. cos = x_hat @ W_hat                          # [B, N]  (or [B, N, K])
3. if sub-centers: cos = max over K             # [B, N]
4. for each row i, target only:
     c = clamp(cos[i, y_i], -1, 1)
     theta = arccos(c)
     cos[i, y_i] = cos(m1 * theta + m2) - m3   # ArcFace: m1=1, m2=0.5, m3=0
5. logits = s * cos
6. loss = cross_entropy(logits, y)
Inference: drop W; compare x_hat by cosine similarity.

Notes:

• The clamp in step 4 is an implementation detail, not something the paper specifies; arccos has an unbounded derivative at ±1 and floating-point cosines can slightly exceed 1. An alternative is computing \cos(\theta+m) from \cos\theta and \sin\theta.
• For \theta_{y_i} > \pi - m, \cos(\theta+m) rises again, so the additive angular margin stops penalizing very wrong samples. The paper does not discuss this; its target-logit plots cover only roughly 20° to 100° 1801.07698 fig. 4. Choose a fallback deliberately.
• During ArcFace training the target angle moves from about 90° at initialization to about 30° at the end 1801.07698 §3.1 / fig. 4.
• Optimizer, schedule and backbone details used with the loss are on the ArcFace card, including which values the paper leaves unstated.

Contradictions

SphereFace and CosFace numbers appear twice in ArcFace's ablation: as quoted from their original papers (99.42 and 99.33 LFW) and as ArcFace's own reimplementations (99.11 and 99.51 LFW) 1801.07698 tab. 2. The reimplemented SphereFace uses a non-integer m_1=1.35 rather than the original integer margin 1801.07698 §3.1, so its row is not the original method. No other card covers these losses, so there is no independent comparison.

Papers

• ArcFace: Additive Angular Margin Loss for Deep Face Recognition: additive angular margin, a unified margin formula covering SphereFace and CosFace, and sub-center ArcFace for cleaning noisy data.
• Generalizing Face Forgery Detection with High-frequency Features: uses AM-Softmax as the classification loss of a two-stream forgery detector; margin and scale are not stated.