--- kind: concept slug: angular-margin-loss title: Angular margin losses (ArcFace and relatives) summary: >- Train an embedding model through a classification head whose features and class weights are L2-normalized, so logits are scaled cosines, and penalize the true class by a margin on the angle or cosine. This forces tight, well-separated classes that transfer to unseen identities. Trade-off: the classifier matrix grows with the number of classes, and scale and margin must be tuned together. tags: [metric-learning, loss-function, face-recognition, embeddings] papers: ["1801.07698", "2103.12376"] related: [face-recognition, noisy-labels, model-inversion] status: draft updated_at: 2026-10-18 --- ## Overview Many embedding models (faces, re-identification, retrieval) are trained as ordinary classifiers over the training identities, after which the classifier is thrown away and only the embedding is kept. Plain softmax cross-entropy only has to separate the training classes; it does not make them compact or far apart, so features of unseen identities at test time are not discriminative enough [arxiv:1801.07698 §1]. Triplet-style losses avoid a classifier but need a combinatorial number of triplets and careful mining [arxiv:1801.07698 §1]. Angular margin losses keep the classifier but change its geometry. Biases are dropped, every class weight $W_j$ is L2-normalized, and the feature is L2-normalized and rescaled to a fixed norm $s$, so the logit for class $j$ becomes $s\cos\theta_j$, where $\theta_j$ is the angle between feature and class weight [arxiv:1801.07698 §3.1 / eq. 2]. A margin then makes the target class harder to win during training. The shared rule across variants: **the margin touches only the ground-truth logit**, the other logits stay $s\cos\theta_j$, and at test time there is no margin and no classifier, only a comparison of normalized embeddings [arxiv:1801.07698 eq. 3 / §4.1]. All three classic margins fit a single formula, with a multiplicative angular margin $m_1$, an additive angular margin $m_2$ and an additive cosine margin $m_3$ [arxiv:1801.07698 eq. 4]: $$ L_4=-\log\frac{e^{s(\cos(m_1\theta_{y_i}+m_2)-m_3)}}{e^{s(\cos(m_1\theta_{y_i}+m_2)-m_3)}+\sum_{j=1,j\neq y_i}^{N}e^{s\cos\theta_{j}}} $$ This page draws on two cards: ArcFace, which carries most of the numbers, and a forgery detector that uses an AM-Softmax loss as one component. SphereFace ([arXiv:1704.08063](https://arxiv.org/abs/1704.08063)) and CosFace ([arXiv:1801.09414](https://arxiv.org/abs/1801.09414)) have no card on wikillm, so they are described here only as ArcFace reimplements and reports them. ## Variants ### Normalized softmax (no margin) Only the normalization and the scale $s$, with no margin ($m_1=1$, $m_2=m_3=0$) [arxiv:1801.07698 eq. 2]. It is sensitive to $s$: with ResNet50 on CASIA, $s=20$ reaches 94.61 CFP-FP, while $s=64$ falls to 89.79, below plain softmax at 94.39 [arxiv:1801.07698 tab. 2 / §4.2]. Adding the auxiliary intra- and inter-class angle penalties of [arxiv:1801.07698 eq. 5, eq. 6] helps it a bit (94.88 CFP-FP) but does not reach margin losses [arxiv:1801.07698 tab. 2]. ### Multiplicative angular margin (SphereFace-style, $m_1$) The target angle is multiplied, $\cos(m_1\theta_{y_i})$. In the original formulation the integer margin makes the target logit curve steep, needs approximations, and needs an auxiliary softmax loss to converge [arxiv:1801.07698 §1 / §3.1]. ArcFace's authors reimplement it through arccos with a non-integer $m_1=1.35$, which they report trains without the auxiliary loss, and measure 99.11 / 94.38 / 91.70 on LFW / CFP-FP / AgeDB, the weakest of the margin variants in their table [arxiv:1801.07698 §3.1 / tab. 2]. ### Additive cosine margin (CosFace-style, $m_3$) The target logit becomes $s(\cos\theta_{y_i}-m_3)$. ArcFace's reimplementation with $m_3=0.35$ reaches 99.51 / 95.44 / 94.56 [arxiv:1801.07698 tab. 2]. ArcFace takes its $s=64$ from this line of work [arxiv:1801.07698 §4.1]. ### Additive angular margin (ArcFace, $m_2$) The margin is added to the angle itself, $s\cos(\theta_{y_i}+m)$ [arxiv:1801.07698 eq. 3]. The authors argue it corresponds exactly to geodesic distance on the hypersphere and gives a constant, linear angular decision margin, where the multiplicative and cosine margins give nonlinear ones [arxiv:1801.07698 §3.1 / fig. 5]. With $s=64$, $m=0.5$ and 512-d features it scores 99.53 / 95.56 / 95.15 on LFW / CFP-FP / AgeDB, the best row of the ablation; margins 0.4 to 0.55 are all within about 0.25 points [arxiv:1801.07698 tab. 2]. Card: [ArcFace: Additive Angular Margin Loss for Deep Face Recognition](/papers/1801.07698). ### AM-Softmax in a forgery detector (margin and scale not stated) A face forgery detector uses AM-Softmax as its classification loss. The authors choose it because, in their view, it gives smaller intra-class and larger inter-class variation than cross-entropy [arxiv:2103.12376 §4.3]. The paper does not give the margin or the scale, so neither value can be taken from it [arxiv:2103.12376 §4.3, hyperparameter table]. Its ablation adds architecture modules and does not compare loss functions, so the card does not show what the loss contributes on its own [arxiv:2103.12376 §5.2 / tab. 3]. Card: [Generalizing Face Forgery Detection with High-frequency Features](/papers/2103.12376). ### Combined margins Using several margins at once, e.g. $(m_1, m_2, m_3) = (1, 0.3, 0.2)$ or $(0.9, 0.4, 0.15)$, scores 95.12 and 95.24 CFP-FP, between CosFace and ArcFace alone [arxiv:1801.07698 tab. 2]. ### Sub-center ArcFace (noisy labels) Each class gets $K$ normalized weight vectors; the $N\times K$ cosines are max-pooled over $k$ to one per class, and the ArcFace margin is applied on top [arxiv:1801.07698 §3.2 / eq. 7]. A mislabeled sample only needs to be close to one of its class's sub-centers, so noise tends to gather in non-dominant sub-centers instead of dragging the main one [arxiv:1801.07698 §3.2]. The full recipe uses it as a cleaning step: train with $K=3$, keep only the dominant sub-center per class, drop samples more than 75° from it, and retrain a $K=1$ ArcFace from scratch [arxiv:1801.07698 §3.2 / §4.1]. On noisy MS1MV0 with ResNet50 this lifts IJB-C TPR@FPR=1e-4 from 90.27 ($K=1$) to 93.72 ($K=3$) and 95.92 (clean and retrain) [arxiv:1801.07698 tab. 3]. Card: [ArcFace](/papers/1801.07698). ## When to use which | Situation | Use | Why | |---|---|---| | Default for training an embedding through a classification head | ArcFace, $s=64$, $m=0.5$ | Best row of the CASIA / ResNet50 loss ablation, ahead of reimplemented CosFace, SphereFace, combined margins and triplet loss [arxiv:1801.07698 tab. 2] | | Classifier for a non-face task, such as forgery detection | AM-Softmax, with margin and scale chosen by you | The one card using this loss picks it for its intra- and inter-class effect, but states no margin or scale [arxiv:2103.12376 §4.3] | | You already have a CosFace-style setup | Additive cosine margin is close behind | 0.12 CFP-FP and 0.59 AgeDB below ArcFace in the same ablation [arxiv:1801.07698 tab. 2]; a small gap, measured on one dataset and backbone | | You want a normalized classifier but no margin | Normalized softmax with a small $s$ (20 in the ablation) | Without a margin, $s=64$ scores 89.79 CFP-FP against 94.61 at $s=20$ [arxiv:1801.07698 tab. 2] | | Training data is web-scraped and noisy | Sub-center ArcFace, $K=3$, then drop >75° and retrain with $K=1$ | Largest gains on MS1MV0 and Celeb500K [arxiv:1801.07698 tab. 3] | | Training data is already clean | Plain ArcFace, $K=1$ | On MS1MV3, $K=3$ is slightly worse than $K=1$ (96.35 vs 96.50 IJB-C) [arxiv:1801.07698 tab. 3] | | Tight GPU memory for the classifier | Keep $K$ small | Classifier size scales with $N\cdot K$; $K=3$ was chosen partly for memory, and $K=10$ collapsed to 67.94 IJB-C [arxiv:1801.07698 §4.3 / tab. 3] | | Adding triplet or angle regularizers on top of ArcFace | Usually not worth it | ArcFace+Intra, +Inter and +Triplet do not beat plain ArcFace on CFP-FP or AgeDB [arxiv:1801.07698 tab. 2] | ## Implementation sketch ```text Input: embeddings x[B, d], class weights W[d, N] (or W[d, N, K] for sub-centers), labels y[B], scale s, margins (m1, m2, m3) 1. x_hat = x / ||x||; W_hat = W / ||W|| along d 2. cos = x_hat @ W_hat # [B, N] (or [B, N, K]) 3. if sub-centers: cos = max over K # [B, N] 4. for each row i, target only: c = clamp(cos[i, y_i], -1, 1) theta = arccos(c) cos[i, y_i] = cos(m1 * theta + m2) - m3 # ArcFace: m1=1, m2=0.5, m3=0 5. logits = s * cos 6. loss = cross_entropy(logits, y) Inference: drop W; compare x_hat by cosine similarity. ``` Notes: - The clamp in step 4 is an implementation detail, not something the paper specifies; arccos has an unbounded derivative at ±1 and floating-point cosines can slightly exceed 1. An alternative is computing $\cos(\theta+m)$ from $\cos\theta$ and $\sin\theta$. - For $\theta_{y_i} > \pi - m$, $\cos(\theta+m)$ rises again, so the additive angular margin stops penalizing very wrong samples. The paper does not discuss this; its target-logit plots cover only roughly 20° to 100° [arxiv:1801.07698 fig. 4]. Choose a fallback deliberately. - During ArcFace training the target angle moves from about 90° at initialization to about 30° at the end [arxiv:1801.07698 §3.1 / fig. 4]. - Optimizer, schedule and backbone details used with the loss are on the [ArcFace card](/papers/1801.07698), including which values the paper leaves unstated. ## Contradictions SphereFace and CosFace numbers appear twice in ArcFace's ablation: as quoted from their original papers (99.42 and 99.33 LFW) and as ArcFace's own reimplementations (99.11 and 99.51 LFW) [arxiv:1801.07698 tab. 2]. The reimplemented SphereFace uses a non-integer $m_1=1.35$ rather than the original integer margin [arxiv:1801.07698 §3.1], so its row is not the original method. No other card covers these losses, so there is no independent comparison. ## Papers - [ArcFace: Additive Angular Margin Loss for Deep Face Recognition](/papers/1801.07698): additive angular margin, a unified margin formula covering SphereFace and CosFace, and sub-center ArcFace for cleaning noisy data. - [Generalizing Face Forgery Detection with High-frequency Features](/papers/2103.12376): uses AM-Softmax as the classification loss of a two-stream forgery detector; margin and scale are not stated.