--- kind: concept slug: factorization-machine title: Factorization machine summary: >- A factorization machine scores a feature vector with a bias, per-feature weights and a factor dot product for every feature pair. On iPinYou it trails logistic regression in one paper (68.18% vs 68.81% pooled AUC) and beats it in another (75.52% vs 73.43%). Under tanh layers it reaches 70.70%. The CTR and DeepCoNN papers do not state a factor size. tags: [recommender-systems, rating-prediction, ctr-prediction, factorization, feature-embeddings, feature-interactions] papers: ["1601.02376", "1701.04783", "1611.00144"] related: [collaborative-filtering, denoising-autoencoder, greedy-layerwise-training, multi-layer-perceptron] status: draft updated_at: 2026-10-24 --- ## Overview A factorization machine (FM) turns a feature vector into one score. The score is the sum of three parts: a global bias, a weight for each feature, and an interaction term over every pair of features. Each feature also gets a factor vector, and the interaction for a pair is the dot product of their two factor vectors, scaled by the two feature values [arxiv:1701.04783 §2.5 eq. 7] [arxiv:1601.02376 §3.1 eq. 6]. The pairwise term lets the score depend on combinations of inputs, not only on each input alone, while the factor vectors keep the number of interaction parameters bounded. The three wiki papers use the FM or its pairwise term in different roles: - **As a click predictor of its own** on one-hot categorical fields, where it is a baseline next to logistic regression [arxiv:1601.02376 §4.1 tab. 1] [arxiv:1611.00144 tab. 2]. - **As the bottom layer of a deep network**, where it is trained first and its bias and factor vectors become the first-layer input of the network [arxiv:1601.02376 §3.1]. - **As the pairwise interaction inside a deep network**, where the inner products of learned field vectors feed the hidden layers and no FM is trained first [arxiv:1611.00144 §3.1.1]. - **As the coupling layer between two learned vectors**, where it scores the concatenation of a user vector and an item vector [arxiv:1701.04783 §2.2] [arxiv:1701.04783 §2.5]. **Shared structure.** The FM variants share a bias, linear weights and factor vectors, or at least the pairwise term. The pairwise-only variant drops the bias and the linear weights. What changes is the output and its target. The CTR paper trains a click probability with cross-entropy loss [arxiv:1601.02376 eq. 7], and PNN uses log loss on a click label [arxiv:1611.00144 eq. 9]. DeepCoNN predicts a rating, and the paper calls its training loss "cost" without writing it out [arxiv:1701.04783 §2.6]. ## Variants ### Standalone FM on one-hot fields (CTR baseline) Each categorical value is one feature, so the FM sees a sparse one-hot vector with one active value per field. The paper reports it as a baseline next to logistic regression (LR) [arxiv:1601.02376 §4.1]. Its pooled AUC is 68.18%, against 68.81% for LR [arxiv:1601.02376 tab. 1]. The paper says that FM is "not significantly better than LR", and that second-order combinations "might not be good enough" [arxiv:1601.02376 §4.2]. Per advertiser it is ahead on 2259 (69.73% vs 69.66%) and 2997 (60.87% vs 60.77%), and behind on 1458, 2261 and 3386 [arxiv:1601.02376 tab. 1]. Card: [the CTR paper](/papers/1601.02376). PNN reports the opposite order on iPinYou: its FM reaches 75.52% AUC against 73.43% for LR [arxiv:1611.00144 tab. 2]. See Contradictions below. ### FM embedding under tanh layers (FNN) The paper trains an FM first. For each field it then keeps a slot holding the bias and the factor vector of that field's active value, and places these slots in the first-layer input. The first layer is therefore connected only within each field. Two tanh layers and a sigmoid output sit on top [arxiv:1601.02376 §3.1 eq. 1–5]. The paper gives the reason for the sparse layout: with about 1 million binary inputs and 100 first-layer units, a dense first layer would need about 100 million links [arxiv:1601.02376 §1]. On pooled data the FNN reaches 70.70% AUC, against 68.81% for LR and 68.18% for FM [arxiv:1601.02376 tab. 1]. It is the best row on advertisers 1458, 2259 and 3386, and SNN-DAE or SNN-RBM is ahead on 2261 and 2997 [arxiv:1601.02376 tab. 1]. The table compares whole models, so the gain cannot be credited to the FM embedding alone. Card: [the CTR paper](/papers/1601.02376). ### Inner products as a layer in a deep network (IPNN) PNN gives each field a learned vector, with no pretrained FM, and takes the inner product of every pair of field vectors [arxiv:1611.00144 §3.1.1]. These pair values form a product layer, and ReLU layers and a sigmoid output sit on top [arxiv:1611.00144 §3.1] [eq. 1–3]. The paper assumes a rank-one weight structure for the first hidden layer. Under that assumption the pairwise part reduces to one squared norm per hidden node, which gives a cost linear in the number of fields [arxiv:1611.00144 §3.1.1] [eq. 11–12]. The reported AUC is 75.66% for FNN and 77.79% for IPNN on Criteo [arxiv:1611.00144 tab. 1]. On iPinYou it is 76.19% for FNN and 79.14% for IPNN [arxiv:1611.00144 tab. 2]. Those FNN rows come from the PNN paper's own table and differ from the 70.70% of the CTR paper, so do not mix the two. These are whole-model comparisons, and the gain over FM is not isolated. Card: [PNN](/papers/1611.00144). ### FM over the concatenation of two tower outputs (DeepCoNN) The two towers each output a 50-dimensional vector, and the FM scores their concatenation of 100 values [arxiv:1701.04783 §2.5] [arxiv:1701.04783 §3.3]. This is the full model in the paper's ablation, with test MSE 1.441 on Yelp, 1.233 on Amazon Music Instruments and 0.273 on Beer [arxiv:1701.04783 tab. 4]. Card: [DeepCoNN](/papers/1701.04783). The paper does not run an ablation that removes only the pairwise term or only the per-feature weights, so their separate contributions are not known from this source [arxiv:1701.04783 §3.5]. ### Dot product of the two tower outputs (DeepCoNN-DP) The ablation swaps the FM for the dot product of the user vector and the item vector [arxiv:1701.04783 §3.5]. It has no bias, no per-feature weights and no learned factor vectors of its own. The test MSE is 1.491 on Yelp, 1.253 on Amazon Music Instruments and 0.278 on Beer [arxiv:1701.04783 tab. 4]. Same card as above: [DeepCoNN](/papers/1701.04783). The paper reports the gap only as MSE. It gives no run time, memory or parameter count for either coupling layer [arxiv:1701.04783 §3.5]. ## When to use which | Situation | Use | Why | |---|---|---| | Sparse categorical fields, and you need a simple baseline | Logistic regression first, then a standalone FM | In the CTR paper the FM is below LR on pooled data, and ahead on only two of five advertisers [arxiv:1601.02376 tab. 1]. PNN reports the FM ahead of LR on iPinYou [arxiv:1611.00144 tab. 2], so measure on your own data | | Sparse categorical fields, and you want the best pooled AUC the CTR paper measured | FM embedding under tanh layers (FNN) | 70.70% pooled AUC against 68.81% for LR. Needs a separate FM training stage before the network [arxiv:1601.02376 §3.1 tab. 1] | | Sparse categorical fields, and you cannot afford an FM pre-training stage | IPNN, with embeddings learned from scratch | PNN reports IPNN ahead of its FNN and FM rows on both datasets [arxiv:1611.00144 tab. 1] [tab. 2]. Author-reported. The paper suggests a pretrained start but reports no experiment with one [arxiv:1611.00144 §3.2] | | About 1 million one-hot inputs | Per-field FM slots, not a dense first layer | A dense first layer would need about 100 million links [arxiv:1601.02376 §1] | | Two learned vectors per pair, rating prediction | FM over the concatenation | Beats the dot product by 0.005 to 0.050 MSE on three ablation datasets [arxiv:1701.04783 tab. 4] | | The simplest coupling is acceptable | Dot product of the two vectors | It is 0.005 to 0.050 MSE worse on the three ablation datasets, and the gap is smallest on Beer [arxiv:1701.04783 tab. 4]. Cost of either is not measured in the paper | | Reproducing the paper's numbers | Measure your own factor size and regularisation | Factor size, FM learning rate and regularisation are not stated in the CTR text [arxiv:1601.02376 §3.1] [arxiv:1701.04783 §2.5 §2.6]. PNN uses a factor order of 10 for its FM [arxiv:1611.00144 §4.4] | The CTR paper and PNN report AUC on iPinYou, and DeepCoNN reports rating MSE. The two AUC results disagree on the order of FM and LR; see Contradictions. The ablation results in DeepCoNN come from the Amazon Music Instruments subset, not the 21-category Amazon average in the headline table [arxiv:1701.04783 tab. 3] [arxiv:1701.04783 tab. 4]. Do not compare the Amazon numbers across the two tables. ## Implementation sketch Naive form of the paper's scoring rule, with every pair written out. The CTR paper and DeepCoNN write this sum, and neither describes a faster evaluation. ```text Input: z[d] (feature values; one-hot inputs have z = 1, while DeepCoNN uses its concatenated tower outputs with d = 100), w0 (scalar bias), w[d] (per-feature weights), V[d, k] (factor vectors; DeepCoNN's notation sets k = d [eq. 7]) 1. # global bias and per-feature linear term [arxiv:1701.04783 eq. 7] score = w0 + sum over i of w[i] * z[i] 2. # pairwise term: dot product of factors, scaled by both values # [arxiv:1601.02376 eq. 6] # [arxiv:1701.04783 eq. 7] for i in 1..d: for j in i+1..d: score += dot(V[i], V[j]) * z[i] * z[j] 3. # output: a click probability (sigmoid, cross-entropy) for CTR # [arxiv:1601.02376 eq. 7] # or the rating itself, trained against r_ui for DeepCoNN return score ``` ## Contradictions - **Within the CTR paper.** The text says the FNN and SNN models beat LR and FM on all tested datasets [arxiv:1601.02376 §4.2]. Table 1 does not show this for SNN: on advertisers 2259 and 3386, both SNN variants are below LR [arxiv:1601.02376 tab. 1]. The FNN row holds: it is above LR and FM on every line of the table [arxiv:1601.02376 tab. 1]. - **FM against LR on iPinYou.** The CTR paper reports FM at 68.18% and LR at 68.81% pooled AUC [arxiv:1601.02376 tab. 1]. PNN reports FM at 75.52% and LR at 73.43% on iPinYou [arxiv:1611.00144 tab. 2]. Both papers state 19.50M instances and 937.67K features for this data [arxiv:1601.02376 §4.1] [arxiv:1611.00144 §4.1.2]. The split and preprocessing differ, and the papers do not say how far that explains the gap. Neither result is resolved here. - **FNN values.** The CTR paper reports FNN at 70.70% pooled AUC [arxiv:1601.02376 tab. 1]. PNN reports FNN at 76.19% on iPinYou [arxiv:1611.00144 tab. 2]. Same model name, different numbers. - **Between the two AUC papers and DeepCoNN.** None known as a direct comparison. DeepCoNN reports rating MSE, so its FM numbers are not comparable with the two AUC tables. ## Pitfalls - **Pre-training is in the paper, not in the code.** The paper initialises the FNN hidden layers by layer-wise RBM pre-training [arxiv:1601.02376 §3.1]. The paper card's "Paper vs code" section records that the FNN code draws these weights at random with no pre-training step. See [the card](/papers/1601.02376). - **Sparse updates.** Only the FM weights linked to active inputs receive an update during fine-tuning. The paper says this reduces computation [arxiv:1601.02376 §3.1 eq. 8–9]. That is the paper's own claim, not a measurement reported in the paper. - **Factor size and FM training.** The CTR paper does not state the FM's factor size `K`, its learning rate or its training length [arxiv:1601.02376 §3.1] [arxiv:1601.02376 §4.3]. The paper card records the repository's value for `K` and notes that it is not in the text. PNN states a factor order of 10 for its FM baseline [arxiv:1611.00144 §4.4]. - **Factor size in DeepCoNN.** Eq. 7 sums the factor inner product over the full input length, which would make each factor vector as long as the input. The paper does not say whether this is intended or a notation slip for a smaller factor size [arxiv:1701.04783 §2.5 eq. 7]. The sketch above keeps the printed form. - **Reduced IPNN form.** Eq. 11 and eq. 12 of PNN are written differently: eq. 11 gives a squared inner product per hidden node, and eq. 12 is printed with a single norm. The card takes the squared form as the derivation. Check any implementation against the unreduced sum on a small input before relying on it [arxiv:1611.00144 eq. 11–12]. - **Gradient.** The gradient written in DeepCoNN, eq. 8, sums only over $j > i$. For a symmetric pairwise term the sum must cover every $j \neq i$, so use autodiff rather than coding eq. 8 [arxiv:1701.04783 §2.6 eq. 8]. - **Loss and regularisation.** DeepCoNN calls the FM output the cost and never states a training loss. MSE is the reported metric, so squared error is the likely objective. Regularisation on the FM weights is not stated [arxiv:1701.04783 §2.5 §2.6 eq. 11]. - **Cold start.** DeepCoNN's cold-start analysis measures the whole model against MF, not the FM against the dot product. Do not read it as evidence for the FM's contribution alone [arxiv:1701.04783 §3.6]. - **Whole-model comparisons.** The CTR table compares FNN with FM and LR as complete models. A gain over FM therefore mixes the embedding with the deep layers, and this table does not separate the two. The same holds for IPNN against FM in PNN [arxiv:1611.00144 tab. 1] [tab. 2]. ## Papers - [CTR paper](/papers/1601.02376): FNN puts an FM embedding under tanh layers, and the FM alone is a baseline for click prediction on iPinYou. - [PNN](/papers/1611.00144): IPNN takes inner products of learned field embeddings as its product layer, and reports an FM baseline on Criteo and iPinYou. - [DeepCoNN](/papers/1701.04783): uses an FM as the coupling layer between two review-text CNN towers and compares it with a dot product.