Factorization machine

Overview

A factorization machine (FM) turns a feature vector into one score. The score is the sum of three parts: a global bias, a weight for each feature, and an interaction term over every pair of features. Each feature also gets a factor vector, and the interaction for a pair is the dot product of their two factor vectors, scaled by the two feature values 1701.04783 §2.5 eq. 7 1601.02376 §3.1 eq. 6.

The pairwise term lets the score depend on combinations of inputs, not only on each input alone, while the factor vectors keep the number of interaction parameters bounded. The three wiki papers use the FM or its pairwise term in different roles:

• As a click predictor of its own on one-hot categorical fields, where it is a baseline next to logistic regression 1601.02376 §4.1 tab. 1 1611.00144 tab. 2.
• As the bottom layer of a deep network, where it is trained first and its bias and factor vectors become the first-layer input of the network 1601.02376 §3.1.
• As the pairwise interaction inside a deep network, where the inner products of learned field vectors feed the hidden layers and no FM is trained first 1611.00144 §3.1.1.
• As the coupling layer between two learned vectors, where it scores the concatenation of a user vector and an item vector 1701.04783 §2.2 1701.04783 §2.5.

Shared structure. The FM variants share a bias, linear weights and factor vectors, or at least the pairwise term. The pairwise-only variant drops the bias and the linear weights. What changes is the output and its target. The CTR paper trains a click probability with cross-entropy loss 1601.02376 eq. 7, and PNN uses log loss on a click label 1611.00144 eq. 9. DeepCoNN predicts a rating, and the paper calls its training loss "cost" without writing it out 1701.04783 §2.6.

Variants

Standalone FM on one-hot fields (CTR baseline)

Each categorical value is one feature, so the FM sees a sparse one-hot vector with one active value per field. The paper reports it as a baseline next to logistic regression (LR) 1601.02376 §4.1. Its pooled AUC is 68.18%, against 68.81% for LR 1601.02376 tab. 1. The paper says that FM is "not significantly better than LR", and that second-order combinations "might not be good enough" 1601.02376 §4.2. Per advertiser it is ahead on 2259 (69.73% vs 69.66%) and 2997 (60.87% vs 60.77%), and behind on 1458, 2261 and 3386 1601.02376 tab. 1. Card: the CTR paper.

PNN reports the opposite order on iPinYou: its FM reaches 75.52% AUC against 73.43% for LR 1611.00144 tab. 2. See Contradictions below.

FM embedding under tanh layers (FNN)

The paper trains an FM first. For each field it then keeps a slot holding the bias and the factor vector of that field's active value, and places these slots in the first-layer input. The first layer is therefore connected only within each field. Two tanh layers and a sigmoid output sit on top 1601.02376 §3.1 eq. 1–5. The paper gives the reason for the sparse layout: with about 1 million binary inputs and 100 first-layer units, a dense first layer would need about 100 million links 1601.02376 §1.

On pooled data the FNN reaches 70.70% AUC, against 68.81% for LR and 68.18% for FM 1601.02376 tab. 1. It is the best row on advertisers 1458, 2259 and 3386, and SNN-DAE or SNN-RBM is ahead on 2261 and 2997 1601.02376 tab. 1. The table compares whole models, so the gain cannot be credited to the FM embedding alone. Card: the CTR paper.

Inner products as a layer in a deep network (IPNN)

PNN gives each field a learned vector, with no pretrained FM, and takes the inner product of every pair of field vectors 1611.00144 §3.1.1. These pair values form a product layer, and ReLU layers and a sigmoid output sit on top 1611.00144 §3.1 eq. 1–3. The paper assumes a rank-one weight structure for the first hidden layer. Under that assumption the pairwise part reduces to one squared norm per hidden node, which gives a cost linear in the number of fields 1611.00144 §3.1.1 eq. 11–12.

The reported AUC is 75.66% for FNN and 77.79% for IPNN on Criteo 1611.00144 tab. 1. On iPinYou it is 76.19% for FNN and 79.14% for IPNN 1611.00144 tab. 2. Those FNN rows come from the PNN paper's own table and differ from the 70.70% of the CTR paper, so do not mix the two. These are whole-model comparisons, and the gain over FM is not isolated. Card: PNN.

FM over the concatenation of two tower outputs (DeepCoNN)

The two towers each output a 50-dimensional vector, and the FM scores their concatenation of 100 values 1701.04783 §2.5 1701.04783 §3.3. This is the full model in the paper's ablation, with test MSE 1.441 on Yelp, 1.233 on Amazon Music Instruments and 0.273 on Beer 1701.04783 tab. 4. Card: DeepCoNN.

The paper does not run an ablation that removes only the pairwise term or only the per-feature weights, so their separate contributions are not known from this source 1701.04783 §3.5.

Dot product of the two tower outputs (DeepCoNN-DP)

The ablation swaps the FM for the dot product of the user vector and the item vector 1701.04783 §3.5. It has no bias, no per-feature weights and no learned factor vectors of its own. The test MSE is 1.491 on Yelp, 1.253 on Amazon Music Instruments and 0.278 on Beer 1701.04783 tab. 4. Same card as above: DeepCoNN.

The paper reports the gap only as MSE. It gives no run time, memory or parameter count for either coupling layer 1701.04783 §3.5.

When to use which

The CTR paper and PNN report AUC on iPinYou, and DeepCoNN reports rating MSE. The two AUC results disagree on the order of FM and LR; see Contradictions.

The ablation results in DeepCoNN come from the Amazon Music Instruments subset, not the 21-category Amazon average in the headline table 1701.04783 tab. 3 1701.04783 tab. 4. Do not compare the Amazon numbers across the two tables.

Implementation sketch

Naive form of the paper's scoring rule, with every pair written out. The CTR paper and DeepCoNN write this sum, and neither describes a faster evaluation.

Input: z[d] (feature values; one-hot inputs have z = 1, while DeepCoNN
       uses its concatenated tower outputs with d = 100),
       w0 (scalar bias), w[d] (per-feature weights),
       V[d, k] (factor vectors; DeepCoNN's notation sets k = d [eq. 7])
1. # global bias and per-feature linear term        [arxiv:1701.04783 eq. 7]
   score = w0 + sum over i of w[i] * z[i]
2. # pairwise term: dot product of factors, scaled by both values
   #                                                 [arxiv:1601.02376 eq. 6]
   #                                                 [arxiv:1701.04783 eq. 7]
   for i in 1..d:
     for j in i+1..d:
       score += dot(V[i], V[j]) * z[i] * z[j]
3. # output: a click probability (sigmoid, cross-entropy) for CTR
   #                                                 [arxiv:1601.02376 eq. 7]
   #   or the rating itself, trained against r_ui for DeepCoNN
   return score

Contradictions

• Within the CTR paper. The text says the FNN and SNN models beat LR and FM on all tested datasets 1601.02376 §4.2. Table 1 does not show this for SNN: on advertisers 2259 and 3386, both SNN variants are below LR 1601.02376 tab. 1. The FNN row holds: it is above LR and FM on every line of the table 1601.02376 tab. 1.
• FM against LR on iPinYou. The CTR paper reports FM at 68.18% and LR at 68.81% pooled AUC 1601.02376 tab. 1. PNN reports FM at 75.52% and LR at 73.43% on iPinYou 1611.00144 tab. 2. Both papers state 19.50M instances and 937.67K features for this data 1601.02376 §4.1 1611.00144 §4.1.2. The split and preprocessing differ, and the papers do not say how far that explains the gap. Neither result is resolved here.
• FNN values. The CTR paper reports FNN at 70.70% pooled AUC 1601.02376 tab. 1. PNN reports FNN at 76.19% on iPinYou 1611.00144 tab. 2. Same model name, different numbers.
• Between the two AUC papers and DeepCoNN. None known as a direct comparison. DeepCoNN reports rating MSE, so its FM numbers are not comparable with the two AUC tables.

Pitfalls

• Pre-training is in the paper, not in the code. The paper initialises the FNN hidden layers by layer-wise RBM pre-training 1601.02376 §3.1. The paper card's "Paper vs code" section records that the FNN code draws these weights at random with no pre-training step. See the card.
• Sparse updates. Only the FM weights linked to active inputs receive an update during fine-tuning. The paper says this reduces computation 1601.02376 §3.1 eq. 8–9. That is the paper's own claim, not a measurement reported in the paper.
• Factor size and FM training. The CTR paper does not state the FM's factor size K, its learning rate or its training length 1601.02376 §3.1 1601.02376 §4.3. The paper card records the repository's value for K and notes that it is not in the text. PNN states a factor order of 10 for its FM baseline 1611.00144 §4.4.
• Factor size in DeepCoNN. Eq. 7 sums the factor inner product over the full input length, which would make each factor vector as long as the input. The paper does not say whether this is intended or a notation slip for a smaller factor size 1701.04783 §2.5 eq. 7. The sketch above keeps the printed form.
• Reduced IPNN form. Eq. 11 and eq. 12 of PNN are written differently: eq. 11 gives a squared inner product per hidden node, and eq. 12 is printed with a single norm. The card takes the squared form as the derivation. Check any implementation against the unreduced sum on a small input before relying on it 1611.00144 eq. 11–12.
• Gradient. The gradient written in DeepCoNN, eq. 8, sums only over j > i. For a symmetric pairwise term the sum must cover every j \neq i, so use autodiff rather than coding eq. 8 1701.04783 §2.6 eq. 8.
• Loss and regularisation. DeepCoNN calls the FM output the cost and never states a training loss. MSE is the reported metric, so squared error is the likely objective. Regularisation on the FM weights is not stated 1701.04783 §2.5 §2.6 eq. 11.
• Cold start. DeepCoNN's cold-start analysis measures the whole model against MF, not the FM against the dot product. Do not read it as evidence for the FM's contribution alone 1701.04783 §3.6.
• Whole-model comparisons. The CTR table compares FNN with FM and LR as complete models. A gain over FM therefore mixes the embedding with the deep layers, and this table does not separate the two. The same holds for IPNN against FM in PNN 1611.00144 tab. 1 tab. 2.

Papers

• CTR paper: FNN puts an FM embedding under tanh layers, and the FM alone is a baseline for click prediction on iPinYou.
• PNN: IPNN takes inner products of learned field embeddings as its product layer, and reports an FM baseline on Criteo and iPinYou.
• DeepCoNN: uses an FM as the coupling layer between two review-text CNN towers and compares it with a dot product.