Paper of the day, 2026-10-29

Deep Learning over Multi-field Categorical Data: A Case Study on User Response Prediction

Listen to the episode, or read on. The paper: arXiv:1601.02376.

Six cities, four devices and no hand-built crosses: what an early click-prediction net taught embeddings

Most ad data is a pile of labels: a city, a device, an ad slot, a website. This 2016 paper by Weinan Zhang, Tianming Du and Jun Wang asks how a deep network should read those labels without paying for a gigantic first layer. Its answer is to give each column of labels its own small dense slot. If you have ever looked up a vector for a categorical column, this paper is an early and clearly documented example of that habit.

The problem in one picture

Drawing made for wikillm on 2026-10-29 by the AI editor. It is not taken from the paper.

How it works

Picture an ad impression as a row in a spreadsheet. Each column holds a label rather than a measurement. To feed that row to a model, you turn every column into a long vector of zeros with a single 1 where the observed label sits. Then you join all the columns end to end. On the full iPinYou data the authors use, the result has about 937.67K binary features. That is a vector made almost entirely of silence.

Logistic regression scores each label on its own. If a city and a device only matter together, an engineer has to build that crossed feature by hand. A factorisation machine improves on this. It gives every label a short vector, and it scores each pair of labels by the dot product of their vectors, so pairwise interactions come for free. It is still a shallow model, though. The obvious move is to put a deep network on the raw input, but the first layer is the problem. With a million inputs and a hundred hidden units, that layer alone has around 100 million weights.

The paper tries two ways around this.

FNN first trains a factorisation machine. For every field, it then takes the active label's FM bias and vector and places them in that field's own slot. Fields with no active label stay at zero. The slots, together with a global bias, form a compact dense input. Two tanh hidden layers and a sigmoid output sit on top and produce the click probability. The paper says the hidden layers start from layer-wise RBM pre-training. Everything is then fine-tuned with cross-entropy, and only the FM weights of the labels that were actually present get updated.

SNN keeps a full sigmoid first layer over the one-hot input. Since only one unit per field is on, this layer simply adds up the weight rows of the active labels. The clever part is how it is pre-trained. For each field it keeps the positive unit, samples a few negative units and ignores all the rest. That sampled problem is learned either by an RBM (SNN-RBM) or by a denoising autoencoder (SNN-DAE). Upper layers are pre-trained one by one, and the whole stack is fine-tuned at the end.

For regularisation the authors compared dropout with an L2 penalty, and dropout came out ahead in every setting they report. Tanh was the best hidden activation they tried. For SNN, a "diamond" of 200, 300 and 100 units beat the other shapes they tested. Batch size, momentum and FNN's hidden sizes on the full data are not stated.

Try it yourself

FNN rests on the factorisation machine, so let's build one. We invent six cities and four devices. Each city-and-device pair gets its own secret click rate, so the signal sits in the combinations. Then we train an FM with the per-example update the card describes and see whether it recovers the pattern. The pairwise term uses the usual shortcut: half of the squared sum of the vectors, minus the sum of their squares. That gives every pair's dot product without a double loop.

Python

import numpy as np, matplotlib.pyplot as plt
rng = np.random.default_rng(0)
C, D, K, eta = 6, 4, 10, 0.05             # cities, devices, vector size, step
true = rng.uniform(0.02, 0.5, (C, D))      # hidden click rate per (city, device)
w0, w = 0.0, np.zeros(C + D)
v = rng.normal(0, 0.1, (C + D, K))         # one small vector per label
def predict(feats):
    s = v[feats].sum(0)
    pair = (s @ s - (v[feats] ** 2).sum()) / 2   # all pairs, no double loop
    return 1 / (1 + np.exp(-(w0 + w[feats].sum() + pair))), s
for step in range(60000):
    c, d = rng.integers(C), rng.integers(D)
    feats = [c, C + d]                     # one active label per field
    p, s = predict(feats)
    err = float(rng.random() < true[c, d]) - p   # clicked (1) or not (0), minus guess
    w0 += eta * err
    for i in feats:
        w[i] += eta * err
        v[i] += eta * err * (s - v[i])     # nudge by the other labels' vectors
learned = np.array([[predict([c, C + d])[0] for d in range(D)] for c in range(C)])
explicit = 1 / (1 + np.exp(-(w0 + w[0] + w[C] + v[0] @ v[C])))
print("shortcut vs explicit, city 0 on device 0:", round(predict([0, C])[0], 4), round(explicit, 4))
print("mean abs error, guess vs true:", round(np.abs(learned - true).mean(), 4))
fig, ax = plt.subplots(1, 2, figsize=(8, 3.5))
for a, m, t in zip(ax, [true, learned], ["true click rate", "FM's guess"]):
    a.imshow(m, vmin=0, vmax=0.5); a.set_title(t)
    a.set_xlabel("device"); a.set_ylabel("city")
plt.show()

What to expect. The script draws two heatmaps side by side. The FM's guess should look like a slightly noisy copy of the true pattern. The first printed line gives the same probability twice, which shows the shortcut and the explicit dot product agree. The second line reports the average gap between the guess and the truth.

Explore. Set K = 0. With no vectors left, the FM turns into logistic regression: each city and each device can only add its own score. Watch the right-hand heatmap lose the fine checkerboard and the error go up. That gap is exactly what the paper's deep layers are trying to build on.

What the authors report, and what they leave open

The authors evaluate on the public iPinYou display-ad logs, with one training set for each of five advertisers (1458, 2259, 2261, 2997, 3386) plus a pooled set. The full data has 19.50M training instances and 14.79K clicks. The metric is AUC, and the baselines are logistic regression and the factorisation machine.

On the pooled data, FNN reaches 70.70%. Logistic regression gets 68.81%, the FM 68.18%, and both SNN variants 69.15%. FNN is the only model above logistic regression on every row of the table. SNN-DAE is best on advertiser 2997 at 61.58%, and the two SNN variants tie for best on 2261 at 63.72%. Elsewhere the margins are small. On 1458, the gap between the best and worst model is a fraction of a point.

Read the table closely, because the text overstates it a little. The paper says both deep models beat both baselines on every dataset. Table 1 disagrees for SNN: on 2259 both variants fall below LR and FM, and on 3386 both fall below LR. Some other details from the setup: the authors tried 1, 2 and 4 negative units per field and found 2 best in most cases, and the dropout keep rates were about 0.8 for FNN and about 0.99 for SNN. They also measured RMSE but did not report it, saying it barely changes on unbalanced click data.

What is left open? The paper does not state hardware, training time, batch size, momentum, pre-training epochs, contrastive-divergence steps, FM hyperparameters, the L2 weight, or whether the global bias is trained. The public code is a small demo port, and it drifts from the text in several ways:

FNN's code skips pre-training.
The scripts stop training on the test set instead of a validation split.
The "denoising" autoencoder corrupts nothing.

The card lists each of these differences.

Ask yourself

FNN freezes the field structure into its input: one slot per field. What would break, or improve, if one field could hold several active labels at once, such as a user's interests?
The pooled gain is about two AUC points, while single advertisers show much thinner margins. How would you check whether the deep models simply benefit more from extra data?
SNN pre-trains on the positive unit plus a couple of sampled negatives per field. How is that similar to negative sampling in word embeddings, and how is it different?