Joint Deep Modeling of Users and Items Using Reviews for Recommendation

In plain words

Recommender systems usually learn what you like only from the star ratings you gave. Many users rate just a handful of things, so there is little to learn from. People often write reviews as well, and those reviews say why they liked or disliked something.

DeepCoNN reads those reviews. It glues together everything one user has written and everything written about one product, and sends each pile of text through its own small text-reading neural network. Each network compresses its pile into a short list of numbers: a profile of the user and a profile of the product. A final layer combines the two profiles, including how each pair of profile entries interacts, and predicts the star rating.

Both networks are trained together, only to get the rating right. The authors report lower error than rating-only and topic-model baselines on restaurant, product and beer reviews. They also report that the gain is largest for users and items with very few reviews, which is exactly where rating-only methods struggle.

TL;DR

DeepCoNN is a two-tower CNN for rating prediction. The user tower reads the concatenation of all reviews by user u, and the item tower reads all reviews of item i. Both use pretrained word2vec embeddings (fine-tuning not stated), 1-D convolution, max-over-time pooling and a fully connected layer. Their outputs are concatenated and scored by a factorization machine (FM). Training uses RMSprop on mini-batches with dropout on the FC layers. Author-reported MSE improves 8.5% (Yelp), 7.6% (Amazon, averaged over 21 categories) and 8.7% (Beer) over the best baseline (CDL). The paper does not say whether the target review is excluded from the input documents, which matters for any reimplementation.

Problem

Collaborative filtering and matrix factorization learn only from the rating matrix, which is very sparse. Users and items with few ratings get poor predictions §1. Earlier review-based models (LDA, CTR, HFT, RMR) use bag-of-words topic models. They throw away word order and lexical semantics, often model text separately from ratings, and scale poorly to new users and items §1 §4. DeepCoNN learns user and item representations from reviews, end to end for rating prediction §2.

Method

The input is a training set of tuples (u, i, r_{ui}, w_{ui}): user, item, rating and review text §2.1. There are two parallel networks, Net_u and Net_i, with the same layer structure but different inputs. A shared FM layer sits on top §2.2 fig. 1. The paper describes Net_u; Net_i is "the same process" applied to item reviews §2.2.

1. Document construction. Merge every review written by user u into a single document d^u_{1:n} of n words. The item document is built the same way from all reviews of item i §2.3.
2. Look-up layer. Map each word to a c-dimensional pretrained vector and stack the vectors in order. Word order is preserved §2.3 eq. 1: V^u_{1:n} = \phi(d^u_1) \oplus \phi(d^u_2) \oplus \dots \oplus \phi(d^u_n)
3. Convolution. Use n_1 kernels K_j \in \mathbb{R}^{c \times t}, each spanning a window of t words, with bias and ReLU §2.4 eq. 2 eq. 3: z_j = f(V^u_{1:n} * K_j + b_j), \qquad f(x) = \max\{0, x\}
4. Max-over-time pooling. Keep the largest value of each feature map. This handles variable-length documents §2.4 eq. 4 eq. 5: o_j = \max\{z_1, z_2, \dots, z_{(n-t+1)}\}, \qquad O = \{o_1, \dots, o_{n_1}\}
5. Fully connected layer. It produces the user latent vector \mathbf{x}_u \in \mathbb{R}^{n_2}. Net_i produces \mathbf{y}_i the same way §2.4 eq. 6: \mathbf{x}_u = f(W \times O + g)
6. Shared layer (factorization machine). Concatenate \hat{\mathbf{z}} = (\mathbf{x}_u, \mathbf{y}_i) and score it with an FM over all first- and second-order interactions §2.5 eq. 7: J = \hat{w}_0 + \sum_{i=1}^{|\hat z|} \hat{w}_i \hat{z}_i + \sum_{i=1}^{|\hat z|} \sum_{j=i+1}^{|\hat z|} \langle \hat{\mathbf{v}}_i, \hat{\mathbf{v}}_j \rangle \hat{z}_i \hat{z}_j, \qquad \langle \hat{\mathbf{v}}_i, \hat{\mathbf{v}}_j \rangle = \sum_{f=1}^{|\hat z|} \hat{v}_{i,f} \hat{v}_{j,f} Here \hat w_0 is a global bias, \hat w_i are per-feature weights, and \hat{\mathbf{v}}_i are factor vectors §2.5.
7. Training. Minimise "Eq. 7" (see Pitfalls) by backpropagation. The paper gives this gradient with respect to \hat z §2.6 eq. 8: \frac{\partial J}{\partial \hat{z}_i} = \hat{w}_i + \sum_{j=i+1}^{|\hat z|} \langle \hat{\mathbf{v}}_i, \hat{\mathbf{v}}_j \rangle \hat{z}_j Optimise with RMSprop over shuffled mini-batches, using the update rule exactly as printed §2.6 eq. 9 eq. 10: r_t \leftarrow 0.9 \left(\frac{\partial J}{\partial \theta}\right)^2 + 0.1\, r_{t-1}, \qquad \theta \leftarrow \theta - \frac{\lambda}{\sqrt{r_t} + \epsilon} \frac{\partial J}{\partial \theta} Apply dropout to the fully connected layers of both networks §2.6.

Algorithm: DeepCoNN forward pass and training  [§2.2–§2.6]
Input: training tuples (u, i, r_ui, w_ui); pretrained word vectors phi (dim c)
Preprocess:
  for each user u: D_u = concat of all reviews written by u
  for each item i: D_i = concat of all reviews written for i
  (whether w_ui of the current pair is removed: not stated)
Forward(u, i):
  1. V_u = [phi(w) for w in D_u]            # n x c, order preserved   [eq. 1]
  2. for each kernel j in 1..n1:
       z_j = ReLU(conv1d(V_u, K_j, window t) + b_j)                    [eq. 2, 3]
       o_j = max over positions of z_j                                 [eq. 4]
  3. x_u = ReLU(W_u · O_u + g_u)   (dropout on this FC layer)          [eq. 6]
  4. repeat 1–3 with Net_i's own parameters on D_i -> y_i
  5. z_hat = concat(x_u, y_i)
  6. pred = FM(z_hat) = w0 + <w, z_hat> + sum_{a<b} <v_a, v_b> z_hat_a z_hat_b   [eq. 7]
Train: RMSprop (lr lambda) on shuffled mini-batches of size 100            [§2.6] [§3.3]
       loss between pred and r_ui: not stated explicitly (MSE is the eval metric [eq. 11])
Output: predicted rating

The paper argues that the model can be updated online and handles new users and items, because a new review document can be pushed through the networks without refitting a factorization §2.7.2. No experiment tests this claim.

Hyperparameters

Baseline settings §3.3: MF and PMF use a grid over latent factors {25, 50, 100, 150, 200} and regularisation {0.001, 0.01, 0.1, 1.0}. MF is fit with ALS §3.2. LDA, CTR and HFT choose topics from {5, 10, 20, 50, 100}, with K=10 for LDA and CTR. CTR and HFT use α=0.1, λ_u=0.02, λ_v=10. CTR and CDL use one precision/confidence value for all observed ratings. CDL is grid-searched over λ_u, λ_v, λ_n, λ_w and L (values not stated). HFT is run on item reviews.

Results

Author-reported test MSE, averaged over 3 runs. Splits are 80/10/10 and the "Improvement" is relative to the best baseline (CDL) §3.4 tab. 3. Amazon is the average over 21 categories; the per-category table is not in the compiled paper. Dataset statistics follow tab. 2:

MSE comparison tab. 3:

Ablations on Yelp, Amazon Music Instruments and Beer, MSE §3.5 tab. 4:

The two towers together beat either single tower. Pretrained embeddings matter most: TF-IDF and random vectors are far worse. The FM beats a plain dot product §3.5. Grouped by number of training reviews, the MSE reduction over MF is positive for every user and item group on Yelp, Beer and Amazon Music Instruments. The reduction is largest for groups with the fewest reviews, and the conclusion names the one-review case as the biggest gain §3.6 §5 fig. 3. The figure shows the per-group values only as plots.

Paper vs code

No official code.

Pitfalls

• Possible target leakage. The user document holds all reviews by u and the item document all reviews of i §2.3. The paper never says whether the review w_{ui} of the pair being predicted is removed, at training or at test time. If test reviews are included, the model sees the text written alongside the rating it must predict. This can inflate the reported MSE gains a lot. For a realistic setup, build documents from training-split reviews only and drop w_{ui} for the pair being scored.
• The loss is not actually written down. Eq. 7 is the FM prediction, but the paper calls it the "cost" and says the network is trained by minimising it §2.5 §2.6. Minimising a prediction makes no sense. The objective is presumably squared error between J and r_{ui}, since MSE is the evaluation metric eq. 11, but the paper does not state it. Regularisation on the FM weights is not stated either.
• The gradient in Eq. 8 is incomplete. For a symmetric FM, the derivative with respect to \hat z_i sums \langle \hat v_i, \hat v_j\rangle \hat z_j over all j \ne i, not only j > i. Use autodiff rather than coding eq. 8.
• FM factor dimension. Eq. 7 sums the inner product over f = 1..|\hat z|, which makes each factor vector as long as the input (100 here). The paper does not say whether this is intended or a notation slip for a smaller factor size k.
• RMSprop weights are reversed from the usual form. Eq. 9 gives 0.9 to the new squared gradient and 0.1 to the running average. The common form does the opposite. Eq. 10 also divides by \sqrt{r_t} + \epsilon with \epsilon unspecified. Treat the printed rule as possibly a typo, and confirm against your validation curve before trusting it.
• Unbounded documents. A user or item document can hold thousands of reviews (Beer averages 72.7 reviews per user tab. 2). Max length, truncation and padding are not stated, and memory grows with document length. You must choose a cap yourself.
• Notation clash in eq. 2 and eq. 4. z_j means the j-th feature map in eq. 2 but indexes positions inside one map in eq. 4. Max pooling is over positions of one kernel's map, giving one scalar per kernel.
• "Same layers" for Net_i §2.2: the towers have separate weights in the description. The paper does not say whether the word-embedding table is shared or fine-tuned.
• Headline percentages. The 8.3% "average improvement" equals the mean of the three per-dataset percentages. The relative gain on the averaged MSE row (1.081 → 0.994) is about 8.0% tab. 3. The conclusion also mislabels them: it gives 8.5% and 7.6% to Yelp and Beer and 8.7% to Amazon §5, whereas tab. 3 gives Yelp 8.5%, Amazon 7.6%, Beer 8.7%.
• Amazon subset. The text describes the full Amazon corpus as over 143.7M reviews §3.1, but tab. 2 lists 34.7M reviews. The selection criterion for the subset is not stated.
• MSE, not RMSE. All tables report MSE eq. 11. §3.6 says "RMS reduction" once, but the figure plots differences in MSE.
• Cold start is only partly addressed. The cold-start analysis (§3.6) covers users and items with few training reviews. A user with zero reviews has an empty document, and the paper does not say how to handle it.

Lineage

• Builds on: matrix factorization for collaborative filtering (Koren et al.); the sentence-CNN layers of Collobert et al.; factorization machines (Rendle) as the coupling layer; pretrained word2vec embeddings (Mikolov et al.) §2.4 §2.5 §3.3.
• Compared against: MF, PMF, LDA-based rating prediction, CTR, HFT (McAuley & Leskovec) and CDL (Wang et al.) §3.2. Related work it contrasts with: RMR, and the RNN-regularised MF of Almahairi et al. §4.
• Superseded by: not known from content in this wiki.