Paper · 2020-07-07 · cs.LG, cs.IR, cs.SI, stat.ML
PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterest
Represents each user by several medoid pins instead of one vector. Ward clustering groups 90 days of pin actions; each cluster keeps a medoid and a decay-weighted importance (decay rate 0.01), and up to 3 medoids query an ANN index. Author-reported: 110% relevance lift over last-pin retrieval offline; +4% homefeed engagement volume in an A/B test.
Hierarchical clusteringNearest-neighbor searchk-means clusteringrecommendationuser-embeddingsclusteringapproximate-nearest-neighbor
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Most recommender systems describe a user with one point in a space where items also live. A person who reads about cooking, sneakers and sci-fi ends up as a single point that sits between those topics and near none of them. The recommendations then drift toward the average of their interests.
PinnerSage keeps several points per user instead. It looks at what the user saved or clicked over the last few months, groups those items by how similar they are, and keeps one real item from each group as that group's stand-in. A stand-in is always an item that exists, not an average, so it cannot land in an empty region between topics. Each group also gets a score that is higher for groups that are large and recent.
When the system needs candidates, it takes a few of the highest-scoring stand-ins and searches a large index for items close to each one. The authors run this at Pinterest, with billions of items, and report gains over single-vector models. The idea is a way to represent a user as a short list of interests rather than one blurred vector.
TL;DR
PinnerSage represents each user with several embeddings, one per cluster of recent actions, instead of one vector. Each embedding is the medoid pin of a Ward cluster of the user's last 90 days, weighted by a decay-based importance. Up to 3 medoids are sampled per request and used to query an approximate nearest-neighbour index. The paper reports offline and A/B gains, and no official code was found. Relevant if you need multi-interest user retrieval over fixed item embeddings.
Problem
Single-embedding user models struggle when one user has several unrelated interests. The authors measure how often the next action falls within cosine 0.8 of the user representation, and a single-vector baseline trails an oracle that picks one of the user's past pins §3 tab. 1. They list four open questions: how many embeddings a user needs, how to run inference and updates for hundreds of millions of users, how to choose embeddings for a request, and whether gains hold in production §1. They also assume pin embeddings are fixed, produced by a separate model (PinSage), so user vectors live in the same space as items §3 §2.
Method
The method has three per-user steps, then a serving layer. Pin embeddings
P_i come from PinSage and are not learned jointly §3 §2 Design Choice 1.
Each user's actions are the pins they repinned or clicked, in time order
§3.
- Cluster the user's action pins from the last 90 days with Ward agglomerative clustering, using a merge threshold alpha. The number of clusters is set by the data, not fixed in advance §3.1.1 alg. 1 eq. 1.
- For each cluster, use its medoid as the representative: the member with the smallest sum of squared distances to the other members. Only the medoid's pin id needs storing; its embedding is fetched from a key-value store §3.1.2 eq. 2.
- For each cluster, compute an importance: a sum over its actions of an exponential decay in age §3.1.3 eq. 3.
The per-user procedure is summarised in alg. 2. The Ward procedure is in alg. 1.
# Per-user PinnerSage pass [alg. 2]
# Input: pins (m, d) for the user's actions; times (m,); alpha, lam
# Output: one (medoid index, importance) pair per cluster
def pinnersage(pins, times, alpha, lam, now):
# Group the action pins into clusters with Ward [alg. 2] [§3.1.1]
clusters = ward(pins, alpha)
results = []
for cluster in clusters:
# Medoid: member with the least total squared distance [eq. 2] [§3.1.2]
medoid = min(
cluster,
key=lambda m: sum(
squared_distance(pins[m], pins[j]) for j in cluster
),
)
# Time-decayed count of the cluster's actions [eq. 3] [§3.1.3]
importance = sum(
exp(-lam * (now - times[i])) for i in cluster
)
results.append((medoid, importance))
return results# Ward clustering with merge threshold alpha [alg. 1] [§3.1.1]
# Input: pins (m, d); alpha (float)
# Output: list of clusters (sets of pin indices)
def ward(pins, alpha):
# Start from squared Euclidean distances between all pin pairs [alg. 1]
dists = pairwise_squared_distance(pins) # (m, m)
# Agglomerate with a stack-based nearest-neighbour chain; distances
# are updated with the Lance-Williams rule [eq. 1] [alg. 1]
merges = agglomerate(dists) # (distance, set_a, set_b)
# Order the merges from the largest distance downwards [alg. 1]
merges = sorted(merges, key=lambda t: t[0], reverse=True)
clusters = []
for distance, set_a, set_b in merges:
# Accept a merge only if it is within alpha and does not reuse
# pins already in an accepted cluster [alg. 1]
union = set_a | set_b
if distance <= alpha and all(union.isdisjoint(c) for c in clusters):
clusters.append(union)
return clustersThe Lance-Williams update for Ward distances, with n the cluster sizes:
d(C_i \cup C_j, C_k) = \frac{(n_i + n_k)\, d_{ik} + (n_j + n_k)\, d_{jk} - n_k\, d_{ij}}{n_i + n_j + n_k} eq. 1
The medoid and the importance:
\mathrm{embedding}(C) = P_m, \quad m = \arg\min_{m \in C} \sum_{j \in C} \lVert P_m - P_j \rVert_2^2 eq. 2
\mathrm{Importance}(C, \lambda) = \sum_{i \in C} e^{-\lambda (T_{now} - T[i])} eq. 3
Serving §4 §2. Two paths produce the medoids and importances. A daily batch job runs over the user's last 90 days on a MapReduce-style cluster and writes the output to a key-value store. An online version runs on the most recent 20 actions of the latest day and updates clusters starting from the stored batch state §4.2. The two outputs are combined before use §3.1. At request time, up to 3 medoids are sampled, with probability related to their importance, and each one queries an approximate nearest-neighbour index over pin embeddings §4 §2 Design Choice 4. The index is HNSW, chosen over LSH Orthoplex and product quantisation on cost, latency and recall §4.1.1. Retrieved candidates are refined: near-duplicates and low-quality pins are filtered with separate in-house models §4.1. Medoid pin ids are used as cache keys, so a cache can be shared across users §4.1.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| action window, batch | last 90 days | §3.1.1 §4.2 §5.3 | §2 Design Choice 5 says "60-90 day" activities; the method and serving text use 90 |
| action window, online | most recent 20 actions, latest day | §4.2 | how it is merged into the batch output is not stated |
| Ward merge threshold alpha | not stated | alg. 1 | an input only; no value appears in the text |
| Ward distance | squared Euclidean on pin embeddings | alg. 1 | |
| importance decay lambda | 0.01 | §3.1.3 | 0 and 0.1 are ablated in §5.3.1; time unit not stated |
| medoids used per request | at most 3 | §4 §2 Design Choice 4 | sampling scheme (with or without replacement, fewer clusters) not stated |
| clusters per user | tens to a few hundreds; light users 3-5, heavy 75-100 | §3.1.3 §2 Design Choice 2 | |
| pin embedding dimension d | not stated | fig. 2 | fig. 2 caption shows 256-dimensional pins in a 2-D illustration only |
| relevance cosine threshold | 0.8 | §3 §5.3.1 | used for the oracle task and the relevance metric |
| candidates per embedding | floor(400 / e), set size at most 400 | §5.3.1 | e is the number of embeddings; e used for PinnerSage in the main table is not stated |
| ranking impressions | 20 per action | §5.3.2 | |
| offline evaluation | tens of millions of users, 90 days of activity, daily chronological test batches | §5.3 | |
| ANN index | HNSW | §4.1.1 | index build parameters not stated |
| k in motivating oracle | k=3 for the k-means oracle | §3 tab. 1 | |
| k for k-means baseline | k=5 | §5.3.1 §5.3.2 |
Results
Author-reported. Setup: offline experiments on tens of millions of sampled users with their activity from the last 90 days, compared against single-vector and sequence baselines, with hardware not stated §5.3. Online results come from randomised A/B tests against a decay-average single-embedding control; test duration and user counts are not stated §5.2.
Headline results:
| Setting | Metric | Baseline | This paper | Source |
|---|---|---|---|---|
| candidate retrieval, offline | relevance / recall lift | last pin 0% / 0%; HierTCN 31% / 16% | Ward + medoid, lambda 0.01: 110% / 88% | tab. 4 |
| candidate ranking, offline | R-precision / reciprocal-rank lift | last pin 0% / 0%; HierTCN 21% / 16% | Ward + medoid, lambda 0.01: 37% / 28% | tab. 5 |
| Homefeed A/B | volume / propensity | decay-average control | +4% / +2% | tab. 3 |
| Shopping A/B | volume / propensity | decay-average control | +20% / +8% | tab. 3 |
Motivating oracle experiment: accuracy lift for predicting the next action, relative to the last-pin model tab. 1. Last pin 0%, decay average 25%, k-means oracle with k=3 98%, oracle with future knowledge 140%. The oracle rows bound what past-engagement methods could achieve; they are not methods.
Full retrieval table tab. 4, lift relative to the last-pin model:
| Model | Relevance | Recall |
|---|---|---|
| Last pin model | 0% | 0% |
| Decay average, lambda 0.01 | 28% | 14% |
| Sequence model (HierTCN) | 31% | 16% |
| PinnerSage, 1 sampled embedding | 33% | 18% |
| PinnerSage, k-means (k=5) | 91% | 68% |
| PinnerSage, complete linkage | 88% | 65% |
| PinnerSage, centroid embedding | 105% | 81% |
| PinnerSage, HierTCN embedding | 110% | 88% |
| PinnerSage, importance lambda 0 | 97% | 72% |
| PinnerSage, importance lambda 0.1 | 94% | 69% |
| PinnerSage, Ward + medoid, lambda 0.01 | 110% | 88% |
Full ranking table tab. 5, lift relative to the last-pin model:
| Model | R-precision | Reciprocal rank |
|---|---|---|
| Last pin model | 0% | 0% |
| Decay average, lambda 0.01 | 8% | 7% |
| Sequence model (HierTCN) | 21% | 16% |
| PinnerSage, 1 sampled embedding | 24% | 18% |
| PinnerSage, k-means (k=5) | 32% | 24% |
| PinnerSage, complete linkage | 29% | 22% |
| PinnerSage, centroid embedding | 37% | 28% |
| PinnerSage, HierTCN embedding | 37% | 28% |
| PinnerSage, importance lambda 0 | 31% | 24% |
| PinnerSage, importance lambda 0.1 | 30% | 24% |
| PinnerSage, Ward + medoid, lambda 0.01 | 37% | 28% |
A/B table tab. 3: Homefeed +4% volume, +2% propensity; Shopping +20% volume, +8% propensity. Volume counts repins and clicks; propensity counts them per user §5.2.
Cost table tab. 2, described as a rough estimate of cost reduction during implementation §4: LSH Orthoplex to HNSW -60%; full index to index refinement -50%; cluster centroid to medoid -75%.
The paper also reports a qualitative visualisation of clusters and sample recommendations for one user fig. 5 fig. 6, and a diversity-relevance trade-off in which e=3 is called a sweet spot fig. 7 §5.3.3.
Paper vs code
No official code. The paper and arXiv metadata link no repository or project page; the only URLs are footnotes to benchmark blog posts §4.1 §3.1.1. A GitHub repository search for "pinnersage" returned one repository, a third-party reimplementation of a different project, which was excluded. A search for "pinnersage" within the pinterest organisation returned no results.
Pitfalls
- Pins outside every accepted merge. Algorithm 1 returns only the merged clusters, so as written a pin that never enters an accepted merge appears in no output cluster. The text does not say whether such pins become singleton clusters or are dropped alg. 1 §3.1.1.
- The threshold alpha is not given. Every Ward result depends on it, but no value is stated in the text. Tune it on your own data and report the value you used alg. 1.
- Selection rule is not a plain dendrogram cut. Algorithm 1 sorts merges by distance, then accepts those within alpha that do not overlap accepted clusters. Implement that rule as written alg. 1.
- Complexity is stated ambiguously. The text says the cost is O(m^2) with m = |A|^2 §3.1.1. Read literally, that is O(|A|^4) in the number of actions, and the pairwise distance matrix also grows with |A|^2 [alg. 1, line that initialises d_ij]. Budget by measurement, not by the stated bound.
- The medoid is the member closest to the centroid, in effect. Expanding eq. 2 gives the sum of squared distances as a constant plus n times the squared distance to the cluster mean, so the argmin is the member nearest to the centroid eq. 2. This is a derivation from the equation, not a claim the paper makes; the paper argues for medoids by their robustness and their being real pins §2 Design Choice 3 §3.1.2.
- The lambda time unit is not stated. Equation 3 uses T_now minus action time without units. If time is in days, which the paper's age definition in fig. 3 suggests, a 90-day-old action weighs e^(-0.9), about 0.41, at lambda 0.01. Confirm the unit before you port the decay.
- Importance is unnormalised. A cluster's score grows with its number of actions, so large clusters can dominate the sampling. The lambda 0 ablation shows that counting alone performs worse §5.3.1.
- Sampling of medoids is under-specified. The paper says up to 3 medoids are sampled by importance, but not how: with or without replacement, and what happens with fewer clusters §4 §2 Design Choice 4.
- Batch and online outputs are merged without a stated rule. The text says the two are combined, and the online version starts from stored batch state, but the merge rule and how new pins join existing clusters are not spelled out §3.1 §4.2.
- Candidate budget breaks above 400 embeddings. The per-embedding budget floor(400 / e) is zero when e exceeds 400. The e used for the PinnerSage rows in tab. 4 and tab. 5 is not stated §5.3.1.
- The relevance metric depends on the embedding space. Relevance and recall use a cosine threshold of 0.8 in PinSage space, which this paper does not describe. The threshold is not justified in the text §3 §5.3.1.
- "Significant" lacks statistics. The A/B section calls the gains significant but gives no test, no sample sizes and no duration §5.2 tab. 3.
- Cost figures are relative and rough. The percentages in tab. 2 are described as a rough estimate, with no absolute numbers §4.
- Internal inconsistency on history length. Design Choice 5 says past 60 to 90 days, while the method and serving text use 90 days §2 Design Choice 5 §3.1.1 §4.2.
- Oracle numbers are upper bounds, not methods. The 140% oracle uses future actions to choose the representation, so it does not show what is achievable without them §3 tab. 1.
Lineage
- Builds on: PinSage, the pin embedding model the paper treats as a fixed black box (Ying et al., KDD, cited as [pinsage]; no arXiv id in this wiki) §3.
- Compared against: last-pin, decay-average, LSTM, GRU and HierTCN single-embedding models §5.3; k-means and complete-linkage clustering variants §5.3.1. HierTCN is the authors' earlier work [htcn], cited as a baseline §5.3.
- Superseded by: not stated in this paper; nothing in the wiki yet.
Source and license
Adapted from “PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterest” by Aditya Pal, Chantat Eksombatchai, Yitong Zhou, Bo Zhao, Charles Rosenberg, Jure Leskovec (arXiv:2007.03634), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.