Paper of the day, 2026-10-30

ConveRT: Efficient and Accurate Conversational Representations from Transformers

Listen to the episode, or read on. The paper: arXiv:1911.03688.

ConveRT: the chatbot that picks its replies instead of writing them

A lot of agent plumbing these days comes down to one question: which of these stored texts fits the one in front of me? ConveRT answered that for chat back in 2019. Its model is small, trains in about a day for about $85 by the authors' estimate, and on their Reddit benchmark it beat encoders many times its size. If you are building retrieval on a budget, its tricks still pay off.

The problem in one picture

Drawing made for wikillm on 2026-10-30 by the AI editor. It is not taken from the paper.

How it works

Picture a help-desk bot with a drawer full of good, pre-written answers. It never has to write anything. It only has to choose. ConveRT treats the next turn of a conversation as exactly that choice. One encoder turns the conversation into a vector, another turns each candidate reply into a vector, and the reply whose vector points most nearly the same way as the conversation's wins. Both vectors are 512-dimensional and normalised to unit length, so the score is a plain cosine similarity.

The two encoders aren't really separate. They share one six-block Transformer and only part ways at the end, where each side gets its own small feed-forward network. That split is the practical win. You can encode every reply in the drawer once, ahead of time, and at serving time you only encode the conversation. This is the same move as dense retrieval, applied to dialogue.

The training signal costs nothing to collect. The authors take Reddit comment threads, 727M input and reply pairs in all, and treat each real reply as the right answer for its message. The wrong answers come free as well: in a batch of 512 pairs, each message is scored against all 512 replies, and the other 511 serve as distractors. The loss is a softmax over that row of scores, with label smoothing of 0.2, so the true reply aims for 0.8 rather than full certainty. The internet's least tidy comment section, it turns out, makes a very tidy classroom.

A cosine is stuck between -1 and 1, and a softmax over numbers that close together is nearly flat. ConveRT therefore multiplies the scores by a factor that rises linearly from 1 to the square root of d over the first 10K batches. The card points out that the paper never defines d. Taking d = 512 is a reasonable guess, but the paper doesn't say so.

The cost-cutting is where the paper has the most fun:

Subwords instead of words. There are 31,476 subwords, plus 1,000 hashed buckets for characters the vocabulary has never seen.
Two position tables of coprime size, with 47 and 11 rows. Position i adds row i mod 47 to row i mod 11, which gives 517 distinct position codes from only 58 learned rows. It's a clever trick for so little storage.
Attention that starts local. The first two layers look at most 3 and 5 tokens away, and the later layers 48.
Lower precision. Embeddings are stored at 8 bits and the other weights at 16, trained with full-precision shadow copies and a loss scale of 128. You can read more about this under quantization-aware training.

A multi-context version also reads up to ten earlier messages, encodes them separately, and averages that encoding with the immediate context's. The paper combines three ranking terms for it but does not state how they are weighted.

Try it yourself

The exercise: see why the annealed scale matters. The code builds a fake batch of 512 contexts with replies that are noisy copies of them. It computes the in-batch softmax with label smoothing 0.2 and ramps the scale from 1 to √512 over 10K batches.

Python

import numpy as np
import matplotlib.pyplot as plt
rng = np.random.default_rng(0)
K, d = 512, 512        # batch size and encoding size from the card
noise = 3.0            # how far each true reply drifts from its context
unit = lambda v: v / np.linalg.norm(v, axis=1, keepdims=True)
ctx = unit(rng.normal(size=(K, d)))
rsp = unit(ctx + noise * rng.normal(size=(K, d)) / np.sqrt(d))
cos = ctx @ rsp.T      # row i: context i against every reply in the batch
target = np.full((K, K), 0.2 / (K - 1))   # label smoothing 0.2
np.fill_diagonal(target, 0.8)
steps = np.arange(0, 15001, 250)
scale = 1 + (np.sqrt(d) - 1) * np.minimum(steps / 10_000, 1)
loss, p_true = [], []
for s in scale:
    z = s * cos
    logp = z - z.max(1, keepdims=True)
    logp -= np.log(np.exp(logp).sum(1, keepdims=True))
    loss.append(-(target * logp).sum(1).mean())
    p_true.append(np.exp(np.diag(logp)).mean())
print("in-batch top-1 accuracy:", (cos.argmax(1) == np.arange(K)).mean())
print(f"scale 1    -> loss {loss[0]:.2f}, p(true reply) {p_true[0]:.4f}")
print(f"scale {scale[-1]:.1f} -> loss {loss[-1]:.2f}, p(true reply) {p_true[-1]:.4f}")
fig, ax = plt.subplots(1, 2, figsize=(9, 3.5))
ax[0].plot(steps, scale); ax[0].set(title="score scale", xlabel="batch")
ax[1].plot(steps, p_true, color="C1"); ax[1].set(title="mean p(true reply)", xlabel="batch")
plt.tight_layout(); plt.show()

What you should see. Top-1 accuracy is the same at every scale, because multiplying all scores by one number doesn't change which is largest. What changes is confidence. At scale 1 the true reply gets barely more probability than a random distractor. As the scale climbs, its probability rises steeply and then levels off once the ramp ends. The scale changes the size of the gradients, not the ranking.

Explore. Raise noise until top-1 accuracy starts to drop. Then shrink K and watch the task get easier, because there are fewer distractors. This is why the authors liked that quantization let them double the batch.

What the authors report, and what they leave open

All numbers are the authors' own, measured as R100@1: the share of cases where the one correct reply ranks first among 100 candidates.

Reddit, no fine-tuning: 68.2 for single-context ConveRT and 71.8 for multi-context, against 61.3 for the authors' earlier PolyAI-dual and 24.0 for BERT-large with a mapping layer.
AmazonQA, fine-tuned: 84.3, against 71.3 for PolyAI-dual. Without fine-tuning, ConveRT scores 67.0.
DSTC7-Ubuntu: 71.2 for multi-context, against 70.9 for the Humeau et al. bi-encoder. The GPT and BERT rows in that table come from another paper and were not re-run.
Intent classification with the encoder frozen: 92.7, 94.5 and 64.3 on three internal datasets, close to or above fine-tuned BERT-large. The metric is not named.
Size: 116 MB, or 59 MB after quantization, against 1341 MB for BERT-large.

The ablations are modest. No single change costs more than 0.5 points, and removing five tricks together costs 1.6.

The card is blunt about the loose ends:

Hardware. One section says K80s and another says T4s.
Size. The 59 MB figure does not follow from the stated bit widths. The card's own arithmetic gives about 42 MB.
DSTC7. Single-context ConveRT scores only 38.2 there, a gap the paper never discusses.
Missing details. The weights for the multi-context terms, how the 99 evaluation distractors are sampled, and how long the main model trains are all not stated.
No code. The models were later withdrawn and no code was released, so none of these details can be checked.

Ask yourself

In-batch negatives are cheap but random. What happens to ConveRT-style training when your batch is full of near-duplicate replies, as in a narrow customer-support domain?
The coprime position tables give 517 codes, yet inputs are capped at 60 subwords. What is the extra range for, and would you keep the trick?
Single-context ConveRT drops from 68.2 on Reddit to 38.2 on DSTC7-Ubuntu. Before you trust any reply ranker in a new domain, what would you test first?