ConveRT: Efficient and Accurate Conversational Representations from Transformers

In plain words

A chatbot often has to pick a good reply from a large pile of candidate replies, instead of writing one from scratch. ConveRT is a small neural network built for that job. It turns the conversation so far into a list of numbers, and it turns each candidate reply into a list of numbers too. The reply whose numbers match the conversation best is the one to send.

The model learns this from ordinary Reddit threads. For each real reply, it must score that reply above the replies written for the other messages in the same batch. The authors built it to be cheap: word pieces instead of whole words, one set of weights shared by both sides, and numbers stored at lower precision. In return they report that it trains in about a day on modest hardware and beats earlier sentence encoders at picking replies. A version that also reads up to ten earlier messages does better still. The same numbers can also feed a simple classifier that labels what a user wants.

The results are the authors' own. The paper runs no independent evaluation.

TL;DR

ConveRT is a dual encoder: a context encoder and a response encoder, scored by cosine similarity. It is pretrained on Reddit reply pairs with an in-batch softmax, and it uses subword embeddings, a six-block shared Transformer and quantization-aware training with 8-bit embeddings and 16-bit network weights §2.1 §3 tab. 2. The authors claim state-of-the-art reply-selection scores on Reddit and AmazonQA and a strong DSTC7-Ubuntu score, plus transfer to three intent-classification sets §4 tab. 3 tab. 5 tab. 6. Use it as a blueprint for a small response-selection encoder. Check the Pitfalls first: the paper contradicts itself on hardware, its size figure does not follow from its own bit widths, and several details are not stated.

Problem

Pretrained language-model encoders such as BERT are large and slow to train and run, which the authors say hinders deployment and portability for dialog systems §1. Retrieval-based dialog treats the next turn as a choice among candidate responses, so a pretraining task that matches this framing could produce useful conversational representations §1. Earlier dual encoders used only the immediate context to choose a response §1. The goal stated by the paper is a model that is effective, affordable and quick to train at the same time §2.1.

Method

The paper proposes a dual encoder with shared Transformer parameters and side-specific output networks. The overview is in fig. 1 §2.1. Each side turns a token sequence into a 512-dimensional, L2-normalised vector. The score for a pair is a cosine similarity with a scale factor that is annealed during training §2.1.

Steps (single-context model).

1. Vocabulary. Sample and lowercase 10M Reddit sentences, then build a shared subword vocabulary of 31,476 tokens. Unknown characters are hashed into 1,000 extra OOV buckets. Inputs are split by greedy left-to-right prefix matching §2.1.
2. Embedding and positions. Look up a 512-dimensional subword embedding for each token. Add two learned position tables, M1 with 47 rows and M2 with 11 rows. Position i receives M1 at row i mod 47 plus M2 at row i mod 11. The tables are coprime sizes, so they give 517 distinct position codes §2.1.
3. Shared Transformer, six blocks. Use single-headed attention with 64-dimensional attention projections and a 2,048-dimensional first feed-forward layer. The maximum relative attention distance per layer is 3, 5, 48, 48, 48, 48. Scores for pairs beyond that distance are zeroed. A learned bias that depends only on relative position is added before the softmax. All Transformer parameters are shared between the input and response sides §2.1.
4. Reduction. Two self-attention heads each compute a weight vector over the subword positions. The weights are scaled by the square root of the sequence length, then applied to the sequence. The output is a 1,024-dimensional r_x (or r_y); how the two heads combine into it is not stated. The paper fuses the attention and the reduction into one matrix product to avoid an extra 3-tensor multiplication §2.1 fn. fig. 1.
5. Side-specific feed-forward (FF2). Each side has its own network of three hidden layers of width 1,024, with skip connections, layer normalisation and orthogonal initialisation. Activation is the fast GeLU approximation, GeLU(x) = x·σ(1.702x). A final linear layer maps to 512 dimensions and the output is L2-normalised, giving h_x and h_y §2.1 fig. 1.
6. Score and objective. The score is S(x, y) = cosine(h_x, h_y), multiplied by a scale that rises linearly from 1 to √d over the first 10K training batches §2.1. For a batch of K pairs, the objective is J = Σ_i S(x_i, y_i) − Σ_i log Σ_j exp S(x_i, y_j), where the negatives for x_i are the other K−1 responses in the batch §2.1. This is a per-batch softmax over in-batch negatives, with label smoothing of 0.2 §3.

# Single-context ConveRT forward pass and loss  [§2.1] [§3]
# Input:  contexts, responses: batch of K token sequences
#         step: global training step
# Output: scalar loss to minimise

def convert_loss(contexts, responses, step):
    # 1. Embed subwords and add the two learned position tables  [§2.1]
    #    position i uses M1[i mod 47] + M2[i mod 11]
    ctx_seq = embed(contexts) + positions(contexts)  # (K, n, 512)
    rsp_seq = embed(responses) + positions(responses)

    # 2. Shared six-block Transformer; same weights for both sides  [§2.1]
    ctx_seq = transformer(ctx_seq)
    rsp_seq = transformer(rsp_seq)

    # 3. Square-root-of-N reduction with two attention heads  [§2.1]
    ctx_vec = sqrt_n_reduce(ctx_seq)          # (K, 1024)
    rsp_vec = sqrt_n_reduce(rsp_seq)

    # 4. Side-specific FF2 with 3 hidden layers, then L2 normalisation  [§2.1]
    ctx_enc = l2_normalise(ff2_context(ctx_vec))  # (K, 512)
    rsp_enc = l2_normalise(ff2_response(rsp_vec))

    # 5. Cosine scores scaled by the annealed factor  [§2.1]
    #    the factor rises linearly from 1 to sqrt(d) over 10K batches
    scale = annealed_scale(step)
    scores = scale * (ctx_enc @ rsp_enc.T)    # (K, K)

    # 6. In-batch softmax with label smoothing 0.2  [§2.1] [§3]
    #    diagonal pairs are positives; the rest are in-batch negatives
    targets = smoothed_one_hot(K, positive=0.8)  # (K, K)
    return cross_entropy(scores, targets)

Multi-context variant. Add up to ten earlier messages from the same Reddit thread. They are concatenated from most recent to oldest and treated as one extra input. The immediate context and the extra context are encoded separately, so candidate responses can still be indexed in advance §2.1. Training uses three ranking terms: the immediate context against the response, the extra context against the response, and the average of the two context encodings against the response §2.1 fig. 2. The paper says these terms are combined linearly but does not give the weights §2.1. At evaluation the averaged context encoding is used §3.

# Multi-context objective  [§2.1] [fig. 2]
# Input:  x (immediate context), z (up to 10 earlier messages, newest first)
# Output: scalar loss; the combination weights are not stated in the paper

def multi_context_loss(x, z, responses, step):
    # 1. Concatenate the earlier messages, most recent first  [§2.1]
    #    the separator and truncation order are not stated
    extra = concat_newest_first(z)
    # 2. Encode immediate context, extra context and their mean  [§2.1]
    h_x = encode_context(x)
    h_z = encode_context(extra)
    h_xz = mean(h_x, h_z)
    # 3. Three in-batch ranking losses against the same responses  [§2.1]
    #    equal weights below are an illustration, not from the paper
    return (
        convert_loss_terms(h_x, responses, step)
        + convert_loss_terms(h_z, responses, step)
        + convert_loss_terms(h_xz, responses, step)
    )

Quantization-aware training. The embeddings are stored at 8 bits and the other network weights at 16 bits. Training follows the mixed-precision scheme of Micikevicius et al. §2.1: FP32 shadow copies are kept, and the forward pass uses FP16 casts. Layer normalisation, L2 normalisation and the attention softmax run in FP32 §2.1. The loss is multiplied by 128, and the updates to the FP32 copies are divided by 128 §2.1. The embedding quantisation range is refreshed during training so it covers all learned values, with a margin of 10% of the range or 0.01, whichever is larger §2.1. Quantization also allowed a doubled batch size, which gives more in-batch negatives §2.1.

# Quantization-aware mixed-precision update  [§2.1]
# Input:  fp32_weights (shadow copies), batch, embedding range
# Output: updated fp32_weights and embedding range

def quantized_step(fp32_weights, batch, embedding_range):
    # 1. Cast to FP16 for compute; embeddings sit on an 8-bit grid  [§2.1]
    fp16_weights = cast_fp16(fp32_weights, embedding_range)
    # 2. Forward pass; norms and attention softmax stay FP32  [§2.1]
    loss = forward(batch, fp16_weights)
    # 3. Scale the loss by 128 before backprop  [§2.1]
    grads = backward(128 * loss)
    # 4. Divide the updates by 128 when applying them to the FP32 copies  [§2.1]
    fp32_weights = optimiser_update(fp32_weights, grads / 128)
    # 5. Refresh the 8-bit range so it covers all learned values  [§2.1]
    #    refresh cadence is not stated
    embedding_range = refit_range(fp32_weights, margin=0.1, floor=0.01)
    return fp32_weights, embedding_range

Transfer to intent classification. The 512-dimensional input-side encoding r_x is used as the feature vector §3. The paper reports that r_x beat the later h_x encoding §3. A two-layer feed-forward classifier with dropout is trained on top, while the encoder stays fixed §3.

Hyperparameters

Results

Author-reported. All scores are R100@1 × 100 (the share of cases where the one correct response is ranked first among 100 candidates) unless noted §3.

Reddit and AmazonQA tab. 3. Reddit scores are without fine-tuning; AmazonQA models (PolyAI-dual and ConveRT) are fine-tuned on its training split.

Extra Reddit numbers in the text: using only the immediate-context interaction term gives 65.7, and replacing the extra context with an empty string gives 65.6 §4. On AmazonQA, ConveRT without fine-tuning scores 67.0; with the mapping function used for USE-QA-MAP, 71.6 §4 fn.

DSTC7-Ubuntu tab. 5. Metrics are R100@1 and MRR. GPT and BERT rows are taken from Vig et al. (2019), not re-run; bi-encoder is from Humeau et al. tab. 5.

Intent classification tab. 6. The metric is not named in the table or text; the values are presumably accuracy, which is not stated. Data are three internal sets tab. 1.

Model size tab. 2. Sizes in MB. "After quantization" follows the table's own assumption: 8-bit embeddings, 16-bit network weights, which the paper says was not verified for the public models.

Ablation (single-context, Reddit R100@1, 966K steps each) tab. 4.

Paper vs code

The paper links github.com/PolyAI-LDN/polyai-models in its title §1. That is the authors' organisation. I cloned it at commit 80573b794da2a38f6fb801048434f333f365bf26 (to raw/code/1911.03688). It contains only a README stating that PolyAI has removed the ConveRT models from public availability. It has no model or training code and no LICENSE file, so no code licence can be recorded and no code is cited. A GitHub search of the PolyAI-LDN organisation for "polyai" and "convert" found no other ConveRT code. The footnoted conversational-datasets repository §2.1 is a data source and I did not read it. No code field is set.

The paper states that fine-tuned models are released §3. The README above says the models are no longer public, so that release cannot be used.

Every detail in this card therefore comes from the paper. None could be checked against code.

Pitfalls

• Hardware conflict. Pretraining is described as 12 GPU nodes with one Tesla K80 each §3. Elsewhere it is 12 16 GB T4 GPUs §4. Plan with the compute budget, not the hardware, and treat both claims as unverified.
• Model size does not match the stated bit widths. Using 16M embedding and 13M network parameters tab. 2, the 116 MB pre-quantisation size is 4 bytes per parameter. The stated quantisation (8 bits for embeddings, 16 bits for the rest) gives about 42 MB, not the 59 MB in the table. The same convention reproduces PolyAI-dual's 118 MB, so the gap for ConveRT is unexplained tab. 2.
• Negative count is inconsistent. Section 2.1 first says there are K′ negatives per input, then says all K−1 other batch responses are used §2.1. Implement K−1 in-batch negatives, the reading that matches the objective.
• The scale schedule uses d, which the text never defines. The end value is √d §2.1. The encodings are 512-dimensional §3, so d = 512 is an inference, not a statement.
• The relative-position bias uses an undefined n. The bias is indexed by n − i + j §2.1. n is presumably the sequence length. The paper does not say.
• Relative-attention masking is ambiguous. The text says scores for distant pairs are "zeroed out" §2.1. It is not stated whether this happens before or after the softmax, or whether the mask is applied per head or per layer.
• Position indexing. Whether positions start at 0 or 1 is not stated §2.1.
• Tokenisation. The text says "any subword tokenization algorithm" §2.1. The footnote names a method with specific thresholds §2.1 fn. Implement the footnote settings, but the exact algorithm is only referenced, not described.
• OOV hashing. The hash function is not specified; only that unknown characters map to 1,000 buckets §2.1.
• Quantisation details not stated. The refresh cadence of the embedding range, the optimiser epsilon, and whether the other 16-bit weights include biases or layer-norm parameters are not stated §2.1.
• Multi-context combination. Combination weights, the separator between concatenated messages, and which tokens are truncated at 60 subwords are not stated §2.1 §3. The reference implementation was not published (see Paper vs code), so these cannot be looked up.
• Single-context DSTC7 is far below its Reddit score. It is 38.2 R100@1 on DSTC7 against 68.2 on Reddit tab. 5 tab. 3. The paper does not discuss this gap. Do not assume the single-context model transfers to DSTC7.
• Evaluation negatives. The paper says it uses N = 100 candidates §3. How the 99 distractors are sampled for Reddit and AmazonQA is not stated.
• Baselines. GPT and BERT on DSTC7 come from another paper's report tab. 5. The paper does not say how its other baselines were trained or tuned.
• Intent evaluation. The three datasets are internal and not released §3. The metric is not named tab. 6. The grid-search ranges for layer sizes, dropout and learning rate are not stated, and neither are the BERT fine-tuning hyperparameters §3.
• Fine-tuning context labels. The paper says extra contexts are "prepended with numerical strings 0–9" §3, but not how they are formatted.
• Compute claims are comparisons. The statements that BERT-style models cost thousands of dollars, and that the bi-encoder is "roughly 50×" more expensive, rest on the authors' estimates and on figures from other papers §3 §4.
• Pretraining length. The paper says 18 hours is "typically sufficient" to converge §3. The total number of steps for the main model is not stated.
• Positive-class smoothing. With label smoothing of 0.2, the positive gets 0.8 and the remaining 0.2 is spread evenly over the in-batch negatives §3 fn. At batch size K, each negative therefore gets 0.2 / (K−1) (derived from that statement, not written in the paper).

Lineage

• Builds on: the authors' earlier dual encoder for Reddit response selection, PolyAI-dual, which is the strongest prior model in the comparison §1 tab. 3.
• Compared against: TF-IDF, USE-large and USE-QA (MAP), BERT-large (MAP), the bi-encoder of Humeau et al. tab. 3 tab. 5, and GPT and BERT on DSTC7 tab. 5.
• Superseded by: not stated in this card.