Ad Headline Generation using Self-Critical Masked Language Model

In plain words

An online shop lets sellers advertise a group of related products under one short headline. Writing those headlines by hand takes a lot of effort across a catalogue of millions of products, and template headlines are dull. This paper trains a language model to write the headline from the titles of the products in the campaign, so one headline can cover several items.

The model is first trained to fill in hidden words in past headlines, the way a reader guesses a missing word. It is then tuned with a reinforcement-learning step: it writes a headline by sampling words, the same headline is also written greedily, and words are pushed toward the sampled version only when it scores better on word overlap with a good headline. The authors report that this beats an older recurrent model and human-submitted headlines on their own audits. Those results come from the authors' own internal data and crowd-sourced ratings.

TL;DR

• Generates one ad headline for several products at once, using a BERT-Large masked language model whose hidden positions are only in the headline §3.1 §3.2.
• Fine-tunes with self-critical policy gradient: a sampled headline is compared with a greedy one, and the reward is ROUGE-L F1 against approved headlines §3.3 eq. 9 §4.1.
• Author-reported: on overlap, the full model's ROUGE-L is 6.33 above the bi-LSTM baseline (5.08 for the model without the RL step) tab. 1; in a crowd audit, 6.53% more headlines are rated 3 of 3 than human-submitted ones tab. 2.
• Data is proprietary: over 500,000 Amazon ad campaigns, English only, with an 85/5/10 split §4.1. No code is released.

Problem

Advertisers group related products into one campaign, and the campaign needs a single headline that describes all of them. Writing headlines for millions of products takes a lot of effort, and template headlines (for example "Save Now on ...") lack variety and brand identity §1.

Earlier ad-headline work uses LSTM pointer networks trained with reinforcement learning §2 §4.2. The authors argue that these do not use the extensive pre-training of Transformer models, and that log-likelihood training does not match headline-quality metrics such as BLEU and ROUGE §2 §3.3. The paper says its method changes only the training procedure and not inference latency §6.

Method

The method has three parts: a multi-product input encoding on a BERT-Large masked language model, a masked-attention decoder for left-to-right generation, and a self-critical fine-tuning stage that uses ROUGE-L as reward. Inference uses beam search with a length-normalised score §3.4.

Input encoding §3.2 fig. 2. Each product title and the headline are split into sub-word tokens. For one product, an end token closes the title and another closes the headline, and a start token opens the sequence. For several products, the titles are concatenated with a product separator; the exact multi-product layout is shown only in the figure. The separator replaces an entry that is unused in pre-training, during multi-product fine-tuning §3.2. Only the headline tokens are hidden, so the source titles are never masked in the main model §3.2. An optional category embedding averages the sub-word embeddings of each product's category label and adds them to its title tokens §3.2.

Masked-language-model loss §3.1 eq. 2. Standard BERT attention is written as a softmax over scaled query-key products applied to values eq. 1. The masked-token cross-entropy loss sums log-probabilities over the hidden positions, conditioned on all visible tokens eq. 2.

Left-to-right generation §3.3 eq. 3. The hidden headline positions cannot see future headline tokens during generation, so the authors add a mask to the attention scores. The mask is 0 where attention is allowed and minus infinity where it is not eq. 3 fig. 3.

Self-critical fine-tuning §3.3 eq. 4–eq. 9.

• The reward for a headline is a function r, and the objective minimises the negative expected reward eq. 4.
• The REINFORCE gradient is written with the sum of masked-token log-probabilities P as the score function eq. 5 eq. 6.
• A constant baseline b reduces variance without changing the expected gradient eq. 7. Instead of a learned baseline, the authors use a greedy headline from the same model as the baseline. The gradient is approximated with one sampled headline x̂ and the greedy headline ẑ eq. 8 eq. 9.

# Self-critical update for one campaign  [§3.3] [eq. 9]
# Input:  product titles, approved headline, model, reward r
# Output: scalar loss

def sc_mlm_loss(model, titles, headline, reward):
    # Pack titles with product separators, then the headline  [§3.2]
    inputs = pack_sequence(titles, headline)
    # Hide headline tokens; the masking rate is not stated  [§3.1]
    masked = random_mask(headline)
    # Sample one token per hidden headline position  [§3.3]
    sampled = sample_tokens(model, inputs, masked)
    # Greedy headline: the most probable token at each step  [§3.3]
    greedy = argmax_tokens(model, inputs, masked)
    # Reward: ROUGE-L F1 against the approved headline  [§4.1]
    advantage = reward(sampled) - reward(greedy)
    # Log-probability of the sampled tokens at the hidden positions  [eq. 6]
    log_prob = masked_log_prob(model, inputs, masked, sampled)
    # Minimising this raises the probability of samples that beat greedy
    # [eq. 9]
    return -advantage * log_prob

Written as an equation, the update is:

\nabla_\theta \mathcal{L}_{SC\_MLM} \approx -\,(r(\hat{x}^h) - r(\hat{z}^h))\,\nabla_\theta P eq. 9

Inference §3.4 eq. 10. Headlines are decoded with beam search until a predetermined maximum length or until every beam emits the end token. The authors replace the usual log-probability score with a length-normalised one that scales each word's log-likelihood by a factor depending on its position. A regular-expression step then removes extra spaces around symbols such as -, + and parentheses §3.4.

# Length-normalised word score for beam ranking  [eq. 10] [§3.4]
# Input:  word log-likelihood, word position i (index base not stated)
# Output: normalised word score

def normalised_score(log_likelihood, i, alpha):
    # alpha is the length-normalisation coefficient; its value is not stated
    scale = (2 + 1) ** alpha / (2 + i) ** alpha
    return log_likelihood * scale

Hyperparameters

Values not given in the paper are marked "not stated". No values are taken from code, because no code is released.

Results

Author-reported, on proprietary Amazon data. Setup: BERT-Large base unless stated, bi-LSTM pointer-network baseline with self-critical training §4.2.

Overlap with approved headlines tab. 1. The paper reports differences from the bi-LSTM baseline, not absolute scores. The caption says the first two columns are F1 differences for ROUGE-L and BLEU-4; the units of the other columns are not stated. The bi-LSTM row is the reference and has no numbers.

Source: tab. 1. "Avg. cos. sim." is measured with an independently pre-trained language model; which model is not stated.

Quality audit against human headlines tab. 2. Crowd raters judged headlines on a 3-point scale: 1 incorrect or irrelevant, 2 correct, 3 correct and attractive. Each headline had three raters and the mode was used. The table gives relative change against human-submitted headlines, on a random blind test set of about 5,000 headlines tab. 2.

Source: tab. 2. The audit covers over 150,000 judgments in total §5.2.

Grammar audit §5.2. On about 10,000 headlines, graders judged 98.13% of SC-MLM headlines and 98.12% of MLM headlines grammatical, against 93.14% of human-submitted headlines.

Latency §5.2. Single-product generation gives about 8% faster inference than the multi-product model, but the authors say it produces less attractive headlines.

Paper vs code

No official code. The paper and arXiv listing link no code or project page. The arXiv comments field names only the venue (NAACL-HLT 2021 Industry Track). Searches: GitHub repositories for "self-critical masked language model headline" (no results); "ad headline generation" (two unrelated third-party repositories, not used); "headline generation masked language model" inside the amazon-science organisation (no results).

Pitfalls

• The training reward is the headline metric. The RL step maximises ROUGE-L F1 against approved headlines §4.1, and the top overlap row reports ROUGE-L tab. 1. Part of the overlap gain comes from optimising the metric itself. The quality audits tab. 2 are the independent check.
• Likelihood term unclear. The text says the loss combines likelihood and reward §3.3, but eq. 9 contains only the reward-weighted term. Whether the cross-entropy loss is also added during stage 2 is not stated.
• Greedy baseline versus beam search. The baseline uses "the token with the maximum probability at each step" §3.3, but inference uses beam search §3.4. Whether the baseline is a per-position argmax or a sequential greedy decode is not stated.
• Attention during training. Masked attention eq. 3 is introduced for generation §3.3, while the base model's attention is written as ordinary self-attention eq. 1 §3.1. The paper does not say which attention pattern the fine-tuning uses.
• Masking rate not stated. The fraction of headline tokens hidden during training, and whether it changes across epochs, are not given §3.1.
• Scale of the log-probability sum. Eq. 6 sums log-probabilities over the hidden positions, so its magnitude grows with the number of hidden tokens. No normalisation is stated eq. 6.
• Length normalisation. The value of α is not stated, and the constant 2 in eq. 10 is not explained. The factor shrinks for later positions when α is positive, so later words are scaled down more. Whether the factor is applied per word or to the beam total is not clear from the equation alone eq. 10.
• Table 1 is not absolute. The values are differences from the bi-LSTM baseline tab. 1. Compare rows with each other, not with published numbers for other systems.
• Small gaps in the audits. SC-MLM and MLM differ by 0.92 percentage points in mean rating (2.07% against 1.15%) and by 5.31 percentage points in 3-of-3 ratings tab. 2. No variance or significance test is reported.
• Epoch-matched, not compute-matched. The MLM-only baseline gets up to 30 epochs, and the SC-MLM model gets 15 MLM epochs plus 15 self-critical epochs §4.1 §5.2. The paper does not report the cost of the second stage on its own.
• Data is not released. The data is proprietary Amazon campaign data, and whether any split, headline set or model is released is not stated. Training uses campaigns that internal experts checked against ad policy §4.1, which an outside replication would have to approximate.
• Training cost. One 30-epoch fine-tuning run took over 20 days on a single V100, with an estimated 150 kWh §4.1.
• Keyword stuffing. The authors note that keyword-stuffed source titles hurt headline quality, and that beam-search scoring filters some of them §5.2.
• Shared first stage. The two model variants share their first 15 epochs, so some generated headlines are identical §5.2.

Lineage

• Builds on: BERT (Devlin et al., 2019) for the masked-language-model base and its pre-training data §3.1 §4.1; a masked-attention design for generation from Dong et al. (2019) §3.3; self-critical sequence training from Rennie et al. (2017) for the greedy baseline §3.3.
• Compared against: a pointer-network bi-LSTM with self-critical training, following Xu et al. (2019) and Hughes et al. (2019) §4.2.
• Related reward-based text generation: Ranzato et al. (2016), cited for treating generation as reinforcement learning §2.
• Superseded by: not known from this wiki.