Controllable text generation

Overview

The goal is to make a language model's output satisfy a condition that is set at generation time, such as a syntax tree, a required keyword, a span, a length or the gender of a speaker 2205.14217 §1 2202.11705 §1 1909.05088 §4.1. Most methods here score the output with a function that measures the wanted property, and move the output toward high scores. Left-to-right models find conditions on the whole output hard, because the end of the text is not yet written when the start is chosen 2205.14217 §1 2202.11705 §1.

For a map of the whole field before choosing a method, start with the survey of controllable generation with pre-trained transformers. It sorts methods by where the control signal enters the model: fine-tuning, retraining or refactoring, and decoding-time post-processing 2201.05337 §3. It lists seven task families 2201.05337 §2.2. It has no experiments section of its own, and its evaluation part only reviews metrics 2201.05337 §4, so no number in this page comes from it.

The seven method cards take seven routes:

• Steer a non-autoregressive diffusion model with classifier gradients on its continuous latents 2205.14217 §5.1.
• Keep a left-to-right model frozen, and sample a whole draft with an energy built from constraint terms, using Langevin dynamics on soft logits 2202.11705 §3.1.
• Keep a left-to-right model, and run a toxicity classifier inside beam search, pushing the output away from the prompt's label. The purpose is to produce hard test data, not user-facing text 2203.09509 §3.2.
• Train a model to emit reflection tokens, then weight their scores in a beam at inference 2310.11511 §3.3 2310.11511 app. A.3.
• Keep a left-to-right model, and drop next-token candidates whose extended text falls below a threshold on a moral direction taken from sentence embeddings, keeping a minimum number of candidates 2103.11790 §Guiding Generative Language Models using the MoralDirection.
• Put the control value into the input as a tag, and train the model with it 1909.05088 §4.1.
• Label pre-training documents with a control token for their toxicity level, and condition the model on a token at inference 2305.10403 §3 2305.10403 §5.1.

The first five steer a single output at generation time, although the reflection-token route relies on tokens learned in training, and the moral-direction route needs a direction estimated beforehand. The last two fix the condition in training, then supply a value with each input.

Coverage is seven method cards and one survey card. Other control methods are named below only as the cards describe them. All results are author-reported. The papers use different models, tasks and metrics, so their numbers should not be compared directly.

Variants

Latent steering of a diffusion LM (Diffusion-LM)

The model denoises a sequence of continuous word vectors. At each diffusion step, after the usual reverse step, three Adagrad updates move the vectors to lower a loss made of the classifier term and the fluency term 2205.14217 §5.1. The control run uses 200 diffusion steps, reduced from 2000 to pay for the extra updates 2205.14217 §5.1 2205.14217 §6.1. Card: Diffusion-LM Improves Controllable Text Generation.

Reported results: on classifier-guided control the method scores 81.2 on semantic content against 69.9 for FUDGE, and 90.0 on parts of speech against 27.0 2205.14217 tab. 2. On syntax-tree and span control it scores 86.0 and 93.8, against 17.9 and 54.2 for FUDGE 2205.14217 tab. 2. Composed controls are mixed. Semantic content with syntax reaches 69.8 content control and 74.8 syntax control, against 61.7 and 15.4 for FUDGE. Semantic content with parts of speech reaches 69.1 on parts of speech, against 24.1, but its content control is 63.7 against 64.5 for FUDGE 2205.14217 §7.2 2205.14217 tab. 4.

Energy-based decoding with Langevin dynamics (COLD)

Each requirement is written as a score for a draft. The scores are combined with weights into an energy, where lower is better 2202.11705 eq. 1 2202.11705 §3.1. The draft is a grid of soft logits, one row per position. It is updated by Langevin steps: a gradient step on the energy plus Gaussian noise whose scale falls over the run 2202.11705 eq. 2 2202.11705 §3.4. The soft draft is turned into words left to right, by taking from the model's top-k next tokens the one with the largest soft logit 2202.11705 eq. 6 2202.11705 §3.3. The requirements are fluency in both directions, a prediction of fixed right-hand tokens, and an n-gram match to a reference 2202.11705 §3.2. The base model is GPT2-XL, frozen, with no fine-tuning 2202.11705 §4. Card: COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics.

Reported results, on three tasks:

• Abductive reasoning: the best overall coherence, 2.96 against 2.87 for DeLorean, while DeLorean is higher on left coherence (4.23 against 4.00) 2202.11705 tab. 1. Raters agree only weakly, with a Krippendorff alpha of 0.36 2202.11705 §4.1.2.
• Counterfactual rewriting: min-edit overlap 56.84 against 52.90 for DeLorean, and human coherence 2.12 against 1.92 2202.11705 tab. 2. Mix-and-Match reaches 85.07 overlap, but its coherence score is lower 2202.11705 tab. 2.
• Lexically constrained generation: keyword coverage of 94.50% against 91.00% for NeuroLogic, with higher perplexity (54.98 against 28.61) and lower human fluency (2.07 against 2.53) 2202.11705 tab. 3.

Classifier-in-the-loop beam search (ALICE)

The statements are first written by GPT-3 from a few example sentences about one group, with top-k sampling 2203.09509 §3.1 2203.09509 §3.3. For a subset, ALICE runs a beam search of width 10, up to 30 tokens long 2203.09509 §3.3. Each candidate next token is scored by a weighted sum of two terms: the language model's log-probability, and the classifier's log-probability for the class the output is pushed towards. Both weights are 0.5 2203.09509 eq. 1 2203.09509 §3.3. The target class is the opposite of the prompt's label. Toxic prompts push towards benign, and benign prompts push towards toxic 2203.09509 §3.2. Tokens copied from the prompt are blocked 2203.09509 §3.3. The classifier is a HateBERT model fine-tuned on OffensEval 2203.09509 §3.3. Card: ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection.

Its share of the output is small. ALICE supplied 14,174 of 274,186 statements, about 5% 2203.09509 tab. 2. Reported results, on 125 prompts: ALICE output fooled HateBERT on 26.4% of statements, against 16.8% for top-k, and toxic prompts matched the human toxicity rating less often (40.3%, against 67.7% for top-k) 2203.09509 §4.3. In a larger study of 8,960 statements, toxic ALICE statements fooled HateBERT 58.97% of the time, against 26.88% for toxic top-k statements 2203.09509 app. E. Training HateBERT on ALICE plus top-k data raised AUC on three human-written sets to 0.71, 0.67 and 0.66, from 0.60, 0.60 and 0.47 with no fine-tuning 2203.09509 tab. 4.

Critique weights in a beam score (Self-RAG)

The model is trained to emit reflection tokens. These mark whether to retrieve, whether a passage is relevant, whether a segment is supported by it, and how useful the answer is 2310.11511 §3.1 2310.11511 tab. 1. At inference, each candidate continuation is scored by its language-model probability plus a weighted sum of the relevance, support and usefulness critiques 2310.11511 eq. 3 2310.11511 eq. 4. The weights are inference-time settings, so the trade-off can change without retraining 2310.11511 §3.3. Hard constraints can also drop a candidate that emits an undesirable token 2310.11511 §3.3. Raising the support weight increases ASQA citation precision and lowers MAUVE fluency 2310.11511 fig. 3b. Card: Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.

Reported results, zero-shot, for the 7B and 13B models: ASQA citation precision 66.9 and 70.3, against 5.5 for the best non-proprietary baseline, and citation recall 67.8 and 71.3, against 7.5 2310.11511 tab. 2. These numbers depend on retrieval as well as on the critique weights. The ablation that removes the support term from the beam score is shown only in a figure, so this page gives no value for it 2310.11511 fig. 3a.

Moral-direction filter at decoding time (MoRT)

The direction is estimated once, from a fixed sentence encoder. Verbs with known moral associations are placed in 10 question frames. Each frame is embedded with BERT-large and pooled as in SBERT, the embeddings are averaged per verb, and the first principal component of the verb embeddings is kept as the direction 2103.11790 §Identifying the Moral Direction of Language Models 2103.11790 suppl. B.1. A phrase is scored by its projection on that direction 2103.11790 eq. 1.

At generation time, a GPT-2 model samples with top-k and then top-p filtering. Each surviving candidate is scored with the whole text so far plus that token. Candidates at or below a threshold t are removed. If too few remain, the best m are kept 2103.11790 §Guiding Generative Language Models using the MoralDirection. The generator is not retrained. The paper's stated aim is a method that needs a fixed encoder and a few example verbs, not a discriminator trained for each generator 2103.11790 §Differences between the MoralDirection approach and related methods. Card: Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do.

Reported results, author-reported. The score correlates with human yes-rates from a regional study and from Amazon Mechanical Turk at r = 0.78 and 0.79 2103.11790 §BERT's MoralDirection Strongly Correlates with Human Moral Norms. With three random 64-verb sets the correlation falls to 0.64, 0.60 and -0.01 2103.11790 suppl. B.2, so the result depends on the verb list. On RealToxicityPrompts, 1.17% of prompts get a generation above toxicity 0.9 at t = 0, against 9.82% for GPT-2 2103.11790 §Reducing Neural Toxic Degeneration in Language Models. In the full testbed, the expected maximum toxicity unprompted is 0.27 at t = 0 and 0.19 at t = 0.5, against 0.44 for GPT-2 2103.11790 tab. 3. Baselines marked with a dagger in that table were recomputed by the authors from testbed data 2103.11790 tab. 3.

Control tag on the input (speaker gender)

Each English source sentence gets one tag at its start, on the source side only. The same LSTM system is trained with and without the tag 1909.05088 §4.1 1909.05088 §4.2. Its evaluation sets are split by speaker gender 1909.05088 §4.1. Card: Getting Gender Right in Neural Machine Translation.

Reported results are BLEU, and the paper gives no gender-accuracy score. The tagged system is significantly better for French (+1.44), Italian (+0.29) and Danish (+0.31), and significantly worse for Finnish (−0.47) 1909.05088 tab. 2. On EN-FR test sets the gain is +1.13 for male speakers and +1.22 for female speakers 1909.05088 tab. 3. The effect on word choice is shown by one qualitative example, not measured 1909.05088 §5. The speaker labels are gender predictions from earlier MEP metadata, not verified labels 1909.05088 §3.

Control tokens in pre-training (toxicity level)

The same idea at pre-training scale. A small fraction of pre-training documents gets a control token that marks its toxicity level, from a fixed version of the Perspective API. At inference the model is conditioned on a low, medium or high token 2305.10403 §3 2305.10403 §5.1. The report says the tokens do not hurt unrelated tasks, and the authors report no clear cost on other evaluations, but they have no systematic ablation 2305.10403 §3 2305.10403 §5.1. The fraction of tagged data and the score thresholds for the three levels are not stated 2305.10403 §3 2305.10403 §5.1. Card: PaLM 2 Technical Report.

Reported results: on non-toxic prompts, the probability of a toxic continuation is 0.075 with no control token, 0.033 with the low token and 0.203 with the high token 2305.10403 tab. 14. The token moves the output in both directions, so it works as a dial, not a filter. Dialog prompting alone reduced toxic responses more than the control tokens did, and adding tokens to dialog prompting helped only on the standard dataset 2305.10403 §5.1.

Plug-and-play steering of an autoregressive model (comparison class)

The paper names PPLM and FUDGE as plug-and-play methods that steer a frozen left-to-right model with a classifier 2205.14217 §2 2205.14217 §6.3. The paper argues they work for coarse attributes but not for fine-grained structure 2205.14217 §1. Its own table shows FUDGE below the latent method on all five tasks, and PPLM reported only for semantic content 2205.14217 tab. 2. These methods have no card here, so this page names them only as the paper describes them.

Fine-tuned oracles (reference points)

The paper trains fine-tuned GPT-2 models as reference points: FT-sample and FT-search 2205.14217 §6.3 2205.14217 tab. 2. On syntax-tree and span control the latent method scores higher. FT-search scores higher on semantic content and parts of speech, and has lower lm-score, so the fine-tuned model is more fluent on those tasks 2205.14217 tab. 2.

Infilling (a different use of the same model)

Filling in a missing middle sentence is run without a classifier. The method draws several samples and keeps the one with the lowest average loss against the others 2205.14217 §5.2 2205.14217 §6.2. On aNLG it reports BLEU-4 of 7.1, against 6.7 for an autoregressive model trained for infilling. Its human rating is slightly lower, 0.37 against 0.39 2205.14217 tab. 5.

When to use which

Implementation sketch

The decoding-time variants share a core. A gradient step moves a continuous version of the output, and a final step turns it back into words. Diffusion-LM takes the step on latent vectors inside its denoising loop 2205.14217 §5.1. COLD takes it on soft logits, then decodes left to right with top-k filtering 2202.11705 §3.1 2202.11705 §3.3. The sketch below is the Diffusion-LM step, with the released code's fluency term.

# One guided denoising step for a classifier control  [§5.1]
# Input:  x_t (B, n, d); step t; classifier; fluency weight lam; Adagrad lr
# Output: x_{t-1} (B, n, d)

def guided_step(x_t, t, classifier, lam, lr):
    # Usual reverse step: mean and std of x_{t-1} given x_t  [§5.1] [App E]
    mean, std = reverse_step(x_t, t)
    x = mean + std * normal_like(mean)
    # Three Adagrad steps per diffusion step  [§5.1]
    for _ in range(3):
        # Control loss from the classifier  [§5.1] [App D]
        control = classifier_loss(x)
        # Fluency: the paper writes lam * log p(x_{t-1} | x_t); the code
        # uses a squared pull toward the mean scaled by std  [code]
        fluency = lam * squared_error(x, mean) / std
        x = adagrad_update(x, control + fluency, lr)
    return x

Notes for implementers of latent steering:

• The clamping step matters. Without it the denoiser does not commit to a word at each step, so the snap to the nearest word embedding must run at sampling time too 2205.14217 §4.2.
• The paper gives a per-task learning rate and trade-off. The released code differs for syntax tree, spans and parts of speech 2205.14217 tab. 6 infill.py.
• The released code applies no control in the last ten diffusion steps. The paper does not describe this infill_util.py.
• The paper does not say how the classifier is trained on latents. The released tree-control path needs a separately trained classifier 2205.14217 §5.1 infill.py.

Notes for implementers of COLD:

• The paper gives a plain Langevin step 2202.11705 eq. 2. The released code optimises an offset with Adam at the step size, and adds noise to the base logits cold_decoding.py cold_decoding.py. Decide which one you follow, and say which.
• Sampling is costly. Each sample runs 2000 Langevin steps, each with a forward and backward pass through GPT2-XL 2202.11705 §4.1.1. The paper draws 16 or 32 samples per input 2202.11705 app. B, so total cost scales with the sample count.
• The paper's choice among samples is not in the released abductive script, which writes all candidates cold_decoding.py 2202.11705 app. B.
• The paper's Table 3 values are not tied to a stated selection rule for the lexical task 2202.11705 §4.3.

Notes for implementers of ALICE:

• Use one weighted sum, as in the paper 2203.09509 eq. 1. The released code adds the classifier term twice in one loop, so with both weights at 0.5 the effective weights are 0.25 and 0.75 alice.py. Pick one behaviour and say which.
• The paper keeps the top 100 candidate tokens 2203.09509 §3.3. The released code keeps 5 by default, because of a limit in the API it calls alice.py.
• The paper's tables for the fine-tuning results were transcribed from a page image in the source, with no text layer. Check every digit of 2203.09509 tab. 4 against the PDF before relying on it.
• The released beam search returns each hypothesis starting from the full prompt, so the output begins with the prompt text alice.py alice.py. Remove that prefix before use. The paper does not say whether its released field has it removed.

Notes for implementers of Self-RAG critique scoring:

• The paper sets the weights for relevance, support and usefulness to 1.0, 1.0 and 0.5 2310.11511 §4.3. The ASQA runner hard-codes these values, so the command-line weight flags have no effect run_long_form_static.py.
• The paper's beam score is the language-model probability plus the weighted critique sum 2310.11511 eq. 3 2310.11511 eq. 4. The released long-form code multiplies each child's score by its parent's score, so the score is a product over segments run_long_form_static.py. Decide which one you follow, and say which.

Notes for implementers of the moral-direction filter:

• The paper's score is a plain projection 2103.11790 eq. 1. The released code centres on the PCA mean first funcs_mcm.py. Decide which one you follow, and say which.
• The paper gives no sign convention. In the code, the sign is set so that the verb cluster without "kill" scores positive pca.py. Check the polarity before use: in the paper's table, "killing people" scores -1.0 and "helping" scores 0.5 2103.11790 tab. 1.
• The paper keeps at least 5 candidates in its experiments 2103.11790 §Experimental setup. The code's class default is 30 gpt2MCM_generation.py, so calling the defaults gives a different filter.
• The paper gives no top-p value and no maximum length. The code uses top-p 0.9 and 20 new tokens generation.py run_prompts_experiment.py.
• The score is divided by a fixed constant in generation mcm_models.py. The paper does not say where it comes from, so a new encoder or verb set needs its threshold tuned again.

Contradictions

None known between papers that disagree on the same result, since the seven method cards test different tasks and models, and the survey has no experiments section. Five points still need care.

• Infilling numbers across papers. Diffusion-LM's table lists COLD at 1.8 BLEU-4 on aNLG and states that the COLD and DeLorean rows are copied from their papers, not re-run 2205.14217 tab. 5. COLD reports 1.79 on its own abductive task 2202.11705 tab. 1. The 7.1 of Diffusion-LM and the 1.8 of COLD are therefore not measured in the same setup.
• Within the control-tag paper. The text of the control-tag paper says French, Italian, Portuguese and Greek improve significantly, but Table 2 marks only French, Italian and Danish as significant gains 1909.05088 §5 1909.05088 tab. 2. Table 2's caption gives p < 0.05, and Table 3's caption gives p < 0.5 1909.05088 tab. 2 1909.05088 tab. 3. This page uses the table and the 0.05 threshold.
• Within the ALICE paper. Two human studies report different label-match rates for toxic ALICE prompts. The 125-prompt study gives 40.3% 2203.09509 §4.3, and the 8,960-statement study gives 66.86% 2203.09509 app. E. The samples differ in size and in how they were drawn, and the paper does not reconcile them. Both are reported here.
• Within the Self-RAG paper. The text says the critic agrees with GPT-4 on most categories above 90%, but the table gives 80.2 for relevance and 73.5 for usefulness on the 7B critic 2310.11511 app. B.1. This page cites the table.
• Within the moral-direction paper. The main text gives 54 verbs for the principal component analysis 2103.11790 §Methods: The MoralDirection of language models. The supplement gives 64 2103.11790 suppl. B.2, and the release uses 64 dataMoral.py. The two counts are not reconciled here.

Papers

• Zhang et al., survey of controllable generation with transformer PLMs: a taxonomy of controllable generation with pre-trained transformers, covering fine-tuning, retraining and post-processing, seven task families and evaluation metrics; it runs no experiments.
• Diffusion-LM Improves Controllable Text Generation: a diffusion language model whose continuous latents are steered by classifier gradients, with results on five control tasks and infilling.
• COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics: energy-based decoding with Langevin dynamics over soft logits, on a frozen GPT2-XL, for abductive reasoning, counterfactual rewriting and keyword-constrained generation.
• ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection: GPT-3 generation of toxic and benign statements, with ALICE, a classifier-in-the-loop beam search, for a small share of the data.
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection: one language model trained to emit reflection tokens, whose critique scores are weighted at inference; the paper also covers retrieval, which this page does not use.
• Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do: a moral direction in BERT sentence embeddings, used to score and filter GPT-2 candidates during decoding, with a toxicity test on RealToxicityPrompts.
• Getting Gender Right in Neural Machine Translation: a speaker-gender tag on the English source, trained with an LSTM translation model, evaluated by BLEU on 10 language pairs and on gender-split test sets.
• PaLM 2 Technical Report: a pre-training corpus with toxicity-level control tokens, conditioned on at inference; the report also covers scaling, multilingual data and reasoning results, which this page does not use.