Texygen: A Benchmarking Platform for Text Generation Models

In plain words

Many systems write text one token at a time. Researchers have built many ways to train them, but there is no shared test bench, and the usual scores each capture only part of what matters. A model can score well on one measure and still repeat itself.

This paper describes Texygen, a shared toolkit for running those writers on the same data and scoring them with the same measures. It comes with ready-made versions of several published training methods, a set of scoring tools, and a synthetic test: a fixed random writer produces the "real" data, so the true answer is known. The toolkit is meant to make results easier to compare and to reproduce.

TL;DR

• A benchmark platform, not a new method: it packages seven baseline text generators and five metrics §1 §2 §3.
• Metrics cover similarity to real text (BLEU, EmbSim), likelihood (NLL-oracle, NLL-test), and diversity (Self-BLEU) §2.2.
• The synthetic benchmark samples 10,000 sentences of length 20 from a random LSTM oracle over a 5,000-word vocabulary §4.1.
• On COCO captions, LeakGAN has the highest test BLEU at every order shown, and MLE has the lowest Self-BLEU tab. 2 tab. 3. All numbers are author-reported.
• The official code (MIT) has several divergences from the paper, listed under Paper vs code and Pitfalls.

Problem

Text generators are trained with maximum likelihood, which the paper notes is not a perfect objective for sequential generation §1. Adversarial and reinforcement-learning methods have been proposed to address that, but the paper argues three gaps block fair comparison §1:

• No single metric is comprehensive enough. Perplexity, oracle log-likelihood, human judgement and BLEU each capture only part of quality §1.
• Evaluation over several metrics is needed to reach firm conclusions §1.
• Many authors do not release code, so reported results are hard to reproduce §1.

The paper also points out a quality-diversity trade-off: a model can shrink to a narrow output pattern (mode collapse) and still score well on quality metrics §1 §2.2.3.

Method

Platform overview. Texygen has two parts: a utils part with the metrics and the oracle, and a model part with a GAN class that wraps a generator, a discriminator, a reward and data loaders. The platform is written in TensorFlow §3 fig. 1. Users plug in their own model through the GAN class and receive the metrics without writing the training loop §3.

Baseline models §2.1. Each model is described in the paper in prose with its objective:

• MLE. The sentence likelihood is maximised token by token, with the true previous tokens as input §2.1.
• SeqGAN. A discriminator scores finished sequences; the generator is trained with policy gradient, and Monte Carlo search estimates the value of each partial sequence §2.1.
• MaliGAN. The reward is normalised inside each batch and a moving-average baseline is subtracted §2.1.
• RankGAN. The discriminator is replaced by a ranker trained with a ranking loss, using the similarity of each sentence to a reference set §2.1.
• GSGAN. A Gumbel-softmax reparameterisation lets the discriminator's gradient flow through discrete samples §2.1.
• TextGAN. Adversarial feature matching with an MMD loss, plus a reconstruction term §2.1.
• LeakGAN. A hierarchical reinforcement-learning model with a manager that sets subgoals and a worker that fulfils them §2.1.

The paper gives the following objectives without equation numbers §2.1:

\max_\theta \sum_{x} \sum_{t} \log \pi_\theta(x_t \mid s_t) \quad [\text{§2.1}]

\min_\phi \; -\mathbb{E}_{Y \sim p_{data}}[\log D_\phi(Y)] - \mathbb{E}_{Y \sim G_\theta}[\log(1 - D_\phi(Y))] \quad [\text{§2.1}]

Metrics §2.2. The paper groups five metrics into three families.

Document-similarity metrics §2.2.1:

• BLEU measures n-gram overlap between a generated sentence and the real data; the paper reports orders 2 to 5 tab. 1 tab. 2.
• EmbSim compares word-embedding similarity structure between real and generated text. Word vectors come from a skip-gram model trained on each text set §2.2.1:

\mathrm{EmbSim} = \log\Big(\frac{\sum_{i=1}^{N} \cos(W'_i, W_i)}{N}\Big) \quad [\text{§2.2.1}]

Here W_{i,j} = \cos(e_i, e_j) from real data, W' is the same matrix for generated data, W_i and W'_i are the i-th columns, and N is the total number of words §2.2.1. The log base is not stated.

Likelihood-based metrics §2.2.2:

• NLL-oracle is the average negative log-likelihood of generated sentences under the oracle LSTM, with T tokens per sentence §2.2.2:

\mathrm{NLL}_{oracle} = -\mathbb{E}_{Y_{1:T} \sim G_\theta}\Big[\sum_{t=1}^{T} \log G_{oracle}(y_t \mid Y_{1:t-1})\Big] \quad [\text{§2.2.2}]

• NLL-test is the dual: the average negative log-likelihood of real test data under the generator. It applies only to autoregressive generators such as RNNs §2.2.2:

\mathrm{NLL}_{test} = -\mathbb{E}_{Y_{1:T} \sim G_{real}}\Big[\sum_{t=1}^{T} \log G_\theta(y_t \mid Y_{1:t-1})\Big] \quad [\text{§2.2.2}]

Divergence-based metrics §2.2.3:

• Self-BLEU uses each generated sentence as the hypothesis and the other generated sentences as references, computes BLEU for each, and averages. A higher value means less diversity §2.2.3 tab. 3.

Synthetic benchmark §4.1 §4.2. A randomly initialised LSTM serves as the oracle, which is the "true" distribution §2.1 §4.1. The protocol the paper describes is below. The paper does not give the oracle's initialisation or a train/test split for synthetic data.

# Synthetic benchmark run  [§4.1]
# Input:  a model name; vocabulary 5000; sentence length 20
# Output: NLL-oracle and NLL-test over training

def run_synthetic(model_name):
    # Build the oracle, an LSTM that is not trained on data  [§2.1] [§4.1]
    oracle = make_lstm_oracle(vocab_size=5000, length=20)
    # Sample 10000 sentences from the oracle as the real data  [§4.1]
    real_data = sample(oracle, count=10000)
    # Pretrain the generator by MLE for 80 epochs  [§4.1]
    generator = pretrain_mle(model_name, real_data, epochs=80)
    # Pretrain the discriminator for 80 epochs  [§4.1]
    discriminator = pretrain_discriminator(generator, real_data, epochs=80)
    for epoch in range(100):  # adversarial epochs  [§4.1]
        # One generator update per adversarial epoch  [§4.1]
        generator = policy_gradient_step(generator, discriminator)
        # 15 mini-batch discriminator updates per adversarial epoch  [§4.1]
        discriminator = train_discriminator(discriminator, updates=15)
        # Evaluate NLL-oracle and NLL-test  [§4.2] [fig. 2] [fig. 3]
        evaluate(oracle, generator, real_data)

Real-data benchmark §4.1 §4.3. The paper uses 20,000 image captions from COCO, with half for training and the rest as the test set §4.1. The paper applies NLL-oracle and NLL-test to the synthetic data only, and BLEU, Self-BLEU and EmbSim to the real data only, because the oracle LSTM cannot produce words with semantic meaning §4.2 §4.3. GSGAN is excluded from the real-data experiment because it did not generate sentences with semantics in the authors' runs §4.3.

EmbSim as the paper defines it §2.2.1:

# EmbSim  [§2.2.1]
# Input:  real sentences and generated sentences, as token lists
# Output: scalar score (log base not stated)

def emb_sim(real_sentences, generated_sentences):
    # Skip-gram embeddings e_i from the real data  [§2.2.1]
    real_embeddings = skip_gram(real_sentences)
    # Skip-gram embeddings e'_i from the generated data  [§2.2.1]
    gen_embeddings = skip_gram(generated_sentences)
    # Cosine matrices W and W', one entry per word pair  [§2.2.1]
    real_sim = cosine_matrix(real_embeddings)  # (N, N)
    gen_sim = cosine_matrix(gen_embeddings)    # (N, N)
    # N is the total number of words  [§2.2.1]
    total = 0.0
    for i in range(N):
        # Cosine between the i-th columns of W' and W  [§2.2.1]
        total += cosine(gen_sim[:, i], real_sim[:, i])
    # Average over the N words, then take the log  [§2.2.1]
    return log(total / N)

Hyperparameters

Values from the paper come first. Values marked code only are set in the official repository and are not stated in the paper; the paper's Pitfalls section explains where they matter.

Results

Author-reported. Setup: real-data BLEU and Self-BLEU on COCO captions, a 20,000-caption set split half and half §4.1. Models compared are the baseline set in §2.1, without GSGAN §4.3. The paper reports no single headline number for the platform; the table below is the test-set BLEU.

Self-BLEU on the same setup; a higher value means less diverse output §2.2.3:

Synthetic benchmark: the NLL-oracle and NLL-test curves are plotted over training fig. 2 fig. 3. The paper gives no table of final values for them. The text says TextGAN reaches the best NLL-oracle, and that LeakGAN converges quickly and performs well on both NLL metrics §4.2 fig. 2. Real-data EmbSim curves are in fig. 4 §4.3, and BLEU on training data is in tab. 1.

Paper vs code

Checked at code_commit 3104e22 of the official repository (MIT). The paper links the repository in its footnote on page 1, which is how we identified it.

• Discriminator updates per adversarial epoch. Paper: 15 mini-batch gradients §4.1. Code: 15 calls to train_discriminator, and each call makes 3 mini-batch updates, for 45 in total Seqgan.py Seqgan.py.
• Discriminator pretraining. Paper: 80 epochs §4.1. Code: 80 calls of train_discriminator, so 240 mini-batch updates Seqgan.py Seqgan.py.
• Initial parameters. Paper: N(0,1) for all generator parameters §4.1. Code: generator weights are drawn with stddev 0.1 SeqganGenerator.py. The oracle draws its embedding and output weights with stddev 1.0 OracleLstm.py OracleLstm.py, but its gate weights with stddev 1e6 OracleLstm.py.
• EmbSim. Paper: log of the mean cosine over all N words §2.2.1. Code: log base 10 DocEmbSim.py, and the comparison runs on 500 words chosen by the set() step below DocEmbSim.py.
• BLEU and Self-BLEU wiring. The paper reports BLEU and Self-BLEU tab. 1 tab. 3. In the code, the SeqGAN path registers only NLL-oracle, NLL-test and DocEmbSim Seqgan.py. SelfBleu is not imported by any model, and Bleu is imported by six models without being constructed. The README values match the paper's tables README.md.
• Real-data split. Paper: half train, half test §4.1. Code: one file is used for both training and the NLL-test metric Seqgan.py Seqgan.py.
• Synthetic split. Not stated in the paper. Code: NLL-test is computed over the same 10000 sentences the generator trains on Seqgan.py Seqgan.py.
• Metric file names. utils/metrics/EmbSim.py is imported by several models but its score comes from the model's LSTM similarity attribute EmbSim.py. The SeqGAN path uses DocEmbSim, which matches the paper's skip-gram construction Seqgan.py.

Pitfalls

• Numbers depend on the oracle. NLL-oracle and NLL-test are only comparable across papers that rebuild the same oracle. The paper does not give the oracle's initialisation, and the code draws gate weights with stddev 1e6 OracleLstm.py. The paper does not analyse what that does to the oracle's distribution.
• Synthetic NLL-test is measured on training data Seqgan.py. Treat it as a fit score, not a generalisation score.
• The discriminator update counts differ. The paper's 15 updates per epoch become 45 in the code Seqgan.py, and the pretraining count is 240, not 80 Seqgan.py. The paper does not say how an "epoch" of discriminator training is defined, so the two readings give different training budgets.
• Roll-out policy lag. The code updates the roll-out policy once per adversarial epoch, blending 0.8 of the old parameters with 0.2 of the new SeqganReward.py Seqgan.py. The paper does not mention this, and the lag changes the reward signal.
• EmbSim word subset. The code sorts words by frequency and then calls list(set(...)), which discards that order. The 500-word cut is therefore the first 500 entries of a set, not the 500 most frequent words DocEmbSim.py.
• EmbSim is not reproducible run to run. Skip-gram batches use Python's random module without a seed DocEmbSim.py, although the TensorFlow initialisers are seeded DocEmbSim.py. Each evaluation retrains the skip-gram models from scratch DocEmbSim.py.
• EmbSim log base. The paper writes log without a base. The code uses log base 10 DocEmbSim.py. Compare curves only if you use the same base.
• Self-BLEU sampling. The paper does not give a sample size or weighting. The class in the repo uses the first 500 sentences, unshuffled, with uniform weights and method-1 smoothing SelfBleu.py SelfBleu.py SelfBleu.py.
• The code uses TensorFlow 1 APIs (tf.placeholder, tf.Session) SeqganReward.py. Porting to TensorFlow 2 needs care.
• Evaluation schedule. The paper's curves are not labelled with an interval. The code evaluates every 5 epochs and at the end Seqgan.py.

Lineage

• Builds on: not stated. The platform implements SeqGAN and the other models listed below as baselines §2.1; for SeqGAN itself see SeqGAN.
• Compared against: MLE, MaliGAN arXiv:1702.07983, RankGAN arXiv:1705.11001, GSGAN arXiv:1611.04051, TextGAN arXiv:1706.03850 and LeakGAN §2.1.
• Superseded by: not stated.