Long Text Generation via Adversarial Training with Leaked Information

In plain words

A text generator trained with a GAN gets one score at the very end: the judge saw the whole sentence and called it real or fake. For a short sentence that is enough. For a long one, the generator cannot tell which word caused the low score, and the signal is too thin to learn from.

LeakGAN lets the judge give hints while the sentence is still being written. The judge is a convolutional classifier that reads the sentence so far and produces a vector of internal features. Those features go to a small manager network, which turns them into a steering vector. A worker network then picks each next word under that steering. The manager learns from how the judge's features change as the sentence grows, so it learns to push the text toward the kind of features real sentences have. The authors report that the two networks end up tracking things such as punctuation, clause boundaries and sentence endings, although nobody labelled those.

On long text the method beats earlier GAN text generators on n-gram overlap (BLEU) and in a Turing-style test with human judges.

TL;DR

LeakGAN is a text GAN whose discriminator's intermediate CNN features are passed to a hierarchical generator: a Manager LSTM turns them into a goal vector at every step, and a Worker LSTM uses that goal to choose the next word §Methodology. The Manager is trained from the change in discriminator features, the Worker from an intrinsic cosine reward, and both use policy gradients eq. 7 eq. 8 eq. 9. Consider it when a GAN must produce long text (more than 20 words). Expect a multi-stage training loop, a moving feature space, and author-reported results on a few datasets, with several training details left unstated.

Problem

A GAN discriminator scores a whole generated sequence, and the reward is available only after the last token §Introduction. The authors argue that this scalar reward becomes less informative as sentences get longer, and that binary feedback from a discriminator needs many training steps §Related Work. Earlier text GANs were mostly evaluated on short sequences of about 20 words or fewer §Introduction. The paper's stated aim is to give the generator intermediate guidance, so that long sentences get a signal at each step §Methodology.

Method

The paper's sections and appendix use titles rather than numbers, so citations below give the section title. Equation numbers are the main-text numbers. The appendix has its own formula list with different numbering (see Pitfalls).

1. Discriminator features. A CNN reads the sentence and yields a feature vector f_t of the prefix s_t. Its sigmoid head gives the scalar reward eq. 1 §Leaked Features from D as Guiding Signals. The feature extractor uses convolutions with windows of 1 to 10, 15 and 20 words, with 100 to 200 kernels per window, which sum to 1,720 features at length 20 app. tab. 1 app. Experiment Settings.
2. Manager. An LSTM takes f_t and produces a raw goal ĝ_t. The goal is normalised to unit length, g_t = ĝ_t / ‖ĝ_t‖ eq. 2 eq. 3 §A Hierarchical Structure of G.
3. Goal embedding. A linear map W_ψ is applied to the sum of the last c goals, giving a k-dimensional goal embedding w_t eq. 4. The paper sets c = 4 after preliminary runs §Experiment, GAN Setting and k = 16 app. Experiment Settings.
4. Worker. An LSTM reads the current word x_t and outputs a matrix O_t with one row per vocabulary word and k columns eq. 5.
5. Next-word distribution. The probabilities are a softmax over O_t · w_t divided by a temperature α eq. 6 §A Hierarchical Structure of G. A higher α is used in training and a lower one when sampling; the values are not stated §Temperature Control.
6. Manager update. A policy gradient with Q(f_t, g_t) = E[r_t] as the weight. Q is estimated by Monte Carlo search. The update pushes each goal to align with the change in features over the next c steps eq. 7 §Training of G.
7. Worker update. A REINFORCE update on the chosen word, weighted by the intrinsic reward r^I_t. That reward is the mean cosine between the feature change and the goals of the last c steps eq. 8 eq. 9 §Training of G.
8. Reward rescaling. Rewards are mapped to a sigmoid of their rank within each batch column, with δ = 12 eq. 11 §Training Techniques.
9. Pre-training. The discriminator is first trained to separate real from generated text. The generator is then pre-trained by maximum likelihood, and the Manager by the separate rule in eq. 10 with Q set to 1 §Training of G alg. 1.
10. Interleaved training. The authors add one epoch of supervised generator training after every 15 epochs of adversarial training §Interleaved Training.
11. Adversarial loop. Generator updates (steps 6 to 8) alternate with discriminator updates until convergence alg. 1.

Key equations (main-text numbers):

D_\phi(s) = \mathrm{sigmoid}(\phi_l^\top f), \quad f = \mathcal{F}(s; \phi_f) eq. 1

w_t = W_\psi\Big(\sum_{i=1}^{c} g_{t-i}\Big) eq. 4

G_\theta(\cdot \mid s_t) = \mathrm{softmax}\big(O_t \cdot w_t / \alpha\big) eq. 6

\nabla^{adv}_{\theta_m} g_t = -Q(f_t, g_t)\, \nabla_{\theta_m} d_{cos}\big(f_{t+c} - f_t,\; g_t(\theta_m)\big) eq. 7

r^I_t = \frac{1}{c} \sum_{i=1}^{c} d_{cos}\big(f_t - f_{t-i},\; g_{t-i}\big) eq. 9

R^t_i = \sigma\Big(\delta \big(0.5 - \tfrac{\mathrm{rank}(i)}{B}\big)\Big) eq. 11

Pseudocode for one generated sequence, from eq. 1 to eq. 6 and the Manager and Worker described above:

# Generate one sequence  [§Hierarchical Structure of G]
# Shapes: T words, feature size d, goal size k, vocabulary size V

def generate(discriminator, manager, worker, T, c, alpha):
    # Both LSTM states start at zero  [app. Pseudo Code]
    manager_state = zeros(hidden_size)
    worker_state = zeros(hidden_size)
    # The start-of-sequence token is fed first  [§Hierarchical Structure of G]
    word = START_TOKEN
    tokens = []
    goals = []
    for t in range(T):
        # Leaked feature of the prefix generated so far  [eq. 1]
        feature = discriminator.features(tokens)     # (d,)
        # Manager maps the feature to a unit-norm goal  [eq. 2] [eq. 3]
        raw_goal, manager_state = manager(feature, manager_state)
        goals.append(raw_goal / norm(raw_goal))      # (k,)
        # Sum of the last c goals, projected to the goal embedding  [eq. 4]
        # Fewer than c goals exist at the start: not stated
        goal_sum = sum(goals[-c:])                   # (k,)
        goal_embedding = W_psi @ goal_sum            # (k,)
        # Worker encodes the current word  [eq. 5]
        output_matrix, worker_state = worker(word, worker_state)  # (V, k)
        # Score each word by its row dot the goal, then soften  [eq. 6]
        logits = output_matrix @ goal_embedding      # (V,)
        probs = softmax(logits / alpha)              # (V,)
        word = sample(probs)
        tokens.append(word)
    return tokens

Pseudocode for the two update rules and the reward rescaling:

# Manager loss: its gradient equals eq. 7  [eq. 7]
# features[t]: discriminator features of the prefix up to step t
# Q[t]: Monte Carlo estimate of E[r_t] at step t

def manager_loss(features, goals, Q, c):
    loss = 0
    for t in range(len(goals) - c):
        # Align each goal with the feature change over the next c steps
        change = features[t + c] - features[t]
        loss -= Q[t] * cosine(change, goals[t])
    return loss

# Worker loss: REINFORCE with the intrinsic reward  [eq. 8] [eq. 9]
# Start index c is an assumption: the paper does not say how t < c is handled

def worker_loss(features, goals, tokens, probs, c):
    loss = 0
    for t in range(c, len(tokens)):
        # Mean cosine between the feature change and each past goal  [eq. 9]
        reward = 0
        for i in range(1, c + 1):
            change = features[t] - features[t - i]
            reward += cosine(change, goals[t - i])
        reward /= c
        # Policy gradient on the chosen word  [eq. 8]
        loss -= reward * log(probs[t][tokens[t]])
    return loss

# Bootstrapped rescaled activation for one timestep  [eq. 11]
# rewards: (B,) over the batch; rank 1 is the highest reward

def rescale_rewards(rewards, delta=12.0):
    B = len(rewards)
    ranks = rank_high_to_low(rewards)            # values in 1..B
    return sigmoid(delta * (0.5 - ranks / B))    # (B,)

Hyperparameters

Values from the paper unless the Where column says code. "Not stated" means the paper does not give it.

Results

Author-reported. The paper evaluates on synthetic data and three real corpora, and it runs a Turing-style test.

Setup:

• Synthetic: an oracle LSTM generates 10,000 training sequences of length 20 or 40. Quality is the oracle's negative log-likelihood of the generated samples. Baselines: MLE, SeqGAN and RankGAN; real oracle samples serve as the reference §Experiment.
• WMT News: the EMNLP2017 WMT News section, after removing words with frequency below 4,050 and sentences shorter than 20 words. The vocabulary is 5,742 words; 200,000 training and 10,000 test sentences. Metric: BLEU-2 to BLEU-5 §Long Text Generation.
• COCO captions: words with frequency below 10 removed, leaving 4,980 words; 80,000 training and 5,000 test captions. Metric: BLEU-2 to BLEU-5 §Middle Text Generation.
• Chinese poems: 4-line, 5-character poems, scored by BLEU-2 §Short Text Generation.
• Turing test: 20 sentences per method. 62 participants each judge 30 sentences drawn from the three compared methods, so each sentence gets 31 judgements. A sentence scores 1 when judged real §Turing Test.

Synthetic negative log-likelihood, lower is better tab. 1:

BLEU on real corpora tab. 2 tab. 3 tab. 4:

The paper reports p < 10⁻⁶ for LeakGAN against the second-best method in every comparison of tab. 1 to tab. 3 §Evaluation Metrics.

Turing test, share of sentences judged real tab. 5:

Other results:

• On WMT News the BLEU gain over the baselines grows with sentence length, from 20 to 45 words fig. 3 §Performance Robustness in Long Text Generation.
• Training curves show LeakGAN converging faster and to a lower NLL than the baselines at both lengths fig. 2.
• In a PCA projection of discriminator features, LeakGAN's feature traces move toward the real-data region during generation, while SeqGAN and RankGAN do not fig. 4 app. fig. 3.
• Single dimensions of the Manager–Worker product track sentence structure. A peak in one marks a token that triggers a long suffix, such as a formal subject. Another fluctuates at the start or end of a clause, where punctuation or a conjunction is likely fig. 5 app. fig. 2 §Model Explanation.

Paper vs code

Repository: CR-Gjx/LeakGAN at commit 4044de6, the paper's footnote link. The repository has no LICENSE file, so code_license is "not stated". We cite lines only and quote no code. It contains the synthetic-data and COCO experiments, not the WMT or poems runs. It needs TensorFlow r1.2.1 and Python 2.7 README.md.

Differences, each with both citations:

• Learning rate, length 20. Paper: 0.001 app. Experiment Settings. Code: 0.0015 Main.py.
• Embedding size, length 40. Paper: 32 app. Experiment Settings. Code: 64 Main.py.
• Dropout. Paper: keep rate 0.75 §Experiment, GAN Setting. Code: 1.0 for the discriminator and for generator rollouts Main.py Main.py.
• L2 regularisation. Paper: applied, no value. Code: 0.2 is defined Main.py but not passed to the discriminator Main.py, so the constructor default of 0.0 applies Discriminator.py. The penalty covers only the output layer Discriminator.py.
• Length-40 CNN windows. Paper's table lists windows 1 to 10, 16, 20, 30 and 40, with 160 kernels for the last four app. tab. 1. Code lists 15 instead of 16 and gives 13 kernel counts for 14 windows. Because zip stops at the shorter list, window 40 is dropped Main.py Discriminator.py. As we read the paper's table, its features total 2,040; the code's total is 1,880.
• Discriminator architecture. Code adds a highway layer and dropout after pooling Discriminator.py. The paper's appendix figure is described as lacking both app. fig. 1.
• Learning-rate decay. Paper: silent. Code: decay schedules are defined LeakGANModel.py but every optimiser uses the constant starting rate LeakGANModel.py LeakGANModel.py LeakGANModel.py.
• Temperature. Paper: α is a divisor in eq. 6, with values not stated. Code multiplies the logits by 1.0 in training and by 1.5 when sampling, which is α = 1/factor LeakGANModel.py. The first two steps always use 1.5 LeakGANModel.py. A "No Temperature" variant drops the factor LeakGANModel.py.
• Worker projection. Eq. 4 has no normalisation. Code L2-normalises the projected goal before the Worker uses it LeakGANModel.py.
• Goal window. Eq. 4 says "recent c goals". Code sums goals in non-overlapping blocks of c steps, and the goal changes only at block boundaries LeakGANModel.py LeakGANModel.py.
• Output activation. The appendix formula list writes G as a sigmoid app. Formulas for Reference. The main text (eq. 6) and code use softmax LeakGANModel.py.
• Intrinsic reward. Eq. 9 averages cosines over i = 1 to c. Code produces one cosine per step against a reference feature that depends on position in the block: the block start, or c steps back at a boundary LeakGANModel.py LeakGANModel.py.
• Rescaling constants and ranking. Paper: δ = 12 with rank high to low eq. 11. Code: 16 with offset 8, so the argument is 16·rank/B − 8 Main.py. Ranks come from iterating a dict keyed by reward value Main.py. Python 2 dict order is not sorted by value, so ranks may not follow reward. Equal rewards also merge into one entry. Treat the rescaling as unverified.
• Q estimation. Paper: Monte Carlo search, count not stated. Code: 4 rollouts, recomputed at block boundaries Main.py Main.py.
• Discriminator input during rollouts. The prefix is padded to full length with −1 before each feature extraction LeakGANModel.py. The paper does not mention padding.
• Discriminator training schedule. The paper states no schedule. Synthetic code: 10 discriminator rounds, each followed by 8 generator epochs Main.py. Then 800 adversarial batches, each with 5 rounds of 3 discriminator epochs Main.py.
• Interleaved supervised step. Paper: one full epoch after every 15 adversarial epochs §Interleaved Training. COCO code: one supervised step every 15 batches Main.py, and each "epoch" uses only num_batch/20 batches Main.py. The synthetic code has no interleaved step Main.py.
• Real samples in the reward (COCO). Code mixes real captions into the discriminator input; the count grows with training to a cap of 64 Main.py Main.py Main.py. Ranks are computed over this combined batch before it is truncated Main.py. The paper does not describe this.
• COCO settings. Code: vocabulary 4,839 Main.py; paper: 4,980 words §Middle Text Generation. Code sequence length 32, embedding and hidden size 128, discriminator embedding 256 Main.py Main.py. The paper states none of these.
• Oracle initialisation. Paper: N(0,1) §Experiment, Synthetic Oracle. Length-20 code loads saved parameters from save/target_params.pkl Main.py. We cannot check their distribution from the code.
• Gradient clipping. Code clips the global norm at 5.0 LeakGANModel.py. The paper does not mention clipping.

Pitfalls

• Numbering. The paper's sections are unnumbered, and its appendix restarts figures and tables, so "Table 1" can mean either. Use the title and prefix (app.) when citing. The appendix formula list (1 to 11) does not match the main-text equations: its equation 7 is the sigmoid generator, while the main text's eq. 6 is the softmax generator. Cite main-text equations.
• First two tokens. Both the synthetic and COCO code sample the first two tokens at the sampling temperature even during training. The condition is i > 1 LeakGANModel.py. Reproduce this or the early distribution will differ.
• Temperature inverts. Eq. 6 divides by α, but the code multiplies by a factor. Implement α = 1/factor.
• Sparse goal index. Eq. 4 and eq. 9 need c past steps. At t < c the paper is silent. The code sums the goals that exist and uses the block start as the reference feature. Choose a rule and state it.
• Feature drift. The Manager reads features from the same CNN that the discriminator trains. The code calls update_feature_function after each discriminator update Main.py. The goal targets therefore move during training, and the paper's appendix reports that replacing the hierarchy with a fully connected layer made training numerically unstable app. Discussions.
• Cost per step. Each generation step re-runs the CNN over the full padded sentence LeakGANModel.py. Monte Carlo rollouts repeat this for each block boundary Main.py. The paper gives no cost figures.
• Unstated values. Optimiser, batch size, adversarial step count, rollout count, α values, L2 coefficient and Chinese poem count are all not stated in the paper. The code fills some of them in, but those choices are the authors' and are not justified in the paper.
• Results. The BLEU and NLL results are author-reported. The paper's quality metrics are NLL, BLEU and a Turing test, and the Turing test uses 62 participants with 31 judges per sentence §Turing Test.

Lineage

• Builds on: the feudal hierarchical RL architecture (Manager and Worker with separately trained parts), arXiv:1703.01161, adapted from RL to a GAN discriminator's features §A Hierarchical Structure of G.
• Compared against: SeqGAN (policy-gradient text GAN, Yu et al. 2017; the paper gives no arXiv id); RankGAN, arXiv:1705.11001; MLE-trained LSTM.
• Superseded by: not known from content in this wiki.