SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient

In plain words

Computers can write text one token at a time. The usual way to train such a writer checks each next token against the training text, so a small slip early on makes the later text drift, and the drift grows with length.

GANs are a training idea where a judge learns to tell real from fake and the writer learns to fool it. That works well for pictures, but it is awkward for text, because a token is a discrete choice and the judge's feedback cannot be pushed back through a choice in the usual way. The judge also only sees the finished text, not the partial text.

This paper treats the writer as a player that picks each next token like a move. The judge scores the whole sequence once it is finished. To tell each earlier choice how much it helped, the method finishes the sequence many times from each point, averages the judge's scores, and uses that average as a reward. The writer is nudged toward choices that led to higher rewards, and the judge is retrained on fresh fakes. The authors report gains over the baselines they compare against, on synthetic data and on poems, speeches and music.

TL;DR

• Frames sequence generation as a reinforcement-learning policy: the state is the tokens so far, the action is the next token § Introduction.
• A CNN discriminator judges complete sequences. Its probability of "real" is the reward § SeqGAN via Policy Gradient eq. 2.
• Monte Carlo roll-outs estimate a per-token action value from that terminal reward eq. 3 eq. 4.
• Reported oracle NLL on synthetic data: 8.736 for SeqGAN versus 9.038 for MLE, and lower is better tab. 1.
• Real-data results (author-reported) are on Chinese poems, Obama speeches and folk-tune music tab. 2 tab. 3 tab. 4.

Problem

Models trained by maximum likelihood see the true previous tokens during training, but they generate from their own previous tokens. This mismatch, called exposure bias, makes errors accumulate along a sequence, and the effect grows with length § Introduction. Scheduled sampling was proposed to narrow the gap, but the paper cites an argument that it is an inconsistent objective and does not fix the problem at its root § Introduction.

Sequence-level losses such as BLEU work for translation, where a reference is available, but not for tasks such as poem or dialogue generation, where no accurate score exists for a generated sequence § Introduction.

GANs can give a sequence-level signal without a hand-written score. Their feedback is a gradient with respect to the generator's continuous outputs, and that gradient has no clear meaning for a discrete token. A GAN also scores only complete sequences § Introduction.

Method

Citation note. The source sets secnumdepth to 0, so the compiled section headings are most likely unnumbered. Section references below use the heading names. Equation, table, figure and algorithm numbers are counted in document order from the source.

Generator as policy. The generator G_\theta is a recurrent network with LSTM cells and a softmax output over the vocabulary. The policy picks the next token y_t given the prefix Y_{1:t-1}. Choosing a token makes the next state deterministic, so the state is simply the prefix so far [§ Sequence Generative Adversarial Nets]. GRU cells or attention could also be used [§ The Generative Model for Sequences].

Discriminator as reward. The discriminator D_\phi is a CNN over token embeddings. It uses several kernel widths, max-over-time pooling, one highway layer and a sigmoid output. It outputs the probability that a finished sequence is real § The Discriminative Model for Sequences.

Terminal reward and objective. The generator maximises the expected terminal reward from the start state eq. 1:

J(\theta) = \mathbb{E}[R_T \mid s_0, \theta] = \sum_{y_1} G_\theta(y_1 \mid s_0)\, Q^{G_\theta}_{D_\phi}(s_0, y_1) eq. 1

The only reward is at the end. For the last token the action value is the discriminator's output on the full sequence eq. 2. For an earlier token, the action value is the average of the discriminator over N roll-outs that keep the prefix and finish the sequence with a roll-out policy G_\beta eq. 3 eq. 4:

Q(s = Y_{1:t-1}, a = y_t) = \begin{cases} \frac{1}{N}\sum_{n=1}^{N} D_\phi(Y^n_{1:T}) & t < T \\ D_\phi(Y_{1:T}) & t = T \end{cases} eq. 4

The paper's experiments set G_\beta equal to the generator [§ SeqGAN via Policy Gradient]. The Algorithm section, however, resets it to the generator after each iteration alg. 1.

Discriminator objective. The discriminator minimises cross entropy between real sequences from the data (label 1) and sequences sampled from G_\theta (label 0) eq. 5:

\min_\phi -\mathbb{E}_{Y \sim p_{\text{data}}}[\log D_\phi(Y)] - \mathbb{E}_{Y \sim G_\theta}[\log(1 - D_\phi(Y))] eq. 5

Policy gradient. The gradient of J is written as a sum over positions and expanded with likelihood ratios eq. 6 eq. 7. It is applied as a plain gradient-ascent step, and the paper notes Adam or RMSprop can be used eq. 8 § SeqGAN via Policy Gradient:

\theta \leftarrow \theta + \alpha_h \nabla_\theta J(\theta) eq. 8

Training schedule. The generator is first pre-trained by maximum likelihood on the training set. The discriminator is then trained on real versus generated sequences. The two are then trained in turns. In each iteration the generator takes g-steps of policy gradient, each with rewards from roll-outs alg. 1. Then the discriminator takes d-steps, each one regenerating negative examples from the current generator and training for k epochs. The paper says the number of negatives matches the number of positives [§ SeqGAN via Policy Gradient] alg. 1. The paper's figures illustrate this loop fig. 1.

# SeqGAN training loop  [alg. 1]
# Inputs: generator (theta), roll-out copy (beta), discriminator (phi),
#         dataset S of real sequences of length T  [alg. 1]

def train_seqgan(generator, discriminator, dataset):
    # Pre-train the generator on S by maximum likelihood  [alg. 1]
    pretrain_mle(generator, dataset)
    # Start the roll-out policy as a copy of the generator  [alg. 1]
    roll_out = copy_of(generator)
    # Train D: real sequences against sequences sampled from G  [alg. 1]
    train_discriminator(discriminator, dataset, generator)
    while not converged:
        # g-steps: policy gradient on the generator  [alg. 1] [eq. 8]
        for step in range(g_steps):
            # Sample Y_{1:T} from the current generator  [alg. 1]
            sample = generator.sample()
            # Estimate the action value of every token  [alg. 1] [eq. 4]
            rewards = estimate_action_values(sample, roll_out, discriminator)
            # Step along the policy gradient  [eq. 6] [eq. 7] [eq. 8]
            generator.update(policy_gradient_loss(sample, rewards))
        # d-steps: refresh negatives from G and retrain D  [alg. 1] [eq. 5]
        for step in range(d_steps):
            train_discriminator(discriminator, dataset, generator)
        # Reset the roll-out policy to the new generator  [alg. 1]
        roll_out = copy_of(generator)

# Estimate Q(Y_{1:t-1}, y_t) for every position t  [eq. 4]
# sample: (B, T) tokens;  returns rewards: (B, T)
# rewards[:, t - 1] holds the reward for token y_t

def estimate_action_values(sample, roll_out, discriminator, n_rollouts):
    # Full sequence: the discriminator scores the sample itself  [eq. 4]
    rewards = zeros_like(sample)  # (B, T)
    # Prefix lengths 1 .. T-1 use Monte Carlo roll-outs  [eq. 3] [eq. 4]
    for t in range(1, T):
        for n in range(n_rollouts):
            # Keep the first t tokens; G_beta finishes the sequence  [eq. 3]
            completed = roll_out.complete(sample[:, :t])  # (B, T)
            rewards[:, t - 1] += discriminator(completed) / n_rollouts
    rewards[:, T - 1] = discriminator(sample)  # [eq. 2] [eq. 4]
    return rewards

# Loss whose gradient is the policy-gradient estimate  [eq. 7] [eq. 8]
# log_probs: (B, T) entries log G(y_t | Y_{1:t-1}) for the sampled tokens
# rewards:   (B, T) from estimate_action_values

def policy_gradient_loss(log_probs, rewards):
    # Minimising the negative sum performs gradient ascent on J  [eq. 8]
    return -sum(log_probs * rewards)  # (B, T) -> scalar

Pre-training and reproducibility. The paper pre-trains with maximum likelihood until the curve levels off, which it describes as about 150 epochs § Results. A 20-epoch pre-training ablation shows adversarial training is then slow and unstable app. More Ablation Study fig. 4.

Hyperparameters

Items marked "not stated" are absent from the paper text and appendix. Code values come from the official repository at code_commit and are not in the paper.

Results

Author-reported. The setups are as follows. Synthetic: an oracle LSTM generates the data, and the metric is NLL under the oracle, so lower is better [§ Evaluation Metric]. Poems: BLEU-2 and a human judgement § Text Generation. Speeches: BLEU-3 and BLEU-4 § Text Generation. Music: BLEU-4 and MSE [§ Music Generation]. The text says SeqGAN outperforms on both music metrics, so lower MSE is treated as better. The baselines are listed in each row. Table 1 names its test (a t-test of SeqGAN against each baseline). The other tables give p-values without naming the test tab. 2 tab. 3 tab. 4.

fig. 2 shows the NLL curves in training. The text reads them as MLE and SS converging at a relatively high NLL after about 150 epochs, while SeqGAN continues to improve § Results.

Paper vs code

The official repository is linked from the paper's synthetic-experiment section § Synthetic Data Experiments. It is owned by the first author. It contains only the synthetic-data experiment (README.md). It has no LICENSE file, so no code is quoted in this card. The code is cited by path and line only.

• Roll-out policy. The paper sets G_\beta = G_\theta [§ SeqGAN via Policy Gradient] and Algorithm 1 resets \beta \leftarrow \theta alg. 1. The code keeps \beta as an exponential average of the generator, with rate 0.8, updated once per iteration sequence_gan.py sequence_gan.py rollout.py. The embeddings are copied, not averaged rollout.py.
• Update order. The paper's order is g-steps, then d-steps, then \beta \leftarrow \theta alg. 1. The code runs one g-step, the test, the roll-out update, then the d-steps sequence_gan.py.
• Evaluation sample count. The paper evaluates 100,000 samples [§ Evaluation Metric]. The code generates generated_num = 10000 samples for each evaluation sequence_gan.py sequence_gan.py sequence_gan.py.
• Which d-steps setting. The Discussion's compared strategies use d=1 [§ Discussion]. The code uses g=1, d=5, k=3 sequence_gan.py sequence_gan.py, which is the Fig. 3(d) setting fig. 3. The paper does not say which setting produced Table 1.
• Discriminator output. The paper describes a sigmoid output [§ The Discriminative Model for Sequences]. The code uses a two-class softmax discriminator.py. For two classes these are equivalent parametrisations.
• Pre-training length. The text says about 150 epochs § Results. The code runs 120 sequence_gan.py.
• Oracle initialisation. The paper says \mathcal{N}(0,1) [§ Training Setting]. The code loads saved oracle weights sequence_gan.py target_lstm.py. The distribution behind the saved file cannot be checked from the code.
• Real-data experiments. The poem, speech and music experiments [§ Real-world Scenarios] have no code in the repository. The baselines SS and PG-BLEU are also absent, so they must be written from scratch.
• Log values. The README log is one run. It ends at NLL 8.6832 at adversarial epoch 150 README.md. That value is not a table value, and the paper does not say how many runs its table averages.

Pitfalls

• Loss scale differs between phases. Pre-training divides the loss by the number of tokens generator.py. The adversarial loss is a sum over batch and time generator.py. Adam is roughly insensitive to a constant scale, but the global-norm clip of 5.0 is not [code: generator.py#L102] generator.py. Keep the two losses as written or the effective step sizes change.
• No baseline. The reward is used directly as Q, with no subtraction of a baseline sequence_gan.py. The paper does not discuss the resulting variance.
• Roll-out cost. For T=20 and N=16, the code runs 16 \times 19 = 304 partial roll-outs and 320 discriminator passes per 64-sequence generator step rollout.py sequence_gan.py. Budget for this before comparing wall-clock times.
• Reward source. The reward is the discriminator's "real" probability rollout.py. If the generator is still close to random, the paper says the discriminator rejects its samples with high confidence, so nearly every action gets the same low reward. This is the stated reason sufficient pre-training matters app. More Ablation Study fig. 4.
• Generator steps too many per discriminator update. Fig. 3(a) uses many g-steps per d-step. The paper reports the discriminator then falls behind and gives misleading rewards § Discussion fig. 3.
• Negatives are regenerated every d-step. The code draws 10,000 fresh negatives and trains 3 epochs on each of 5 d-steps per iteration [code: sequence_gan.py#L163-L167]. The paper's bootstrapping argument for mixing several negative sets relies on this § Discussion.
• Regularisation placement. The paper says dropout and L2 are used but gives no values § Training Setting. The code applies dropout before the output layer only and L2 on the output weights only discriminator.py discriminator.py.
• Silent line filtering. The generator's data loader drops any line that does not have exactly 20 tokens dataloader.py. The discriminator's loader drops negatives of other lengths, not positives [code: dataloader.py#L54].
• Fixed length. The paper assumes fixed-length sequences. The discriminator can handle variable length through max-over-time pooling, but the synthetic and real experiments use fixed lengths § The Discriminative Model for Sequences (footnote).
• Seeds. The code seeds Python and NumPy with SEED = 88 [code: sequence_gan.py#L19] sequence_gan.py. The TensorFlow graph seed is set only inside the oracle model target_lstm.py, so the generator and discriminator initialisations are not seeded and exact runs are not expected to reproduce.
• Missing from the paper. The poem, speech and music experiments have no learning rate, batch size, roll-out count or training length. The human evaluation has no inter-rater statistics. Treat these as not stated.

Lineage

• Builds on: the GAN framework and its minimax objective, Goodfellow et al. 2014, which SeqGAN extends to discrete sequences. Policy gradient and REINFORCE are used as in § Related Work but have no arXiv card here. Monte Carlo search for values in games is cited from § Related Work.
• Compared against: random generation, MLE, scheduled sampling Bengio et al. 2015 and PG-BLEU (synthetic); MLE (real tasks).
• Superseded by: not recorded in this wiki.