wikillm

Paper · 2017-05-30 · cs.AI, cs.CR, cs.MM

Generating Steganographic Text with LSTMs

Hides an encrypted bit string in text written by a word-level LSTM. A shared key splits the vocabulary into 2^b bins, and at each step the LSTM picks its most likely word from the bin for the next b-bit block. With 4 bins the capacity is 2 bits per word; perplexity rises with more bins. Author-reported: 32 bits per tweet, against 2.8 for the best earlier Twitter stegosystem.

Linguistic steganographyLong short-term memory (LSTM)steganographytext-generationlstmlanguage-modelprivacy

arXivPDFMarkdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Ordinary encryption makes a message look like scrambled noise, and that can draw attention. This paper hides a secret message inside text that reads like a normal tweet or email. Sender and receiver share a secret table that splits the vocabulary into groups of words.

The sender writes the text one word at a time with a language model. Each small piece of the secret says which group the next word must come from. The model then picks its most likely word from that group. The receiver looks up which group each word belongs to and rebuilds the secret. Decoding needs only the table, not the model.

Because the words come from a trained language model rather than from edits to an existing text, the output can read naturally. More groups hide more bits per word, but the text becomes less likely under the model. Common words can be added to every group to make the text read better, at the cost of hidden bits. The authors judge readability by perplexity and by reading samples. Formal tests against human readers and against detectors are left to future work.

TL;DR

  • Linguistic stegosystem: the secret is embedded in the choice of words, and the output is generated by a word-level LSTM, not edited from a cover text §1 §2.2.
  • A shared key partitions the vocabulary into 2^b disjoint bins. Each b-bit block of the secret selects a bin, and the LSTM must emit a word from that bin §2.2 §3.1 tab. 1.
  • Trained on Twitter (600k tweets) and on the first 100 MB of cleaned Enron email bodies §4.1.
  • Author-reported: 4 bins without common tokens give 2 bits per word and 32 bits per tweet, against 2.8 bits per tweet for the Twitter stegosystem CoverTweet's successor cited there §4.4 §5.
  • Only Twitter has perplexity and capacity numbers; the email results are sample passages, and no security or human-detection test was run §4.3.2 §5.

Problem

Users who encrypt messages can make themselves conspicuous, since encrypted traffic stands out in a stream of plain text §1. The paper wants messages whose content and existence an observer can conceal, in ordinary communication channels such as tweets and email §1.

Earlier linguistic stegosystems modify an existing cover text by synonym or paraphrase substitution §2.1. Their capacity is bounded by the number of available substitutions, and they can make the text unnatural §2.1 §1. Matryoshka generates candidates with an n-gram model and relies on a human to polish them §2.1. The authors state that they know of no earlier neural-network text stegosystem §2.1.

Method

Overview

The sender encodes a secret bit string into a list of words. Each block of b secret bits picks one of 2^b bins, and a word-level LSTM emits one word from that bin. The output is the stegotext, used directly without human polishing §2.2 §1. Decoding looks up each word's bin in the shared key §2.2.

Components §2.2

  1. Secret data. The secret is compressed and/or encrypted into a bit string S. S is cut into blocks of |B| bits. If |B| does not divide |S|, the remainder is left out of the encryption §2.2 [fn. 1, §2.2].
  2. Key. The vocabulary V holds every token that may appear, which can include punctuation. V is partitioned at random, without replacement, into 2^|B| disjoint bins, each with |V| / 2^|B| tokens. The mapping from bit block to bin is the shared key §2.2 tab. 1.
  3. Embedding. The LSTM emits one token per bit block, chosen from that block's bin. The stegotext therefore has as many tokens as there are blocks §2.2 tab. 2.
  4. Decoder. For each token, find its bin in the key and recover the block's bits. The paper calls this deterministic §2.2.
  5. Common-token variant. A set C of common tokens is added to every bin. When the LSTM picks a common token, it picks again from the same bin until it picks a non-common token. The decoder removes all common tokens before decoding §2.2.

Worked example with the 4-bin key of tab. 1: the bit string 1000011011 splits into the blocks 10, 00, 01, 10, 11. The words chosen are I, am, attaching, an, NDA tab. 2.

Pseudocode
# Embed secret bits as stegotext  [§2.2] [§3.1]
# Input:  secret bits (list of 0/1); block size b; bins (2^b token sets);
#         common tokens C (empty for the plain variant)
# Output: list of tokens (stegotext)

def embed(secret_bits, b, bins, common_tokens, lstm):
    # Cut the secret into b-bit blocks; a remainder is left out  [§2.2]
    blocks = chunk_bits(secret_bits, b)
    stegotext = []
    for block in blocks:
        # The bin for this block comes from the shared key  [§2.2]
        bin_tokens = bins[block_to_index(block)]
        # The LSTM may only emit W_B, or W_B plus C in the variant  [§3.1]
        allowed = bin_tokens | common_tokens
        token = lstm.select_restricted(allowed, stegotext)
        # Common tokens carry no bits; draw again until a non-common one  [§2.2]
        while token in common_tokens:
            stegotext.append(token)
            token = lstm.select_restricted(allowed, stegotext)
        stegotext.append(token)
    return stegotext
Pseudocode
# Recover secret bits from stegotext  [§2.2]
# Input:  stegotext tokens; bins; common tokens C; block size b
# Output: bit string

def decode(stegotext, bins, common_tokens, b):
    # Common tokens sit in every bin, so remove them first  [§2.2]
    payload = [t for t in stegotext if t not in common_tokens]
    bits = []
    for token in payload:
        # Each non-common token sits in exactly one bin  [§2.2]
        index = find_bin_index(token, bins)
        # Write the bin index as b bits  [§2.2]
        bits.extend(index_to_bits(index, b))
    return bits

Model: restricted softmax §3.1

A classic LSTM gives each vocabulary word a probability with a softmax over its output vector, and picks the word with the highest probability. The softmax is written as an unnumbered display in §3.1. The modified LSTM sets the probability of every word outside the allowed set to zero, so the softmax chooses within the set:

  • without common tokens: P[x = w_j] = 0 for every j not in W_B;
  • with common tokens: P[x = w_j] = 0 only for j not in W_B ∪ C §3.1.

The text says the LSTM selects the highest-probability word within W_B and also calls the selection a "multinomial softmax" §3.1. The paper does not say whether the choice is greedy or sampled (see Pitfalls).

Training schedule §4.2

The paper gives the optimiser, batch size and learning-rate rule, but no full algorithm. The schedule below is our reading of §4.2. The decay operation is our interpretation of "decay factor 4" and is not stated in the paper.

Pseudocode
# Language-model training schedule  [§4.2]
# Input:  training token stream; validation token stream
# Output: trained LSTM weights

def train_language_model(train_tokens, valid_tokens):
    # Initial SGD learning rate of 20  [§4.2]
    learning_rate = 20.0
    best_valid_loss = infinity
    # Train until validation loss converges: 12 epochs for Twitter  [§4.2]
    for epoch in range(max_epochs):
        # SGD with batch size 20  [§4.2]
        sgd_epoch(train_tokens, batch_size=20, lr=learning_rate)
        valid_loss = evaluate(valid_tokens)
        # Decay only when validation loss did not improve  [§4.2]
        if valid_loss >= best_valid_loss:
            # "decay factor 4" read as divide by 4 (not stated)  [§4.2]
            learning_rate = learning_rate / 4.0
        best_valid_loss = min(best_valid_loss, valid_loss)

Metrics §3.2

  • Perplexity is defined as exp(−(1/N) Σ_i ln p[w_i]) over the validation data §3.2. For the steganographic model, the word probability is replaced by p[w_i] = Σ_B p[w_i,B] / 2^|B|, which averages over all bit-block values under a uniform-block assumption. The paper justifies this by the law of large numbers §3.2. The exact computation is not given (see Pitfalls).
  • Capacity is encrypted bits per output word. Without common tokens it equals |B| §3.2. With common tokens, if a fraction p of the output words are common tokens, capacity is (1 − p) · |B| §3.2.

Hyperparameters

NameValueWhereNotes
Bin count (tweet experiments)1, 2, 4, 8§4.3.12^|B| bins, so |B| = 0, 1, 2, 3 bits per word
Bins (email samples)1, 2, 4tab. 5 §4.3.2samples from each bin; no capacity or perplexity reported
Bins used for comparison4, no common tokens§4.42 bits per word
Common tokens C, Twitter10 most frequent tokens of the Twitter corpus§4.3.1the token list is not given; email: not stated
Vocabulary, Twitter225 thousand§4.1
Vocabulary, Enron406 thousand§4.1
Corpus, Twitter600 thousand tweets; 45 million words§4.1tokenised with the NLTK tweet tokeniser §4.1
Corpus, Enronfirst 100 MB of cleaned bodies; 16.8 million tokens§4.1bodies extracted with the cleaning method of §4.1
Text normalisation, Twitterusernames → <user>, URLs → <url>, all lower case§4.1 §4.3.1
Word embedding dimension200, trained from scratch§4.2no pretrained embeddings
OptimiserSGD§4.2
Batch size20§4.2
Initial learning rate20§4.2
Learning-rate decayfactor 4 per epoch, only when validation loss does not improve§4.2"divide by 4" is our reading
Twitter LSTM2 layers, 600 units§4.2
Twitter unroll length25 steps§4.2
Twitter gradient clippingnorm 0.25§4.2Enron: not stated
Twitter dropout20 %§4.2
Twitter training length12 epochs (about 10 hours)§4.2stopped on validation-loss convergence
Enron LSTM3 layers, 600 units§4.2
Enron unroll length20 steps§4.2
Enron regularisation"no regularization"§4.2dropout and clipping not separately stated
Enron training length6 epochs (about 2 days)§4.2
Hardwareone NVIDIA GeForce GTX TITAN X§4.2
Sampling / temperaturenot stated§3.1
Stop rule, message length, end markernot stated§2.2
Key-distribution methodnot stated§2.2

Results

Author-reported. Setup: the Twitter LSTMs described above; perplexity is computed with the estimator in §3.2 on validation data, the split sizes are not stated §3.2. The paper calls these experiments "preliminary" §1 §4.

Table 3 tab. 3: capacity and perplexity on Twitter.

Binsbpw, originalppl, originalbpw, common tokensppl, common tokens
1 (non-steganographic)0134.730134.73
21190.840.65171.35
42381.21.17277.55
83833.111.53476.66

The authors read this as capacity rising with the bin count while perplexity rises too, and common tokens cut perplexity at the cost of capacity tab. 3.

Qualitative results, no numbers reported:

  • Tweets: examples at 2, 4 and 8 bins, each marked good, average or poor, with and without common tokens tab. 4. The authors judged about 60 random tweets per category, using grammar, semantic coherence and resemblance to real tweets, and picked tweets at the 25th, 50th and 75th percentile §4.3.1 (footnote on tab. 4). Common tokens make the tweets simpler and improve grammar and coherence, most in the poor examples tab. 4.
  • Emails: sample passages from 1, 2 and 4 bins tab. 5. The authors say context inconsistency appears in every bin, but the text stays syntactically sound as bins increase tab. 5 §4.3.2.

Comparison with other stegosystems §4.4, using the 4-bin model without common tokens:

QuantityValueSource
Capacity, this paper (4 bins, no common tokens)2 bits per word; 32 bits per tweet§4.4 §5
Assumed tweet length16.04 words, from a sample of 2 million tweets§4.4 footnote
Cover-modification stegosystems1–2 bits per sentence§4.4, citing Chang and Clark (2012)
State-of-the-art Twitter stegosystem2.8 bits per tweet§4.4, citing Wilson et al. (2016)

The authors state that their 32 bits per tweet is over 11 times that of the Twitter stegosystem §4.4. Table 6 places three "average" tweets from each system side by side, and the authors say the subjective quality looks comparable. This is a hypothesis: formal experiments are needed §4.4 tab. 6.

Paper vs code

No official code.

The paper names no code repository. It says that the authors "aim to open-source" the stegosystem §5, which is future tense. The GitHub searches on 2026-11-13 found nothing from the authors:

  • repository search for "stegotext": 11 results, none owned by an author;
  • repository search for "stegosystem": 14 results, none owned by an author;
  • repository search for "steganographic lstm": one result, a personal repository with the paper's title and an owner who is not an author. It is a third-party reimplementation, so it is not used;
  • repository search for "steganography lstm" inside the EPFL machine-learning org: no results.

No code citations are therefore possible, and no code-level check of the hyperparameters or the sampling rule is possible.

Pitfalls

  • Greedy or sampled. Section 3.1 says the LSTM picks "the word with the highest probability", and it also says "multinomial softmax" §3.1. Greedy choice makes the output a deterministic function of the key and the context. Sampling would give variety but is not described. Choose one, and record which you chose. The security claims depend on this, and the paper does not test them.
  • Remainder bits. Bits that do not fill a whole block are dropped §2.2 [fn. 1, §2.2]. The paper does not say how the receiver learns the message length or where it ends. Length framing, padding and an end marker are all unstated.
  • Common tokens are in every bin. A common token belongs to all 2^b bins, so the decoder must strip common tokens before the bin lookup §2.2. The variant emits a variable number of tokens per block, so the output length no longer equals the number of blocks §2.2 §3.2.
  • Divisibility of the vocabulary. The bins are said to hold |V| / 2^|B| tokens each §2.2. The paper does not say what happens to tokens left over when the division is not exact.
  • Perplexity estimator is ambiguous. Taken literally, if p[w_i,B] is zero for every block whose bin does not contain w_i §3.1, the average over 2^|B| blocks divides the one non-zero term by 2^|B|. That would multiply every word's perplexity contribution by 2^|B|. The reported ratios do not follow a clean power of two: the 2-bin perplexity is 190.84 / 134.73 ≈ 1.42 times the 1-bin value tab. 3. The exact estimator is therefore unknown, and the 1-bin value is the only point of comparison that can be trusted. Do not compare these numbers with perplexities from other papers.
  • Perplexity context. The paper does not say how the context is set when averaging over bit blocks. The natural choice is the true preceding words of the validation text, but this is not stated §3.2.
  • Email results have no metrics. The Enron results are sample passages only. No perplexity, no capacity and no common-token variant is reported for email §4.3.2 tab. 5.
  • Human judgement is not controlled. The tweet quality judgement is manual, with about 60 tweets per category, selected by percentile and filtered for offensive content §4.3.1. The filter's rule and the raters are not described.
  • Detection is untested. The paper states that it has not tested security against human judges or steganalysis; it calls these future work §4.4 §5.
  • The 32-bit figure mixes data. 32 bits per tweet multiplies 2 bits per word by an assumed 16.04 words per tweet, measured on a separate sample of 2 million tweets §4.4 footnote. The number is an estimate, not a measured stegotext length.
  • Normalisation changes the output. Tweets are lower-cased in preprocessing §4.3.1. The authors suggest post-processing capitalisation but did not apply it §4.3.1. Usernames and URLs are replaced by <user> and <url> tokens. For display they were replaced by mock names, and the paper suggests random followers or followees in practice §4.3.1.
  • Blocked content. Retweets starting with "RT" can give away the cover. The paper suggests removing them from training or from generation, but did not test this §4.3.1.
  • Training cost. The Twitter model took about 10 hours and the Enron model about 2 days on one GPU §4.2. Each corpus needs its own trained model.
  • Decoder needs no model. Decoding uses only the key and the token list §2.2. The LSTM is needed only by the sender, so the receiver does not need the training corpus.

Lineage

  • Builds on: LSTM language models §2.2 §2.2 "Our proposal", with the LSTM as the state of the art in text generation citing Józefowicz et al. (2016) §2.2; the word-level LSTM of Mikolov et al. (2010) for the modified embedding §2.2. See long-short-term-memory.
  • Builds on, stegosystem framework: the building blocks of a stegosystem from Salomon (2003) §2.2.
  • Compared against: CoverTweet, the cover-modification Twitter stegosystem of Wilson et al. (2014) §2.1 §4.4; the Twitter stegosystem of Wilson et al. (2016) §4.4; cover modification by Chang and Clark (2012) §4.4.
  • Related systems not compared quantitatively: Grosvald and Orgun (human-written stegotext) and Matryoshka (Safaka et al., 2016) §2.1.
  • Future work named by the authors: formal human and steganalysis tests §5; Matryoshka-style human polishing of the output §2.2 §5; probabilistic encoders and decoders to improve capacity §5; personalised generation for user types or interest groups §5.
  • Superseded by: not stated in this card.

Source and license

Adapted from “Generating Steganographic Text with LSTMs” by Tina Fang, Martin Jaggi, Katerina Argyraki (arXiv:1705.10742), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.