Sequence-to-sequence learning

Overview

Sequence-to-sequence learning trains a model to read an input sequence and write an output sequence, which may have a different form. An encoder reads the input, and a decoder writes the output one step at a time. This page has five papers. Four of them write a text output: a structured record, a table answer, a SQL query or a corrected sentence. The fifth writes numbers, the vertices of an object outline in an image, and it regresses them as continuous values.

• Aspect sentiment quad prediction: each review's set of (category, aspect term, opinion term, polarity) quads is written as one text string, and a pretrained T5-Base model is fine-tuned to produce that string 2110.00796 §3.2 2110.00796 §3.3. The output format is itself text, so a text-to-text model can be trained on it directly, and labels can be written as ordinary words instead of indices 2110.00796 §1.
• Free-form table question answering: a question and a linearised table are read by T5, and the model writes a full-sentence answer 2104.00369 §3.2.
• Text-to-SQL for electronic health records: a T5-base model writes the query and also reports how uncertain it was while decoding, which lets it refuse 2301.07695 §4.3.
• Referring image segmentation: a Swin and BERT encoder reads the image and the expression, and a decoder writes the box corners and polygon vertices of the named object. Each vertex is a real-valued pair, not a class 2302.07387 §3.1 2302.07387 §3.4.
• Grammatical error correction for Romanian: a Transformer reads an erroneous sentence and writes a corrected one. A base-size model is first trained on synthetically corrupted sentences, then on the gold pairs [arxiv:2604.23627 §4.1].

The objective in the first paper is the log-likelihood of the target given the input, summed over target positions 2110.00796 eq. 4. The paper gives no optimiser, and the card for the paper records the optimiser found in the released code. The polygon paper trains with a per-token loss that mixes regression and classification 2302.07387 eq. 5.

Variants

Linearised records with a pretrained text-to-text model

Each record is written as one sentence with fixed wording. Polarity becomes one of three words, and a missing aspect becomes the pronoun "it". Several sentences are joined by a separator token 2110.00796 §3.2 2110.00796 eq. 1 2110.00796 eq. 2. A pretrained encoder-decoder is fine-tuned on (input, target) pairs and decodes greedily 2110.00796 §3.3 2110.00796 §4. Reported ASQP F1 is 46.93 on Rest15 and 57.93 on Rest16, averaged over five seeds 2110.00796 tab. 2. Card: Aspect Sentiment Quad Prediction as Paraphrase Generation.

Same format for simpler ABSA tasks

Changing the projection functions gives the same framework for aspect sentiment triplet extraction and target aspect sentiment detection 2110.00796 §3.4. Reported F1 is 62.56 on Rest15 and 71.70 on Rest16 for triplets 2110.00796 tab. 3, and 63.06 on Rest15 and 71.97 on Rest16 for targets 2110.00796 tab. 4. Both tables compare against the baselines the paper lists for those tasks.

Cross-task transfer before the target task

A task-specific suffix is added to the input. The model is first trained on the two simpler tasks, then fine-tuned on the target task 2110.00796 §3.5 2110.00796 fig. 1. The paper says that with adequate transfer data (500 ASTE and TASD samples), 5% of the target data is competitive with 50% of the target data alone. This comes from curves, and no table of numbers is given 2110.00796 §5.5 2110.00796 fig. 4.

End-to-end generation from a linearised table

The table is flattened into one string of rows and cells, with separator tokens between rows and between the question and the table. A T5 encoder-decoder reads this string and writes a free-form answer 2104.00369 §3.2 2104.00369 fig. 6. The alternative in the same paper is a pipeline: a TAPAS model first selects the relevant cells, and a T5 model then writes the answer from them 2104.00369 §3.1. On the test split, the end-to-end model scores higher on sacreBLEU, and the score rises with model size. Reported values are 21.60 for T5-small, 28.14 for T5-base and 30.54 for T5-large, against 11.00 for the fine-tuned pipeline 2104.00369 tab. 4. The paper's largest model was split over eight GPUs 2104.00369 §4.1. Card: FeTaQA: Free-form Table Question Answering.

Decoder entropy as a refusal signal

A text-to-SQL model answers a question by writing a query. At decoding time the model records its per-step entropy, and it refuses when the largest entropy over the decoded steps is above a threshold set on validation data 2301.07695 §4.3. The paper's baseline is T5-base with a plain question-to-SQL input 2301.07695 §4.3 2301.07695 app. G. With the percentile-based threshold on MIMIC-III, the test F1 for executed answers is 89.9. Without any refusal it is 76.6, and clustering gives 83.8 2301.07695 tab. 5. Card: EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records.

Continuous coordinates from a regression decoder

The output is a sequence of tokens with a type each: a coordinate, a separator or an end token. A classification head predicts the type. A separate small network regresses the next coordinate as a continuous (x, y) pair, with a sigmoid so that it lies in the unit square 2302.07387 §3.4 2302.07387 eq. 3 2302.07387 eq. 4. On the input side, each coordinate token is embedded by interpolating between the four nearest entries of a learned 64 x 64 grid 2302.07387 eq. 2. The grid makes the input embedding continuous, while the output is never rounded to a bin 2302.07387 §3.4. The ablation on RefCOCO compares this with classifying over the same bins. The regression decoder scores higher on every split, by between 1.67 and 2.70 points 2302.07387 tab. 4. The authors also report that a 32 x 32 grid and a 128 x 128 grid both score below the 64 x 64 grid on RefCOCO 2302.07387 tab. 6. Card: PolyFormer: Referring Image Segmentation as Sequential Polygon Generation.

The vertex order matters more than the decoder. Sorting each polygon clockwise and starting it at a fixed vertex gives the largest single gain in the paper's ablation, 12.43 mIoU on RefCOCO over a random order 2302.07387 tab. 5 2302.07387 §4.4. Polygon augmentation adds 3.72 on the same split 2302.07387 tab. 5. These are ablation numbers for the authors' base model.

Synthetic error pretraining for grammar correction

A small Transformer trained only on gold sentence pairs is the baseline. A base-size Transformer is pretrained on about ten million Wikipedia sentences, each corrupted by an automatic procedure, then trained further on the gold pairs 2604.23627 §3.2 2604.23627 §4.1. The corruption replaces a word with one of its spellchecker suggestions most of the time, and it also deletes, inserts or swaps words. A character-level pass adds more edits 2604.23627 §3.2. The paper does not say which split its results use. The tiny baseline scores F0.5 44.38, the base model after continued training with its optimiser state kept scores 53.76, and the same base model trained with a reset optimiser scores 45.35. The model trained on synthetic data alone scores 16.62 2604.23627 tab. 5. Beam search uses 8 beams 2604.23627 §4.1. Re-ranking the beams with a 5-gram language model lowers the fine-tuned score to 49.52, or 49.83 with length normalisation 2604.23627 tab. 5. Card: Neural Grammatical Error Correction for Romanian.

When to use which

Implementation sketch

Language-agnostic pseudocode of the common core. The variants differ in the projection functions and in the output head only. For triplets, the category is replaced by "it", and for target detection, the opinion projection equals the polarity projection 2110.00796 §3.4.

# Linearise one structured record into text  [arxiv:2110.00796 §3.2]
# Input:  record r with category, aspect (or None), opinion, polarity
# Output: one sentence (string)

def linearize(r):
    # Map the polarity to a word  [arxiv:2110.00796 eq. 1]
    word = {"POS": "great", "NEU": "ok", "NEG": "bad"}[r.polarity]
    # Replace a missing aspect with the pronoun "it"  [arxiv:2110.00796 eq. 2]
    span = "it" if r.aspect is None else r.aspect
    return f"{r.category} is {word} because {span} is {r.opinion}"

# Build the target for one input  [arxiv:2110.00796 §3.2]
# Input:  list of records; output: one target string

def build_target(records):
    # Join the sentences with a fixed separator token  [arxiv:2110.00796 §3.2]
    return " [SSEP] ".join(linearize(r) for r in records)

# Fine-tune a pretrained encoder-decoder  [arxiv:2110.00796 §3.3] [§4]
# Input: model with pretrained weights; pairs of (input, target)

def fine_tune(model, pairs, epochs=20):
    for _ in range(epochs):
        for x, y in batches(pairs, size=16):
            # Negative log-likelihood of the target  [arxiv:2110.00796 eq. 4]
            loss = -model.log_likelihood(x, y)
            # Learning rate 3e-4; the optimiser is not stated in the paper
            model.step(loss, lr=3e-4)
    return model

# Predict all records for one input  [arxiv:2110.00796 §3.3]

def predict(model, x):
    # Greedy decoding, then split on the separator  [arxiv:2110.00796 §3.3]
    text = model.greedy_decode(x)
    records = []
    for piece in text.split("[SSEP]"):
        # Parse the template back into fields  [arxiv:2110.00796 §3.3]
        # A piece that does not match is null  [arxiv:2110.00796 §3.3]
        records.append(parse_template(piece.strip()))
    return records

Parsing is brittle. The paper's template uses the words " is " and " because ", so an opinion or aspect span containing either word cannot be parsed back 2110.00796 §3.2. The card for the paper lists this and other scoring details.

# Mixed loss for a decoder that regresses coordinates  [arxiv:2302.07387 eq. 5]
# Input:  class logits (T, 3), regressed coords (T, 2), target coords and types
# Output: scalar loss

def regression_loss(class_logits, coords, target, types):
    total = 0.0
    for t in range(T):
        # Regression applies to coordinate tokens only, selected by type
        # [arxiv:2302.07387 §3.5]
        if types[t] == COORD:
            # Box corners and polygon vertices get separate weights
            # [arxiv:2302.07387 §4.2]
            w = lambda_box if t < 2 else lambda_poly
            total += w * abs(coords[t] - target[t]).sum()
    # Label-smoothed cross-entropy over the three token types  [arxiv:2302.07387 §3.5]
    total += lambda_cls * smoothed_ce(class_logits, types)
    return total

# Refuse when the decoder is uncertain  [arxiv:2301.07695 §4.3]
# Input:  model, question x, threshold t set on validation data
# Output: SQL string, or None when the model refuses

def answer_or_refuse(model, x, t):
    # Decode and record the entropy of the output distribution at each step
    # The paper's beam setting is in its appendix  [arxiv:2301.07695 app. G]
    sql, step_entropies = model.decode_with_entropy(x, num_beams=5)
    # Compare the largest entropy over all decoded steps with the threshold
    # [arxiv:2301.07695 §4.3]
    if max(step_entropies) <= t:
        return sql
    return None

The threshold is set on validation data, and the paper compares several settings 2301.07695 tab. 5. A new model needs its own threshold. The card for the paper lists the values it reports and where they come from.

Contradictions

None known between the papers. The five papers report results on different tasks, datasets and metrics: ASQP F1 on restaurant reviews, sacreBLEU on Wikipedia tables, execution-based F1 on EHR databases, mIoU and Prec@0.5 on RefCOCO-family images, and F0.5 on Romanian sentences. Their numbers are therefore not comparable with one another. Differences between each paper and its own released code, such as epoch counts, batch sizes, model widths and scoring rules, are recorded on the cards and are not disagreements between papers.

Papers

• Aspect Sentiment Quad Prediction as Paraphrase Generation: casts quad prediction as generating one fixed sentence per quad with T5-Base, and reports ASQP, triplet and target results with cross-task transfer.
• FeTaQA: Free-form Table Question Answering: a dataset of free-form table answers with a pipeline baseline and an end-to-end T5 baseline on linearised tables.
• EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records: a text-to-SQL benchmark over MIMIC-III and eICU, with a T5-base baseline that refuses by thresholding decoder entropy.
• PolyFormer: Referring Image Segmentation as Sequential Polygon Generation: a sequence model that writes box corners and polygon vertices, regressing each vertex as a continuous value.
• Neural Grammatical Error Correction for Romanian: a first Romanian correction corpus, with a Transformer trained from scratch and a base model pretrained on synthetically corrupted sentences.