---
kind: concept
slug: sequence-to-sequence-learning
title: Sequence-to-sequence learning
summary: >-
  Sequence-to-sequence learning trains an encoder-decoder to map an input sequence to an output sequence. Five cards cover text outputs, table answers, SQL with refusal, polygon vertices from a regression decoder, and grammar correction. Each reports its own benchmark, so the scores are not comparable.
tags: [nlp, vision, seq2seq, encoder-decoder, text-to-text, structured-prediction, t5, absa, table-qa, text-to-sql, referring-segmentation, grammatical-error-correction]
papers: ["2104.00369", "2110.00796", "2301.07695", "2302.07387", "2604.23627"]
related: [sentiment-analysis, sequence-labeling, semantic-parsing, table-question-answering, uncertainty-estimation, learning-with-rejection, neural-machine-translation, decoding-strategies, constrained-decoding, transfer-learning, language-model-pretraining, referring-segmentation, coordinate-classification, beam-search, data-augmentation, subword-tokenization]
status: draft
updated_at: 2026-11-06
---

## Overview

Sequence-to-sequence learning trains a model to read an input sequence and
write an output sequence, which may have a different form. An encoder reads the
input, and a decoder writes the output one step at a time. This page has five
papers. Four of them write a text output: a structured record, a table answer,
a SQL query or a corrected sentence. The fifth writes numbers, the vertices of
an object outline in an image, and it regresses them as continuous values.

- Aspect sentiment quad prediction: each review's set of (category, aspect term,
  opinion term, polarity) quads is written as one text string, and a pretrained
  T5-Base model is fine-tuned to produce that string [arxiv:2110.00796 §3.2]
  [arxiv:2110.00796 §3.3]. The output format is itself text, so a text-to-text
  model can be trained on it directly, and labels can be written as ordinary
  words instead of indices [arxiv:2110.00796 §1].
- Free-form table question answering: a question and a linearised table are
  read by T5, and the model writes a full-sentence answer [arxiv:2104.00369 §3.2].
- Text-to-SQL for electronic health records: a T5-base model writes the query
  and also reports how uncertain it was while decoding, which lets it refuse
  [arxiv:2301.07695 §4.3].
- Referring image segmentation: a Swin and BERT encoder reads the image and the
  expression, and a decoder writes the box corners and polygon vertices of the
  named object. Each vertex is a real-valued pair, not a class
  [arxiv:2302.07387 §3.1] [arxiv:2302.07387 §3.4].
- Grammatical error correction for Romanian: a Transformer reads an erroneous
  sentence and writes a corrected one. A base-size model is first trained on
  synthetically corrupted sentences, then on the gold pairs [arxiv:2604.23627
  §4.1].

The objective in the first paper is the log-likelihood of the target given the
input, summed over target positions [arxiv:2110.00796 eq. 4]. The paper gives no
optimiser, and the card for the paper records the optimiser found in the
released code. The polygon paper trains with a per-token loss that mixes
regression and classification [arxiv:2302.07387 eq. 5].

## Variants

### Linearised records with a pretrained text-to-text model

Each record is written as one sentence with fixed wording. Polarity becomes one
of three words, and a missing aspect becomes the pronoun "it". Several sentences
are joined by a separator token [arxiv:2110.00796 §3.2] [arxiv:2110.00796 eq. 1]
[arxiv:2110.00796 eq. 2]. A pretrained encoder-decoder is fine-tuned on
(input, target) pairs and decodes greedily [arxiv:2110.00796 §3.3]
[arxiv:2110.00796 §4]. Reported ASQP F1 is 46.93 on Rest15 and 57.93 on Rest16,
averaged over five seeds [arxiv:2110.00796 tab. 2]. Card:
[Aspect Sentiment Quad Prediction as Paraphrase Generation](/papers/2110.00796).

### Same format for simpler ABSA tasks

Changing the projection functions gives the same framework for aspect sentiment
triplet extraction and target aspect sentiment detection [arxiv:2110.00796 §3.4].
Reported F1 is 62.56 on Rest15 and 71.70 on Rest16 for triplets
[arxiv:2110.00796 tab. 3], and 63.06 on Rest15 and 71.97 on Rest16 for targets
[arxiv:2110.00796 tab. 4]. Both tables compare against the baselines the paper
lists for those tasks.

### Cross-task transfer before the target task

A task-specific suffix is added to the input. The model is first trained on the
two simpler tasks, then fine-tuned on the target task [arxiv:2110.00796 §3.5]
[arxiv:2110.00796 fig. 1]. The paper says that with adequate transfer data (500 ASTE and
TASD samples), 5% of the target data is competitive with 50% of the target
data alone. This comes from curves, and no table of numbers is given
[arxiv:2110.00796 §5.5] [arxiv:2110.00796 fig. 4].

### End-to-end generation from a linearised table

The table is flattened into one string of rows and cells, with separator tokens
between rows and between the question and the table. A T5 encoder-decoder reads
this string and writes a free-form answer [arxiv:2104.00369 §3.2]
[arxiv:2104.00369 fig. 6]. The alternative in the same paper is a pipeline: a
TAPAS model first selects the relevant cells, and a T5 model then writes the
answer from them [arxiv:2104.00369 §3.1]. On the test split, the end-to-end model
scores higher on sacreBLEU, and the score rises with model size. Reported values
are 21.60 for T5-small, 28.14 for T5-base and 30.54 for T5-large, against 11.00
for the fine-tuned pipeline [arxiv:2104.00369 tab. 4]. The paper's largest model
was split over eight GPUs [arxiv:2104.00369 §4.1]. Card:
[FeTaQA: Free-form Table Question Answering](/papers/2104.00369).

### Decoder entropy as a refusal signal

A text-to-SQL model answers a question by writing a query. At decoding time the
model records its per-step entropy, and it refuses when the largest entropy
over the decoded steps is above a threshold set on validation data
[arxiv:2301.07695 §4.3]. The paper's baseline is T5-base with a plain
question-to-SQL input [arxiv:2301.07695 §4.3] [arxiv:2301.07695 app. G]. With
the percentile-based threshold on MIMIC-III, the test F1 for executed answers is
89.9. Without any refusal it is 76.6, and clustering gives 83.8
[arxiv:2301.07695 tab. 5]. Card:
[EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records](/papers/2301.07695).

### Continuous coordinates from a regression decoder

The output is a sequence of tokens with a type each: a coordinate, a separator
or an end token. A classification head predicts the type. A separate small
network regresses the next coordinate as a continuous (x, y) pair, with a
sigmoid so that it lies in the unit square [arxiv:2302.07387 §3.4]
[arxiv:2302.07387 eq. 3] [arxiv:2302.07387 eq. 4]. On the input side, each
coordinate token is embedded by interpolating between the four nearest entries
of a learned 64 x 64 grid [arxiv:2302.07387 eq. 2]. The grid makes the input
embedding continuous, while the output is never rounded to a bin
[arxiv:2302.07387 §3.4]. The ablation on RefCOCO compares this with classifying
over the same bins. The regression decoder scores higher on every split, by
between 1.67 and 2.70 points [arxiv:2302.07387 tab. 4]. The authors also report
that a 32 x 32 grid and a 128 x 128 grid both score below the 64 x 64 grid on
RefCOCO [arxiv:2302.07387 tab. 6]. Card:
[PolyFormer: Referring Image Segmentation as Sequential Polygon Generation](/papers/2302.07387).

The vertex order matters more than the decoder. Sorting each polygon clockwise
and starting it at a fixed vertex gives the largest single gain in the paper's
ablation, 12.43 mIoU on RefCOCO over a random order [arxiv:2302.07387 tab. 5]
[arxiv:2302.07387 §4.4]. Polygon augmentation adds 3.72 on the same split
[arxiv:2302.07387 tab. 5]. These are ablation numbers for the authors' base model.

### Synthetic error pretraining for grammar correction

A small Transformer trained only on gold sentence pairs is the baseline. A
base-size Transformer is pretrained on about ten million Wikipedia sentences,
each corrupted by an automatic procedure, then trained further on the gold pairs
[arxiv:2604.23627 §3.2] [arxiv:2604.23627 §4.1]. The corruption replaces a
word with one of its spellchecker suggestions most of the time, and it also
deletes, inserts or swaps words. A character-level pass adds more edits
[arxiv:2604.23627 §3.2]. The paper does not say which split its results use.
The tiny baseline scores F0.5 44.38, the base model after continued training
with its optimiser state kept scores 53.76, and the same base model trained with
a reset optimiser scores 45.35. The model trained on synthetic data alone scores
16.62 [arxiv:2604.23627 tab. 5]. Beam search uses 8 beams [arxiv:2604.23627 §4.1].
Re-ranking the beams with a 5-gram language model lowers the fine-tuned score
to 49.52, or 49.83 with length normalisation [arxiv:2604.23627 tab. 5]. Card:
[Neural Grammatical Error Correction for Romanian](/papers/2604.23627).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Output has several linked parts and a fixed label set | Linearised records with a pretrained text-to-text model | On ASQP it beats the pipeline, tagging and generation baselines. The margin over the sequence-to-sequence baseline GAS is 0.95 F1 on Rest15 and 1.89 on Rest16 [arxiv:2110.00796 tab. 2]. |
| Labels are words, not arbitrary indices | Keep the label words in the target | Replacing polarity words with index symbols lowers F1 by 0.86 on Rest15 and 0.26 on Rest16. Replacing category words lowers it by 1.33 and 0.62 [arxiv:2110.00796 tab. 2]. |
| Target task has little labelled data | Cross-task transfer from the simpler tasks | The paper reports the benefit only as curves, so the size of the gain is not known [arxiv:2110.00796 §5.5] [arxiv:2110.00796 fig. 4]. |
| Output needs only triplets or target and polarity | The same format with the simpler projection | Reported F1 is in the same range as the paper's main results [arxiv:2110.00796 tab. 3] [arxiv:2110.00796 tab. 4]. |
| Output is a free-form sentence built from table content | End-to-end linearised table with a pretrained encoder-decoder | End-to-end T5-large scores 30.54 sacreBLEU against 11.00 for the pipeline on the same test split [arxiv:2104.00369 tab. 4]. The pipeline's writer is also a sequence-to-sequence model. |
| Input tables are large | Keep tables within the input window | The authors sampled tables of 3 to 34 rows and 3 to 7 columns to avoid truncating the linearised table [arxiv:2104.00369 §2.2]. |
| Output is executable code and a wrong answer costs more than no answer | Sequence-to-sequence model with entropy-based refusal | The refusal raises test F1 for executed answers from 76.6 to 89.9 on MIMIC-III [arxiv:2301.07695 tab. 5]. Refusing also means some questions get no answer, so measure coverage on your own data before use. |
| Target language has operators that a grammar-based parser cannot produce | Sequence-to-sequence model instead of the grammar parser | The authors chose a general sequence-to-sequence model because grammar parsers reject time operators such as strftime, datetime and NULL [arxiv:2301.07695 §4.3]. The Spider grammar parsed only 107 of 1,515 answerable validation questions [arxiv:2301.07695 tab. 6]. |
| Output is an ordered list of real-valued coordinates, such as a polygon | Regression decoder over continuous coordinates | Regressing beats classifying over bins by 1.67 to 2.70 points on RefCOCO in the authors' ablation [arxiv:2302.07387 tab. 4]. Start each polygon at a fixed vertex and order it, since that choice had the largest effect [arxiv:2302.07387 tab. 5]. |
| Low-resource language with a few thousand gold pairs and plenty of clean text | Synthetic error pretraining, then continued training with optimiser state kept | Fine-tuning scored 53.76 F0.5 against 45.35 for a reset optimiser and 44.38 for the tiny model trained alone [arxiv:2604.23627 tab. 5]. The synthetic data is from the authors' pipeline, and it needs a spellchecker [arxiv:2604.23627 §3.2]. |
| Grammar correction where a language model re-ranks the beams | Skip the language-model re-ranking, or tune its weight on your data | Re-ranking lowered the fine-tuned F0.5 from 53.76 to 49.52. The authors fixed the weight at 1 and did not tune it [arxiv:2604.23627 §5] [arxiv:2604.23627 tab. 5]. |
| Need a latency or memory bound | Measure it yourself | Hardware and run time are not stated for the first paper [arxiv:2110.00796 §4]. The paper's own cost remark is that the unified TASO baselines multiply the dataset size [arxiv:2110.00796 §5.1]. The polygon paper gives no memory or wall-clock figures [arxiv:2302.07387 §4.2]. |

## Implementation sketch

Language-agnostic pseudocode of the common core. The variants differ in the
projection functions and in the output head only. For triplets, the category is
replaced by "it", and for target detection, the opinion projection equals the
polarity projection [arxiv:2110.00796 §3.4].

```pseudocode
# Linearise one structured record into text  [arxiv:2110.00796 §3.2]
# Input:  record r with category, aspect (or None), opinion, polarity
# Output: one sentence (string)

def linearize(r):
    # Map the polarity to a word  [arxiv:2110.00796 eq. 1]
    word = {"POS": "great", "NEU": "ok", "NEG": "bad"}[r.polarity]
    # Replace a missing aspect with the pronoun "it"  [arxiv:2110.00796 eq. 2]
    span = "it" if r.aspect is None else r.aspect
    return f"{r.category} is {word} because {span} is {r.opinion}"
```

```pseudocode
# Build the target for one input  [arxiv:2110.00796 §3.2]
# Input:  list of records; output: one target string

def build_target(records):
    # Join the sentences with a fixed separator token  [arxiv:2110.00796 §3.2]
    return " [SSEP] ".join(linearize(r) for r in records)
```

```pseudocode
# Fine-tune a pretrained encoder-decoder  [arxiv:2110.00796 §3.3] [§4]
# Input: model with pretrained weights; pairs of (input, target)

def fine_tune(model, pairs, epochs=20):
    for _ in range(epochs):
        for x, y in batches(pairs, size=16):
            # Negative log-likelihood of the target  [arxiv:2110.00796 eq. 4]
            loss = -model.log_likelihood(x, y)
            # Learning rate 3e-4; the optimiser is not stated in the paper
            model.step(loss, lr=3e-4)
    return model
```

```pseudocode
# Predict all records for one input  [arxiv:2110.00796 §3.3]

def predict(model, x):
    # Greedy decoding, then split on the separator  [arxiv:2110.00796 §3.3]
    text = model.greedy_decode(x)
    records = []
    for piece in text.split("[SSEP]"):
        # Parse the template back into fields  [arxiv:2110.00796 §3.3]
        # A piece that does not match is null  [arxiv:2110.00796 §3.3]
        records.append(parse_template(piece.strip()))
    return records
```

Parsing is brittle. The paper's template uses the words " is " and " because ",
so an opinion or aspect span containing either word cannot be parsed back
[arxiv:2110.00796 §3.2]. The card for the paper lists this and other scoring
details.

```pseudocode
# Mixed loss for a decoder that regresses coordinates  [arxiv:2302.07387 eq. 5]
# Input:  class logits (T, 3), regressed coords (T, 2), target coords and types
# Output: scalar loss

def regression_loss(class_logits, coords, target, types):
    total = 0.0
    for t in range(T):
        # Regression applies to coordinate tokens only, selected by type
        # [arxiv:2302.07387 §3.5]
        if types[t] == COORD:
            # Box corners and polygon vertices get separate weights
            # [arxiv:2302.07387 §4.2]
            w = lambda_box if t < 2 else lambda_poly
            total += w * abs(coords[t] - target[t]).sum()
    # Label-smoothed cross-entropy over the three token types  [arxiv:2302.07387 §3.5]
    total += lambda_cls * smoothed_ce(class_logits, types)
    return total
```

```pseudocode
# Refuse when the decoder is uncertain  [arxiv:2301.07695 §4.3]
# Input:  model, question x, threshold t set on validation data
# Output: SQL string, or None when the model refuses

def answer_or_refuse(model, x, t):
    # Decode and record the entropy of the output distribution at each step
    # The paper's beam setting is in its appendix  [arxiv:2301.07695 app. G]
    sql, step_entropies = model.decode_with_entropy(x, num_beams=5)
    # Compare the largest entropy over all decoded steps with the threshold
    # [arxiv:2301.07695 §4.3]
    if max(step_entropies) <= t:
        return sql
    return None
```

The threshold is set on validation data, and the paper compares several
settings [arxiv:2301.07695 tab. 5]. A new model needs its own threshold. The
card for the paper lists the values it reports and where they come from.

## Contradictions

None known between the papers. The five papers report results on different
tasks, datasets and metrics: ASQP F1 on restaurant reviews, sacreBLEU on
Wikipedia tables, execution-based F1 on EHR databases, mIoU and Prec@0.5 on
RefCOCO-family images, and F0.5 on Romanian sentences. Their numbers are
therefore not comparable with one another. Differences between each paper and
its own released code, such as epoch counts, batch sizes, model widths and
scoring rules, are recorded on the cards and are not disagreements between
papers.

## Papers

- [Aspect Sentiment Quad Prediction as Paraphrase Generation](/papers/2110.00796): casts quad prediction as generating one fixed sentence per quad with T5-Base, and reports ASQP, triplet and target results with cross-task transfer.
- [FeTaQA: Free-form Table Question Answering](/papers/2104.00369): a dataset of free-form table answers with a pipeline baseline and an end-to-end T5 baseline on linearised tables.
- [EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records](/papers/2301.07695): a text-to-SQL benchmark over MIMIC-III and eICU, with a T5-base baseline that refuses by thresholding decoder entropy.
- [PolyFormer: Referring Image Segmentation as Sequential Polygon Generation](/papers/2302.07387): a sequence model that writes box corners and polygon vertices, regressing each vertex as a continuous value.
- [Neural Grammatical Error Correction for Romanian](/papers/2604.23627): a first Romanian correction corpus, with a Transformer trained from scratch and a base model pretrained on synthetically corrupted sentences.