Aspect Sentiment Quad Prediction as Paraphrase Generation

In plain words

A product review often says several things at once: what the reviewer is talking about, the exact words they use for it, the opinion they express, and whether that opinion is good or bad. Earlier systems usually picked out only one or two of these pieces, or chained several small models together, so a mistake in an early step spoiled the later ones.

This paper asks a model to produce all four pieces in one go. The trick is to write each set of pieces as a plain English sentence, such as "food quality is great because pasta is delicious", and then train a text-to-text model to write that sentence from the review. Because the target is ordinary language, a pretrained language model already knows much of what the sentence means, and the labels themselves carry meaning, such as "great" for a positive opinion. At test time the model writes the sentences, and a short parser turns them back into the four-part records.

The authors report that this works better than the earlier pipelines and earlier generation-based methods on two restaurant-review datasets. They also show that the same format can be used for two simpler related tasks, and that training on those tasks first helps when labelled data is scarce.

TL;DR

For each review, predict every (aspect category, aspect term, opinion term, polarity) quad. The method linearises each quad into a fixed sentence, fine-tunes T5-Base on sentence pairs, decodes greedily and parses the result back into quads. Author-reported ASQP F1 is 46.93 (Rest15) and 57.93 (Rest16) tab. 2. Code is released, but it does not run as committed (see Paper vs code).

Problem

ASQP asks for all sentiment quads of a sentence. Each quad has an aspect category from a fixed set, an aspect term (a span of the sentence, or null when the target is implicit), an opinion term (a span of the sentence) and a polarity in {POS, NEU, NEG} §3.1. Pipeline solutions suffer from error propagation, and classification-style sub-tasks treat labels as indices, which discards their meaning §1. Predicting the whole quad at once, with generation, is the motivation §1.

Method

The model is a sequence-to-sequence Transformer that maps the input sentence x to a target sequence y that contains all quads of x §3.2. It is initialised from a pretrained encoder-decoder (T5) and fine-tuned on (x, y) pairs §3.3.

Target construction. Each quad q = (c, a, o, p) becomes one sentence of the form "P_c(c) is P_p(p) because P_a(a) is P_o(o)" §3.2. The projection functions are:

• Category c and opinion o are kept as written §3.2.
• Polarity: POS becomes "great", NEU becomes "ok", NEG becomes "bad" §3.2 eq. 1. The paper also allows a dataset-specific choice of opinion word per polarity, taken from the most common opinion term of that polarity §3.2. The released code uses the fixed mapping data_utils.py.
• Aspect: a null aspect becomes the pronoun "it", otherwise the term is kept §3.2 eq. 2.
• Several quads of one sentence are joined with the separator token "[SSEP]" §3.2.

Training. Fine-tuning maximises the log-likelihood of the target given the input, summed over target positions §3.3 eq. 4. The paper does not give an optimiser; the code uses AdamW (see Hyperparameters).

Inference. Decoding is autoregressive and greedy §3.3 §4. The generated string is split on "[SSEP]", and each piece is parsed back into a quad by the template above. A piece that does not match the template is treated as a null prediction §3.3.

# Linearise one quad into a sentence  [§3.2] [eq. 1] [eq. 2]
# Input:  category c, aspect a (None if implicit), opinion o, polarity p
# Output: one sentence (string)

def linearize_quad(c, a, o, p):
    # Map the polarity to an opinion word: POS, NEU, NEG  [eq. 1]
    p_word = {"POS": "great", "NEU": "ok", "NEG": "bad"}[p]
    # Replace an implicit aspect by the pronoun "it"  [eq. 2]
    a_word = "it" if a is None else a
    # Category and opinion are already text, so keep them  [§3.2]
    return f"{c} is {p_word} because {a_word} is {o}"

# Build the target sequence for one sentence  [§3.2]
# Input:  list of quads (c, a, o, p)
# Output: target string

def build_target(quads):
    # Linearise each quad, then join with the separator  [§3.2]
    sentences = [linearize_quad(*q) for q in quads]
    return " [SSEP] ".join(sentences)

# Fine-tune a pretrained T5-Base on (input, target) pairs  [§3.3] [§4]
# Input:  pretrained parameters theta; training pairs (x, y)
# Output: fine-tuned parameters theta

def fine_tune(theta, pairs):
    # Train for 20 epochs over the training split  [§4]
    for epoch in range(20):
        # Shuffle and form mini-batches of 16 pairs  [§4]
        for batch_x, batch_y in batches(pairs, size=16):
            # Negative of the summed token log-likelihood  [§3.3] [eq. 4]
            loss = -log_likelihood(theta, batch_x, batch_y)
            # Gradient step at learning rate 3e-4; optimiser not stated  [§4]
            theta = optimizer_step(theta, loss, lr=3e-4)
    return theta

# Predict all quads of one sentence  [§3.3]
# Input:  fine-tuned parameters theta; sentence x
# Output: list of quads (c, a, o, p)

def predict_quads(theta, x):
    # Greedy decoding: the most probable token at each step  [§3.3] [§4]
    generated = greedy_decode(theta, x)
    quads = []
    # Split the generated text on the separator  [§3.3]
    for piece in generated.split("[SSEP]"):
        # Parse "c is w because a is o"; otherwise the quad is null  [§3.3]
        parsed = parse_template(piece.strip())
        if parsed is None:
            quads.append(None)
            continue
        c, p_word, a_word, o = parsed
        # Map the opinion word back to a polarity  [§3.2] [eq. 1]
        p = {"great": "POS", "ok": "NEU", "bad": "NEG"}[p_word]
        # Map the pronoun "it" back to a null aspect  [§3.2] [eq. 2]
        a = None if a_word == "it" else a_word
        quads.append((c, a, o, p))
    return quads

Figures: Figure 1 shows the framework and notes that the underlined task identifier in the input is used only for cross-task transfer fig. 1. Figure 2 shows two examples of target construction for ASQP fig. 2.

Extension to other ABSA tasks. Changing the projection functions gives the same framework for aspect sentiment triplet extraction (ASTE; aspect, opinion, polarity) and target aspect sentiment detection (TASD; category, aspect, polarity). For TASD the opinion projection equals the polarity projection; for ASTE the category is mapped to "it" §3.4.

Cross-task transfer. A task-specific suffix (for example "ASQP") is added to the input. The model is trained on ASTE and TASD first, then fine-tuned on ASQP §3.5 fig. 1.

Hyperparameters

Dataset statistics tab. 1, as sentences (S) and quads by polarity (positive, neutral, negative):

The datasets are built from SemEval restaurant reviews. Opinion terms come from an existing ASTE dataset and categories from an existing TASD dataset; the authors merged them and annotated the missing quads. Each sample was annotated by two annotators, with conflicts checked §4. The repo's ASQP files have the sentence counts of this table train.txt, train.txt.

Results

Author-reported. Setup: T5-Base fine-tuned per dataset, scores averaged over five seeds, micro precision, recall and F1 over quads, with exact match of all four elements §4 tab. 2.

ASQP main results tab. 2:

The label-semantics rows replace the words for polarity and/or category with index symbols (SP1..SP3, AC1..); the paper's text describes this §5.2.

ASTE results, F1 tab. 3:

TASD results, F1 tab. 4:

Error analysis §5.4 fig. 3: the most frequent error is the opinion term, because the model often does not produce the exact span. Category errors come from similar categories such as food quality and food style options. Polarity errors are mostly positive versus neutral. Predictions outside the defined label vocabulary are rare. Of 100 sampled dev sentences per dataset, the paper counts the error types §5.4 fig. 3.

Cross-task transfer §5.5 fig. 4: "adequate" transfer uses 500 ASTE and TASD samples, and "scanty" transfer uses 100 §5.5. The paper says that with adequate transfer data, 5% of the ASQP data is competitive with training on 50% of ASQP data alone. This is a description of curves, and no table of numbers is given fig. 4.

Paper vs code

Repository: https://github.com/IsakZhang/ABSA-QUAD at commit ce4d4a51fc5b1abf67a315e8d15296880af3c326, linked from the paper's footnote 1 §1. The repository has no LICENSE file, and the readme gives no license. So the code is cited by path and line only, and no code is quoted.

The released code does not run as committed. get_transformed_io calls read_line_examples_from_file(data_path) with one argument data_utils.py, but the function is defined with two required parameters, data_path and silence data_utils.py. The sanity check in main.py builds the dataset before training and reaches this call main.py. Both main.py calls to the reader also pass one argument main.py, main.py. Verified by reading; I did not execute the code.

Other differences, each cited:

• Epochs. Paper: 20 §4. run.sh passes 20 run.sh. The default in main.py is 30 main.py.
• Optimiser and schedule. Not in the paper. Code: AdamW with linear decay, no warmup, gradient clipping at 1.0, weight decay 0.0 main.py main.py main.py.
• Loss. The paper writes eq. 4 as a sum over target positions §3.3. The code passes labels to the model's built-in loss and takes outputs[0] main.py. Pad positions are set to -100 main.py. The code does not compute the sum itself; whether the library loss is a sum or a mean is not checked here.
• Model selection. The paper does not say. Code keeps the final epoch, saving is commented out, and dev loss is only logged main.py main.py main.py.
• Beam search. Paper: beams 3, 5 and 8 tried, greedy used §4. Code: greedy, with num_beams=8 commented out main.py.
• Task switch. --task is parsed main.py, but get_transformed_io hard-codes task = 'asqp' data_utils.py. The ASTE and TASD target builders exist data_utils.py but are unreachable from the entry point. For ASTE the paper writes the opinion projection as "it" §3.4; the code puts "It" in the category slot of the template, and the polarity word in the polarity slot data_utils.py.
• Missing parts. The repo has only the rest15 and rest16 ASQP files train.txt, train.txt. No ASTE or TASD data, no transfer code, no task suffix (inputs are the raw sentence data_utils.py), and no code for the index-symbol ablation. The variable numopinion2word with SP1..SP3 is defined but unused eval_utils.py.
• Evaluation counts. A quad is counted correct only if all elements match §4. The code counts each predicted quad that appears in the gold list as a hit, with no one-to-one matching, so repeated predicted quads each count eval_utils.py. A failed parse becomes an empty quad that is counted in the prediction total eval_utils.py eval_utils.py. The paper says a failed parse is null §3.3.
• Gold strings. Evaluation does not read the gold quads from the data files. It decodes the tokenised, 128-token-truncated targets back to text and parses them main.py main.py. Gold and predictions both go through the same tokenizer and parser.
• Polarity words. In the ASQP branch the parser keeps the raw word (great, ok, bad) and does not map it to POS, NEU or NEG, unlike the TASD branch eval_utils.py vs eval_utils.py. Gold goes through the same path, so the words are compared consistently.
• Test batch. Evaluation uses batch size 32, hard-coded for test main.py, whereas the paper gives no evaluation batch size.
• Requirements. Code pins transformers 4.0.0, sentencepiece 0.1.91 and pytorch_lightning 0.8.1 requirement.txt readme.md. The paper gives no versions.
• Consistent parts. Dataset sizes match Table 1 tab. 1. The target template, the polarity words, the "it" pronoun and the "[SSEP]" separator match §3.2 data_utils.py data_utils.py.

Pitfalls

• The code is not runnable as released. Re-implement the parser and the data reader yourself, following the paper's templates, and treat the code as a reference only.
• The parser is brittle. It splits on the exact strings " because " and " is ", and each split must give exactly two parts. An aspect or opinion span that contains " is " or " because " makes the whole quad null eval_utils.py. The separator must also appear exactly as "[SSEP]" in the decoded text eval_utils.py.
• Polarity words are fixed. The model must produce exactly great, ok or bad. "good", "okay" or "bad." with a period is scored as wrong by the code's exact matching. Use the same words in training and parsing.
• Pronoun collision. Any predicted aspect equal to "it" in any case becomes a null aspect eval_utils.py. A real aspect term "it" would be scored as null.
• Duplicates inflate precision. The released scorer does not match each gold quad to one prediction. Use one-to-one matching if you need a strict score.
• Failed parses cost precision. A malformed piece counts as a prediction in the total, as described above. The paper's "null" wording suggests it should not count.
• Truncation is silent. Inputs and targets are truncated at 128 tokens data_utils.py. Long reviews with many quads lose quads without a warning, and the gold side is truncated the same way at evaluation. The paper does not state the length, nor whether any sample exceeds it.
• Tokenizer round-trip. Gold and predicted strings are decoded from T5 token ids main.py. Any character the tokenizer cannot represent is changed the same way on both sides, so scores may differ from the raw labels in the data files. The size of this effect is not measured here.
• Hard element: opinion span. The paper names exact opinion-span matching as the most common error §5.4 fig. 3. Expect the main gains and losses there.
• Neutral class is small. Neutral quads are 34 (Rest15) and 62 (Rest16) in training tab. 1. The paper names positive versus neutral confusion as the main polarity error §5.4.
• Cost. Each dataset is trained for 20 epochs on T5-Base §4. The Unified TASO baselines multiply the dataset by |V_c| x |V_p| §5.1, which the paper cites as their computation cost.
• Results are author-reported and come from five seeds with no stated variance. The paper gives no significance test.
• Gold data order. The data file orders each quad as (aspect term, category, polarity, opinion term) train.txt, which differs from the paper's (c, a, o, p) order.

Lineage

• Builds on: T5 §3.3 as the pretrained encoder-decoder (no arXiv id given in the paper, and no card yet); the paraphrase framing follows earlier work on casting NLP tasks as generation §1.
• Compared against: the pipeline models HGCN and BERT-based span models §4; TAS-BERT, the unified tagging model of §4; GAS, the sequence-to-sequence baseline that writes the quad sequence directly §4; and, for ASTE and TASD, Jet, GTS, Two-Stage, MEJD and others §5.3 tab. 3 tab. 4.
• Superseded by: not stated in this card.