Exploiting BERT for End-to-End Aspect-based Sentiment Analysis

In plain words

Product reviews often praise one part and criticise another. This paper looks at one job for such reviews: in a single pass, find the words that name the thing being judged (the aspect, such as "service") and say whether the opinion about it is positive, negative or neutral.

Earlier systems often used word vectors that give each word one fixed meaning, so the same word gets the same vector whatever the sentence. This paper puts BERT, a language model pre-trained on a large amount of text, underneath a small output layer and fine-tunes the whole stack on the review data. The authors try several output layers, from a plain linear one to a conditional random field that scores whole label sequences.

The notable result is how well the simplest version works: a plain linear layer on top of BERT beats the earlier systems the paper compares against. The authors also argue for a stricter test setup, where a held-out development set is used only to pick the checkpoint, which earlier work often skipped.

TL;DR

The model feeds BERT-base token states into one task layer and tags each word with a BIEOS-style label that marks aspect spans and their sentiment §2. A linear layer alone scores above the best non-BERT baseline on both datasets (60.43 vs 60.35 F1 on LAPTOP, 73.22 vs 72.78 on REST) tab. 2. The best variants are a GRU on LAPTOP (61.12) and self-attention on REST (74.72) tab. 2. All numbers are author-reported. The REST data is not the released REST data, so do not compare them with the repository's re-run numbers README.md.

Problem

Aspect-based sentiment analysis (ABSA) usually assumes the aspect terms are already given, which makes it less useful in practice, since annotating aspects by hand is expensive §1. The end-to-end setting (E2E-ABSA) asks the model to find the aspects and their sentiment together §1 tab. 1.

Earlier neural models mostly used task-agnostic embeddings (Word2Vec, GloVe), which give one context-independent vector per word, and the datasets are small §1. Earlier work had coupled contextual embeddings with models for the simpler classification task; one work used BERT for E2E-ABSA but focused on a task-specific architecture §1. The paper asks how much a pre-trained BERT encoder helps with simple heads on E2E-ABSA §1. It also argues that many ABSA works skip a held-out development set for model selection, and uses one here §1 §3.1.

Method

The model has two parts. BERT (12 transformer layers) produces a contextual state for each input token §2.1 eq. 1. A single task-specific layer then maps those states to tags, with one of five heads: linear, GRU, self-attention network (SAN), transformer layer (TFM), or linear-chain CRF §2.2. The tag set is B, I, E, S (begin, inside, end, single) crossed with POS, NEG and NEU, plus O for outside any aspect, 13 tags in total §2.

The paper's formulation, in its own notation:

# E2E-ABSA tagger: BERT encoder plus one task-specific layer  [§2]
# Input:  word-piece ids of one sentence, T tokens  [§2.1]
# Output: one tag per token, from the BIEOS tag set  [§2]

def tag_sentence(token_ids):
    # 1. Contextual states from the L = 12 transformer layers  [§2.1] [eq. 1]
    hidden = bert(token_ids)                  # (T, 768)

    # 2. Task-specific layer: linear, GRU, SAN, TFM or CRF  [§2.2]
    features = task_layer(hidden)             # (T, 768)

    # 3. Score every token against every tag  [§2.2] [eq. 2] [eq. 4]
    scores = features @ W_o + b_o             # (T, 13)

    # 4. Greedy choice per token; CRF heads use Viterbi instead  [§2.2]
    return argmax(scores, axis=-1)

The BERT encoder is fine-tuned together with the task layer; the paper also reports a run with BERT frozen §3.4.

# GRU head, one time step; layer norm on every gate input  [§2.2] [eq. 3]
def gru_step(x_t, h_prev):
    # Reset and update gates from the input and the previous state  [eq. 3]
    gates = sigmoid(ln(W_x @ x_t) + ln(W_h @ h_prev))
    reset, update = split(gates, 2)           # (d,), (d,)
    # Candidate state, with the reset gate applied to the old state  [eq. 3]
    candidate = tanh(ln(W_xn @ x_t) + reset * ln(W_hn @ h_prev))
    # Blend the candidate with the previous state  [eq. 3]
    return (1 - update) * candidate + update * h_prev

# Self-attention head (SAN): attention, residual, layer norm  [§2.2] [eq. 5]
def san_layer(hidden):
    # Queries, keys and values are linear maps of the same states  [eq. 5]
    attended = self_attention(hidden, hidden, hidden)
    return ln(hidden + attended)              # (T, 768)

# Transformer head (TFM): SAN followed by a feed-forward block  [§2.2] [eq. 6]
def tfm_layer(hidden):
    # Attention step with a residual connection and layer norm  [eq. 6]
    attended = ln(hidden + self_attention(hidden, hidden, hidden))
    # Point-wise feed-forward network, then residual and layer norm  [eq. 6]
    return ln(attended + feed_forward(attended))  # (T, 768)

# CRF head: score whole tag sequences, not single tokens  [§2.2] [eq. 7]
def crf_decode(emissions, transitions):
    # emissions: (T, 13), per-token scores from a linear map  [§2.2]
    # transitions: (13, 13), learned tag-to-tag scores  [eq. 7]
    # Highest-scoring tag sequence, found with Viterbi search  [eq. 8]
    return viterbi(emissions, transitions)    # (T,)

# Training: softmax heads maximise token-level likelihood  [§2.2]
# CRF maximises the likelihood of the whole gold tag sequence  [§2.2] [eq. 7]
def training_loss(batch, head):
    if head == "crf":
        # Negative log-likelihood, normalised over all tag sequences  [eq. 7]
        return -log_likelihood(batch.gold_tags, batch.emissions)
    # Negative log-likelihood of each gold token tag  [§2.2]
    scores = tag_scores(batch, head)          # (B, T, 13)
    return cross_entropy(scores, batch.gold_tags)

The equations as written in the paper:

H^l = \mathrm{Transformer}_l(H^{l-1}), \quad H^0 = \{e_1,\dots,e_T\} eq. 1

P(y_t \mid x_t) = \mathrm{softmax}(W_o h^L_t + b_o) eq. 2

\begin{bmatrix} r_t \\ z_t \end{bmatrix} = \sigma\big(\mathrm{Ln}(W_x h^L_t) + \mathrm{Ln}(W_h h^T_{t-1})\big)

n_t = \tanh\big(\mathrm{Ln}(W_{xn} h^L_t) + r_t \cdot \mathrm{Ln}(W_{hn} h^T_{t-1})\big)

h^T_t = (1 - z_t) \cdot n_t + z_t \cdot h^T_{t-1} eq. 3

p(y_t \mid x_t) = \mathrm{softmax}(W_o h^T_t + b_o) eq. 4

H^T = \mathrm{Ln}\big(H^L + \mathrm{Slf\text{-}Att}(Q,K,V)\big), \quad Q,K,V = H^L W^Q, H^L W^K, H^L W^V eq. 5

\hat H^L = \mathrm{Ln}\big(H^L + \mathrm{Slf\text{-}Att}(Q,K,V)\big), \quad H^T = \mathrm{Ln}\big(\hat H^L + \mathrm{Ffn}(\hat H^L)\big) eq. 6

s(x,y) = \sum_{t=0}^{T} M^A_{y_t,y_{t+1}} + \sum_{t=1}^{T} M^P_{t,y_t}, \quad p(y \mid x) = \mathrm{softmax}\big(s(x,y)\big) eq. 7

y^\ast = \arg\max_{y}\, s(x,y) eq. 8

Symbols: \mathrm{Ln} is layer normalisation §2.2; M^A is the CRF transition matrix and M^P the emission matrix eq. 7.

Head-specific details stated in the text:

• Linear: one softmax per token §2.2 eq. 2.
• GRU: a single GRU with the paper's gate equations, layer norm on each gate input, and \mathrm{dim}_h hidden units §2.2 eq. 3.
• SAN: one self-attention layer with a residual connection and layer norm §2.2 eq. 5.
• TFM: a layer the paper describes as having the same architecture as BERT's encoder layer, i.e. attention plus a feed-forward block, each with residual and layer norm §2.2 eq. 6.
• CRF: a linear-chain CRF over the emissions, trained by likelihood and decoded with Viterbi §2.2 eq. 7 eq. 8.

Every non-CRF head ends with a softmax over the 13 tags §2.2 eq. 4.

Hyperparameters

Values the paper states, plus the code's values where the paper is silent. Code values are marked, and an implementer should treat them as the authors' choices, not as paper facts.

Results

Author-reported. Setup: bert-base-uncased with one task layer; five random seeds, averaged; micro-averaged precision, recall and F1 over aspect spans with sentiment §3.1. The baselines are the E2E-ABSA models and LSTM-CRF taggers listed in tab. 2 §3.1.

Datasets tab. 3: LAPTOP has 2741 / 304 / 800 train / dev / test sentences (2041 / 256 / 634 aspects); REST has 3490 / 387 / 2158 sentences (3893 / 413 / 2287 aspects). The paper uses two review datasets derived from SemEval, re-prepared in li2019unified §3.1. The paper does not name the REST release; the repository's REST run uses rest_total, which its README calls invalid. See pitfalls.

Hardware: not stated in the paper. The repository README lists a GTX 1080 Ti README.md.

The table's own notes: rows marked with an official-report symbol are the original papers' numbers, and LSTM-CRF rows are taken from li2019unified tab. 2. Bold marks the best F1 per dataset.

Headline points:

• The best existing F1 is 60.35 on LAPTOP and 72.78 on REST tab. 2.
• A BERT linear head already reaches 60.43 and 73.22 tab. 2 §3.2.
• Best variants: BERT + GRU on LAPTOP (61.12), BERT + SAN on REST (74.72) tab. 2.
• Gains from a slightly richer head over the linear one are small on LAPTOP (within 0.7 F1) and up to 1.5 F1 on REST tab. 2.
• Over-parameterisation: BERT-GRU, BERT-TFM and BERT-CRF were trained up to 3000 steps on REST; the paper reports dev-set F1 stays roughly flat and concludes the model is robust to overfitting §3.3 fig. 2.
• Frozen BERT: when the BERT weights are kept fixed, the paper reports results "far from satisfactory", and fine-tuning is needed §3.4 fig. 3. No numbers for the frozen run appear in the text; they are only in the figure.

Paper vs code

Code: https://github.com/lixin4ever/BERT-E2E-ABSA at commit 238174b4c22d58e91ba8634c339ebc04702ecb9e. The paper's abstract footnote links this repository, which is the basis for treating it as official.

• REST data. The paper's REST numbers are not tied to a named release §3.1 tab. 3. The repository's REST training script uses rest_total train.sh. The README says the authors built rest_total by concatenating the rest14, rest15 and rest16 train and test files, that the train and test parts overlap, and that it must not be used README.md. The repository ships no rest_total data . The README's re-run numbers on the released splits (for example BERT-Linear 72.61 / 60.29 / 69.67 on rest14 / rest15 / rest16) are on different data and are not comparable to tab. 2 README.md.
• Seeds. The paper averages five seeds §3.1. The code's fast_run.py runs one seed (42) by default, with the five-seed list commented out fast_run.py.
• LAPTOP batch size. The paper says 25 §3.1. The code uses 32 per GPU fast_run.py. The effective batch is the per-GPU size times the number of GPUs main.py main.py, so it also depends on the hardware.
• GRU direction. The paper's GRU equations run one way over the sequence eq. 3. The code's GRU always runs both directions, halves the hidden size per direction and concatenates them to 768 absa_layer.py absa_layer.py absa_layer.py.
• SAN. The paper's equation is a single attention eq. 5. The code uses a 12-head MultiheadAttention absa_layer.py absa_layer.py.
• TFM. The paper says TFM has the same architecture as BERT's encoder layer §2.2. The code builds a new nn.TransformerEncoderLayer with 12 heads and feed-forward width 4 × 768 and does not set an activation, so PyTorch's default applies. The layer starts from random weights, not from BERT's absa_layer.py.
• CRF start and end. The paper's score sums from t = 0 eq. 7, which implies a start term. The code's CRF has no start or end transitions absa_layer.py absa_layer.py.
• CRF constraints. The paper does not restrict transitions. The code's constraint mask is all ones and is never changed, so Viterbi may pick transitions that are invalid under BIEOS absa_layer.py absa_layer.py.
• Loss scale. The softmax heads use a mean cross-entropy over tokens absa_layer.py. The CRF returns the sum over the batch absa_layer.py absa_layer.py. The paper does not state a normalisation, so the two heads' gradients are not on the same scale in the code, and the clipping at 1.0 acts on each differently.
• Tag set. The paper lists 13 tags §2. The code adds an EQ label for word pieces after the first one of each word, which makes 14 labels glue_utils.py glue_utils.py.
• Optimizer and schedule. The paper is silent. The code uses AdamW with a linear schedule, no warmup and clipping at 1.0 main.py main.py.
• Model selection. Matches: selection from step 1000, every 100 steps, by micro-F1 §3.1 main.py main.py.

Pitfalls

• REST cannot be reproduced from the release. The REST setting in the code is rest_total, which the README calls invalid, and the data is not in the repository train.sh README.md. Table 2 and Figure 2 therefore cannot be checked against released data. For new work, use rest14, rest15 and rest16, and say which data you used.
• Batch size scales with GPUs. The per-GPU batch is multiplied by the GPU count main.py. The train.sh script uses three GPUs for REST at 16 per GPU, which is an effective batch of 48 train.sh train.sh train.sh. Fix the global batch size and record it.
• Subword labels. A word's tag goes to its first word piece. Later pieces get EQ, and the model is trained to predict EQ as a class glue_utils.py. Predictions are read from the first piece glue_utils.py glue_utils.py.
• Span decoding. Spans are rebuilt from BIEOS tags. A B…E span is kept only when all its sentiment tags agree, and a B with no matching E is dropped seq_utils.py. Matching is exact on both boundaries and sentiment, so a one-word boundary error counts as a false positive and a false negative glue_utils.py.
• Invalid tag sequences are allowed. With no transition constraints, the Viterbi and greedy decoders can output sequences that BIEOS would not, and the span rebuild above silently repairs or drops them absa_layer.py seq_utils.py.
• Padding is attended to. Sentences are padded to the longest sentence in the dataset plus two glue_utils.py glue_utils.py. The SAN and TFM heads are called without a padding mask, so padded positions take part in attention absa_layer.py. The --max_seq_length argument is not passed to the converter, so the 128 default does not truncate training data main.py main.py.
• Test numbers for every checkpoint. The code logs test scores for each checkpoint, but selects only on dev main.py. The paper does not say which checkpoint Table 2 reports §3.1.
• Variant choice. The paper selects checkpoints on dev, but does not say how the head variant was chosen. The bold entries in tab. 2 are the best test F1 in the table.
• Hard-coded CUDA. The GRU and LSTM hidden-state initialisers call .cuda(), so the GRU head fails without a GPU absa_layer.py absa_layer.py.
• Cost. The GRU, the CRF forward pass and Viterbi all loop over time steps in Python, and Viterbi loops over the batch absa_layer.py absa_layer.py absa_layer.py. The model itself has 110M parameters §3.3.
• Robustness claim. The stability argument rests on one dev-set curve on REST §3.3 fig. 2. The paper does not say how many seeds it used for that curve.
• Missing details. The paper omits the optimizer, schedule, clipping, dropout and sequence length. The code values are in the Hyperparameters table, and they are the authors' choices, not paper facts.

Lineage

• Builds on: BERT, the pre-trained transformer encoder (Devlin et al., 2019) §1 §2.1. The E2E-ABSA formulation as a single sequence tagger follows li2019unified (Li et al., 2019) §1 §2.
• Compared against: the E2E-ABSA models of li2019unified, luo-etal-2019-doer and he-etal-2019-interactive, and the LSTM-CRF taggers of lample-etal-2016-neural, ma-hovy-2016-end and liu2018empower §3.1 tab. 2.
• Superseded by: not stated in the paper. The repository README cites later reports on rest14 to rest16 by He et al. (2019), Liu et al. (2020), Chen and Qian (2020) and Liang et al. (2020) README.md.