Utilizing Cognitive Signals Generated during Human Reading to Enhance Keyphrase Extraction from Microblogs

In plain words

Keyphrase extraction means picking the few words or short phrases that best describe a post. Tweets are short, noisy and often lack hashtags, so tagging them is hard. Earlier work helped by using eye-tracking data from people reading the text, because where readers linger hints at what matters.

This paper adds brain-activity data (EEG) on top of eye tracking. Each word in a tweet gets extra numbers taken from the reading records of people who read ordinary sentences. The model then gets these numbers along with the word itself. The attention layers of the tagger are changed so that these numbers can influence which words get selected.

The authors report that the brain-activity numbers help more than the eye-tracking numbers, and that combining both sits between the two. With large pre-trained models the gains are smaller. The idea is worth knowing about, but the released code does not match the paper in several places, so check the Paper vs code section before reusing it.

TL;DR

• Task: keyphrase extraction (tagging each token as B/I/E/S/O) on tweets, with two datasets, Election-Trec and General-Twitter §3.1.2.
• Method: per-word EEG and eye-tracking features from ZuCo are averaged per word type and attached to the word inputs of BiLSTM(+CRF) taggers, to the soft-attention input, or to the self-attention query §3.1.3, §3.2.
• Author-reported result: EEG features give the largest gains; SATT-BiLSTM+CRF reaches 67.82 F1 (Election-Trec) and 82.46 F1 (General-Twitter) tab. 4, §4.2.
• Cost and caveat: the features are per word type, not per reader, at inference; most words lack real ZuCo values (Pitfalls). Released code differs from the paper.

Problem

Keyphrase extraction from microblogs suffers from short, noisy and sparse text §1. Eye-tracking features have improved it, but the authors argue that eye tracking alone is limited by physiological and acquisition factors. They also say the most representative prior eye-tracking study improved results by only around 1% §1. They ask whether EEG features, which they say reflect neural activity more directly, can add to eye tracking §1.

Method

Overview fig. 1, §3: four steps. (1) Build a "multi-source cognitive signals" corpus by attaching ZuCo word-level features to the microblog corpus. (2) Build baseline and attention-based taggers. (3) Inject the cognitive features at different points depending on the model. (4) Run experiments on whole-band and per-band EEG, joint EEG and eye-tracking combinations, and pre-trained models.

Step 1: cognitive data. ZuCo records eye-tracking and EEG for 12 adult native speakers reading about 1100 English sentences. The authors use only the normal reading mode §3.1.1. Eye tracking gives 17 features grouped into early-stage, late-stage and contextual groups tab. 4, §3.1.1. EEG gives 8 features: two per band for theta, alpha, beta and gamma, using a 128-channel system. Each band is split into two sub-ranges tab. 5, §3.1.1. The features are averaged per word type over ZuCo's vocabulary §3.1.3. Words missing from ZuCo get an imputed value §3.1, §3.1.3.

Step 2: keyphrase tagging. Keyphrase extraction is treated as sequence labelling. The tags used in the code are B, I, E, S and O config.py. Baselines are BiLSTM and BiLSTM+CRF §3.2.1, fig. 2. The attention models add a soft-attention layer (ATT-) or a self-attention layer (SATT-) on top of the BiLSTM, with or without CRF §3.2.2, fig. 3.

Step 3: injecting cognitive features. The paper injects features at three different points:

• Baseline models: the cognitive vector is concatenated to the word embedding before the BiLSTM §3.2.1, fig. 2.
• Soft-attention models: the cognitive vector is concatenated to the input of the soft-attention layer §3.2.2.
• Self-attention models: the cognitive vector is appended after the query vector §3.2.2.

Soft attention. A learned vector scores tanh-transformed states eq. 1 eq. 2, and the scores are normalised with softmax eq. 2. The output is the attention-weighted sum of the states over positions eq. 3. The paper does not say how this pooled vector becomes per-token tag scores (see Pitfalls).

Self attention. The standard key, query and value projections eq. 4 eq. 5 eq. 6 are used, with scaled dot-product attention eq. 7. Eq. 8 sums attention over positions, but its exact form is not stated.

Pre-trained models. Three improvements are tested §3.2.3:

• GloVe embeddings are added to the input of SATT-BiLSTM+CRF §3.2.3(1).
• BERT (768-dimensional token outputs) with a linear classifier, treating tagging as sequence labelling §3.2.3(2).
• T5-Base and T5-Large with a linear classifier on top of the encoder output §3.2.3(3).

# Build one cognitive vector per word type  [§3.1.3]
# Input:  zuco_tokens: (word, et[17], eeg[8]) for each ZuCo token
# Output: cog_vocab: dict word -> vector (25 values)

def build_cognitive_vocab(zuco_tokens):
    # Average each feature over all ZuCo occurrences of the word  [§3.1.3]
    per_word = group_by_word(zuco_tokens)
    cog_vocab = {}
    for word, rows in per_word.items():
        cog_vocab[word] = mean(rows)                   # (25,)
    # Impute missing words with the corpus average  [§3.1]
    cog_vocab["<OOV>"] = mean_over_words(cog_vocab)    # (25,)
    # Rescale each feature to 0-9; the method is not stated  [tab. 7, §3.1.3]
    return rescale_to_0_9(cog_vocab)

# Soft-attention BiLSTM-CRF forward pass  [§3.2.2] [eq. 1-3]
# Input:  word_ids (B, L); cog (B, L, k); mask (B, L)
# Output: per-token tag scores (B, L, num_tags); mapping not stated

def att_bilstm_crf_forward(word_ids, cog, mask):
    # Embed words and encode them with a BiLSTM  [§3.2.1]
    embedded = embedding(word_ids)                     # (B, L, d)
    hidden = bilstm(embedded)                          # (B, L, 2h)
    # Concatenate cognitive features at the attention input  [§3.2.2]
    attn_input = concat(hidden, cog)                   # (B, L, 2h + k)
    # Score each position: M = tanh(H*), s = w^T M  [eq. 1] [eq. 2]
    scores = matmul(tanh(attn_input), w)               # (B, L, 1)
    alpha = masked_softmax(scores, mask, dim=1)        # (B, L, 1)
    # Weighted sum over positions  [eq. 3]
    pooled = sum(attn_input * alpha, dim=1)            # (B, 2h + k)
    # Per-token tag scores from the pooled vector: not stated  [§3.2.2]
    return tag_scores(pooled)

# Self-attention with cognitive features on the query  [§3.2.2] [eq. 4-8]
# Input:  X (B, L, d); cog (B, L, k)
# Output: attended states (B, L, d_out)

def self_attention_block(X, cog):
    # Project inputs to keys, queries and values  [eq. 4] [eq. 5] [eq. 6]
    keys = W_K @ X                                     # (B, L, d)
    queries = W_Q @ X                                  # (B, L, d)
    values = W_V @ X                                   # (B, L, d)
    # Append cognitive features after the query vector  [§3.2.2]
    queries = concat(queries, cog)                     # see Pitfalls
    # Scaled dot-product attention  [eq. 7]; d_k is not stated
    weights = softmax(queries @ transpose(keys) / sqrt(d_k))
    return weights @ values

# Evaluation protocol as described  [§4.1] [eq. 9-11]
# Input:  train_set, test_set
# Output: mean P, R, F1 over five runs

def run_experiment(train_set, test_set):
    best_per_run = []
    # Five training iterations, each with 20 epochs  [§4.1]
    for run in range(5):
        model = new_model()
        records = []
        for epoch in range(20):
            train_one_epoch(model, train_set)
            # Top-k keyphrase P, R, F1 on the test data  [eq. 9-11]
            records.append(evaluate(model, test_set))
        # Keep the best F1 and its P and R over the epochs  [§4.1]
        best_per_run.append(best_by_f1(records))
    # Average over the five runs  [§4.1] [eq. 9-11]
    return average(best_per_run)

Figures in words: fig. 1 is the four-step pipeline. Fig. 2 shows where features enter the BiLSTM+CRF baseline. Fig. 3 shows the soft-attention and self-attention variants. Fig. 4 and fig. 5 plot the F1 change (ΔF1) for each EEG band and each EEG and eye-tracking combination on both datasets. Fig. 6 plots the pre-trained model comparisons.

Hyperparameters

Stated in the paper, or in code where the paper is silent. "Not stated" means neither source gives a value for that setting.

Results

Author-reported. Setup: Election-Trec (24,210 train / 6,054 test tweets) and General-Twitter (78,760 / 33,755) §3.1.2. Metric: keyphrase F1 at top-k §4.1. "Baseline" is the Bi-LSTM without cognitive features, marked Baseline* in the paper tab. 4, §4.2. Rows are F1 values only; P and R are in the source tables.

Author conclusions: EEG features help more than eye-tracking features. The β-band pair EEG5,6 gives the best contiguous-band result §4.3, tab. 5. The combined eye-tracking and EEG setting falls between the two single-signal results §4.2, §4.4, abstract. A five-tweet case study is also given (tab. 8, §4.6): the authors score 12 (no features), 86 (EEG), 29 (eye tracking) and 53 (combined) points §4.6.

Paper vs code

Official repository: github.com/yan-xinyi/AKE, owned by the first author. The clone (commit e56d7a2) has no LICENSE file, so code is cited by path and line only, with no quotes. The dataset folders hold only README files, so the data cannot be re-run from the repo alone.

• EEG features are fixed to EEG5 and EEG6 in every non-BERT/T5 model. The paper's "All_EEG" uses all 8 EEG features tab. 4, §4.2. The code slices eeg[:,:,4:6] in the baseline, attention and self-attention models BILSTMCRF.py ATT_BILSTMCRF.py SATT_BILSTMCRF.py. The config's default fs_name = "EEG56" matches this config.py. The paper's All_EEG results cannot be reproduced from this code, and the eye-tracking features never enter these models.
• Soft-attention injection point differs. Paper: features go into the input of the attention layer §3.2.2. Code: features are concatenated before the BiLSTM, at the embedding output ATT_BILSTMCRF.py.
• Soft-attention pooling differs. Paper eq. 3 is a weighted sum over positions. Code multiplies each position's state by its own weight, then applies tanh, and keeps one output per position ATT_BILSTMCRF.py. There is no sum over positions.
• Self-attention injection point differs. Paper: cognitive features go after the query vector §3.2.2. Code: they are concatenated to the value vector and passed through a new linear layer SATT_BILSTMCRF.py.
• Extra residual and LayerNorm in self-attention. Code adds a residual path and LayerNorm after the attention output SATT_BILSTMCRF.py. The paper gives only eq. 7 and eq. 8.
• Released SATT-BiLSTM+CRF cannot be built as written. The constructor passes linear_dropout_value, which is not defined in config.py or anywhere else in the repo SATT_BILSTMCRF.py. This is the paper's best model in tab. 4. I did not run the code (no GPU or torch environment); this comes from reading the source.
• GloVe dimension conflict. Paper: GloVe uses 100-dim vectors §4.1. Code builds the embedding with the config's 128-dim embed_dim SATT_BILSTMCRF_GloVe.py and copies 100-dim vectors into rows of that size SATT_BILSTMCRF_GloVe.py SATT_BILSTMCRF_GloVe.py. As written this should raise a size error on the first copy, which I did not run.
• Epoch selection uses the test set. Paper: best F1 over the top 20 epochs, averaged over five runs §4.1. Code: each epoch is scored on the test loader, and the best F1 is kept SATT_BILSTMCRF.py. No dev split exists in the repo. The run loop logs per-run bests but does no averaging main.py.
• BERT and T5 epochs. README says 5 epochs README.md. The training loop uses the global epochs, which is 20 BERT.py config.py.
• T5 output is probably the loss, not hidden states. The T5 training loop passes the tags as labels T5.py. The wrapper then reads outputs[0] as the hidden states T5.py and concatenates it with the per-token features T5.py. With labels set, Hugging Face's T5ForConditionalGeneration returns the scalar loss first, so the concatenation would fail as written. I did not run this.
• Dropout is switched off during testing and not restored. In the BERT and T5 test loops, every dropout module's p is set to 0 BERT.py T5.py. Later training epochs therefore run with p = 0.
• EEG scaling differs. Paper tab. 7 note: values normalised to 0–9 [tab. 7, §3.1.3]. Code: per-sentence min-max to [0,1] processing.py, with min and max starting at 0.0.
• OOV imputation differs. Paper: OOV words get the average over ZuCo §3.1. Code: the OOV vector is min-max normalised across its own 25 values processing.py.
• Vocabulary size. Paper: 37,347 for the AKE dataset §5.2. Table 6 gives 33,583 (Election-Trec) and 82,736 (General-Twitter) tab. 6, §3.1.2. Config defaults to 85535 config.py and README gives 37347 for Election-Trec README.md.
• Module imports. The model files use from ..config import * SATT_BILSTMCRF.py. utils.py imports models.* as a top-level package utils.py. That combination normally fails with "attempted relative import beyond top-level package". I did not run this.
• Hard-coded paths. config.py points at ./dataset/GT-*.json and ./datas/*.txt, and those files are not in the repo config.py.

The repo has no script that regenerates the paper's tables, and I did not train any model, so the reported numbers are not reproduced here.

Pitfalls

• Test-set selection. The paper says hyperparameters were "fine-tuned … on each dataset and each model" §4.1, and the code keeps the best test-set epoch. The reported F1 values are therefore optimistic. Differences between rows are often under one point (for example BiLSTM+CRF with no features at 81.63 vs 81.74 for SATT-BiLSTM+CRF on General-Twitter tab. 4, §4.2), and the paper reports no variance or significance test.
• Band choice made on the test set. The β-band result (EEG5, EEG6) is chosen by comparing per-band test scores tab. 5, §4.3. The same test results then select the feature set for the combination experiments §4.4. A held-out set would be needed to claim the band choice generalises.
• Features are per word type, not per reader. Each word gets the average of its ZuCo values §3.1.3. At inference the same word always receives the same vector, whatever the tweet or reader. Context-dependent gaze and EEG effects are lost. The paper's gains may come from word-type priors rather than reading signals.
• Most words lack ZuCo values. ZuCo coverage of the microblog vocabulary is 8.579% (Election-Trec) and 4.038% (General-Twitter) tab. 6, §3.1.2. The paper itself says a high OOV proportion is a main limitation §5.2. Read the reported gains as mostly imputed-feature gains.
• Two OOV rules. §3.1 says OOV words get the average over the entire ZuCo corpus. §3.1.3 says OOV values come from summing each word's values under a signal and taking the average. The code follows neither (Paper vs code).
• Eq. 3 does not give per-token outputs. Eq. 3 pools over positions into one vector. Sequence tagging needs one output per token. The paper does not say how it gets per-token scores, and the code uses per-position outputs instead.
• Query width does not match keys. Appending cognitive features to the query changes the query's width, but the key projection is not extended (§3.2.2, eq. 5, eq. 7). Without a stated rule the dot product in eq. 7 is undefined. The code moves the features to the value instead.
• Eq. 8 and d_k. Eq. 8 sums attention terms over j for a fixed i, which is not the usual per-position output. d_k is not defined. Implementers must choose.
• Table and section numbering is reused. Tables 4 to 7 appear twice each: once for dataset or feature descriptions and once for results. The case study is headed "4.5" on its page, but the text cites §4.6. Cite tables by section.
• Case study is anecdotal. The five tweets per dataset in tab. 8 §4.6 are scored with an ad hoc 10/3/0 rubric. Do not treat the 12/86/29/53 points as evidence.
• Pre-trained model results are close. On General-Twitter, BERT gains are small across feature sets tab. 7, §4.5. The paper itself says that BERT and T5-Base improvements on General-Twitter are weak and similar §4.5.

Lineage

• Builds on: the eye-tracking microblog keyphrase work of Zhang & Zhang (2021), which used Bi-LSTM models with reading time features §1, §2.1 (not a card on wikillm).
• Compared against: Bi-LSTM and Bi-LSTM+CRF baselines, attention variants, GloVe, BERT, T5-Base and T5-Large tab. 4, tab. 7, §4.5.
• Superseded by: not stated.