Revealing the Technology Development of Natural Language Processing: A Scientific Entity-Centric Perspective

In plain words

Most studies of how a research field changes look at topics, which are broad and hard to pin down. This paper looks at something finer: the named methods, datasets, metrics and tools that papers mention. The authors read the full text of NLP conference papers from 2000 to 2022 with a tagger that finds those names. They merge spelling variants of the same name, count how often pairs of names appear together in the papers of each year, and score each name by how central it sits in that network. Names with a high score count as high-impact.

The findings are that papers name more of these technologies over time, that most high-impact names are methods, and that pre-trained language models score highest in recent years. The scores depend on how well the names were recognised and merged, so the ranking is only as good as those two steps.

TL;DR

• Entity-level study of NLP papers from ACL, EMNLP and NAACL, 2000-2022 §3.1.
• A SciBERT + BiLSTM cascade tagger reaches F1 87.00 on the authors' own test split. Author-reported tab. 7.
• 179 high-impact new entities (z > 2.5): 130 methods, 24 datasets, 19 metrics, 6 tools tab. 9. BERT has the highest z-score, 43.31 tab. 10.
• Relevant if you need a worked method for tracking named technologies in a text corpus. The released code departs from eq. 10 and from several stated rules; see Paper vs code before you implement the z-score.

Problem

Research-topic analyses are coarse and need manual work to normalise their topics §1. The authors want a finer unit, the technology-related entities that papers name: methods, datasets, metrics and tools §1 tab. 3. They ask three research questions: how the number of technology entities per paper changes (RQ1); which new technologies emerged since 2001 and drove research (RQ2); and how the popularity and speed of high-impact new technologies change over time (RQ3) §1.

Method

The pipeline has four stages: build an annotated corpus and train a recogniser, normalise the extracted names, build yearly co-occurrence networks and compute z-scores, then analyse the results §3 fig. 1.

Corpus and annotation

1. Corpus. Full-text PDFs of ACL, EMNLP and NAACL papers from 2000 to 2022, taken from the ACL Anthology. pdf2htmlEX converts them to HTML and a regular expression extracts the text. 17,783 articles converted successfully §3.1 (1) fig. 2.
2. Annotation. The authors randomly selected 50 full-text ACL papers. Five NLP researchers annotated them in WebAnno. Two annotators pre-annotated 14 papers and refined the guidelines; the other 36 papers were split into three groups of 12, each annotated by a pair of annotators. Cohen's kappa between the pairs was 0.90, 0.91 and 0.83 §3.1 (2).
3. Entity types. Method (algorithms, models), Dataset (corpora, datasets, dictionaries), Metric (indicators for experimental results) and Tool (programming languages, software, open-source tools). Examples per type are in tab. 3.
4. Extension. The 2815 annotated sentences tab. 4 were extended with 300 sentences selected from sentences where the model predicted entities, giving 3115 sentences §3.1 (3). The sentences were split 8:1:1 into train, validation and test sets tab. 5.

Annotation counts, tab. 4:

Split sizes, tab. 5:

Entity recognition model

The model uses cascaded binary tagging §3.2 fig. 3. SciBERT encodes the tokens eq. 1. Two linear heads, one for the entity start and one for the entity end, each give a binary label per token eq. 2 eq. 3. A BiLSTM over the encoder output feeds a third head that gives each token an entity type, with an O class for non-entities eq. 6 eq. 7. The training loss is the sum of three unweighted cross-entropy losses eq. 4 eq. 5 eq. 8 eq. 9.

# Cascaded binary tagging with a BiLSTM type head  [§3.2] [fig. 3]
# Input:  tokens (B, N); start, end (B, N) in {0, 1}; types (B, N)
# Output: scalar loss

def training_loss(tokens, start, end, types):
    # Encode each token with SciBERT, last hidden state  [eq. 1]
    encoded = scibert(tokens)                        # (B, N, d)
    # Two separate binary heads for entity start and end  [eq. 2, 3]
    start_logits = linear_start(encoded)             # (B, N, 2)
    end_logits = linear_end(encoded)                 # (B, N, 2)
    # Contextual features from a BiLSTM over the encoder output  [eq. 6]
    context = bilstm(encoded)                        # (B, N, 2H)
    # Entity type per token, with an O class for non-entities  [eq. 7]
    type_logits = linear_type(context)               # (B, N, T)
    # Unweighted cross-entropy per head, then sum the three  [eq. 4, 5, 8, 9]
    loss_start = cross_entropy(start_logits, start)
    loss_end = cross_entropy(end_logits, end)
    loss_type = cross_entropy(type_logits, types)
    return loss_start + loss_end + loss_type

Total loss, eq. 9:

\text{Loss} = \text{Loss}_{start} + \text{Loss}_{end} + \text{Loss}_{type}

Baselines are BERT, RoBERTa, SciBERT and T5, each with BIOES tags and each also with a CRF layer §3.2 §4.1 (1).

Data augmentation

Easy text augmentations are said to risk changing entity spans, so the authors use a model-based procedure instead §3.3:

1. Collect every entity annotated in the training set, and all unannotated abstracts.
2. Build candidate sentences from the abstracts that contain a known entity.
3. Tag each candidate with a model trained on the training set.
4. Keep a candidate only when every known entity it matches is also found by the model.
5. Add to the training set as many sentences as the training set has, drawn from the kept candidates §3.3.

Entity normalisation

Normalisation has four steps §3.4:

1. Abbreviation map. Split each name that has brackets into the short part (abbreviation) and the long part (full name). Keep high-frequency pairs after manual auditing. One abbreviation can map to several full names tab. 6.
2. Expand per paper. Assume each abbreviation has one meaning per paper. Train fastText on the paper full texts. For each abbreviation, pick the full name whose vector is most similar to the paper's entities, and replace the abbreviation with it.
3. Clean names. Lemmatise, remove words with little meaning (for example "model" and "method"), delete spaces, and compute edit-distance similarity between all names.
4. Cluster. Start a cluster for each pair of names with similarity above 0.95. Merge two clusters when their average pairwise similarity exceeds 0.85, repeat until the cluster count stops changing, and keep clusters that have at least 5 mentions in a year.

The authors checked 1000 random pairs of clusters by hand and got a precision of 91.30 §4.2.

Co-occurrence network and z-score

For each year, the nodes are the normalised entities. Two entities are linked when they are mentioned in the same paper, and the edge weight is the number of papers that mention both §3.5. Entities mentioned in fewer than three papers are left out §4.3 (1).

The impact of entity i is its community-based z-score eq. 10 fig. 4:

z\text{-score}_i = \frac{W_C^i - \frac{1}{N}\sum_{j \in C} W_C^j}{\sqrt{\frac{1}{N}\sum_{j \in C}\left(W_C^j\right)^2 - \left(\frac{1}{N}\sum_{j \in C} W_C^j\right)^2}} eq. 10

Here W_C^i is the sum of the edge weights from entity i to the other entities in network community C, and N is the number of entities in the network §3.5.

# Community-based z-score of one entity  [§3.5] [eq. 10]
# Input:  weights[k][j] = papers mentioning both k and j in one year
#         community = entities in the community C that contains entity i
#         n_entities = N, the number of entities in the network
# Output: z-score of entity i

def z_score(i, community, weights, n_entities):
    # W_C^k: edge weight from k to the entities of C  [eq. 10]
    community_weight = {
        k: sum(weights[k][j] for j in community)
        for k in community
    }
    # Mean with divisor N, as written in eq. 10  [eq. 10]
    mean = sum(community_weight.values()) / n_entities
    # Mean of squares with the same divisor  [eq. 10]
    mean_square = sum(
        w * w for w in community_weight.values()
    ) / n_entities
    std = (mean_square - mean * mean) ** 0.5
    return (community_weight[i] - mean) / std

An entity is high-impact if its z-score exceeds 2.5, the threshold the authors take from Guimerà et al. (2007). An entity's impact is the highest z-score it reaches after its first appearance §4.3 (1). The first appearance is the first year in which the entity is mentioned at least five times §4.2 (2).

Analysis

• Number of distinct normalised entities per paper, and number of new entities per year, 2001-2022 §4.2 fig. 5 fig. 6.
• Share of papers that contain at least one new entity fig. 7 and the distribution of new entities per paper fig. 8.
• Cumulative z-scores of the top 100 new entities, grouped in three-year periods and read at the N-th year after appearance §4.4 (1) fig. 10.
• Average years until an entity's z-score exceeds 2.5, per period. If the z-score is above 2.5 in the year the entity appears, that counts as one year §4.4 (2) fig. 11.

Hyperparameters

Stated in the paper, or set in the official code. The paper gives no values for most training settings, so the code values are the only ones available. Values marked "code" come from the repository and are not stated in the paper.

Results

Author-reported. Setup: the authors' annotated test split of 311 sentences tab. 5; SciERC and TDM for comparison tab. 8; a single model run on an A5000 GPU §4.1 (3). No standard deviation or repeated runs are reported.

Headline comparisons:

Tagger results on the annotated test split tab. 7:

Tagger results on SciERC and TDM tab. 8:

Corpus-level results:

Top five high-impact entities per type, highest z-score first tab. 10:

Trends, described in words:

• Method entities have the highest z-scores. Among the top 10 entities by highest z-score, eight are methods. Wikipedia and BLEU are the other two, and their z-scores kept rising §4.3 (2) fig. 9.
• BERT appears in 2019 with a z-score of 14.37, rises to its peak in 2021, and eases in 2022 §4.4 (2) fig. 9.
• LSTM, attention, Adam, CNN and RNN rise until about 2018 or 2019 and then fall §4.3 (2) fig. 9.
• The cumulative z-score of the top 100 entities is much higher for entities that first appeared in 2019-2021 than for earlier periods, at N = 1 and N = 2 §4.4 (1) fig. 10.

Paper vs code

The official repository is https://github.com/ZH-heng/technology_development at commit 105a3ba92221a4cec30bf2fc5bcb59e81a298f6c. The paper names it in its abstract and introduction. The repository has no LICENSE file, so this card quotes nothing from it and cites by path and notebook cell only. The shipped top-ents.csv matches the 179 entities, the type counts and the top-five z-scores of tab. 9 and tab. 10 after rounding. The notebooks were not re-run for this card.

Differences between the paper and the code:

1. z-score is not community-based in the code. Paper eq. 10 sums over community C. The code computes each entity's total edge weight over the whole yearly network, and standardises it against all entities in that year. It contains no community step [code: Code/z-score_calculation .ipynb cell f6b15e55def5be09] eq. 10.
2. Mention floor. Paper: annual frequency of at least 5 §3.4 (4). Code: total mentions across all years of at least 5 [code: Code/z-score_calculation .ipynb cell ee68d31f62481b8d] [code: Code/entity-normalization.ipynb cell 370f3540e4ec576a].
3. First appearance. Paper: the first year with frequency of at least 5 §4.2 (2). Code: for entities with more than 100 total mentions, the first year where that year and the next both have at least 5 [code: Code/z-score_calculation .ipynb cell ee68d31f62481b8d].
4. Papers before first appearance. The code drops an entity from any paper published before its first-appearance year [code: Code/z-score_calculation .ipynb cell ee68d31f62481b8d]. The paper does not state this.
5. Entity type. The code assigns each cluster the type it was predicted as most often z-score_calculation .ipynb cell ee68d31f62481b8d. The paper does not say how a cluster gets its type.
6. Merge threshold. Paper: merge when average similarity exceeds 0.85 §3.4 (4). Code: the threshold starts at 93 and drops by 2 per pass until 85. It is at least 90 for clusters whose shortest name has fewer than six characters, and the loop stops after 15 passes [code: Code/entity-normalization .ipynb cell e9a97b6f].
7. Similarity measure. Paper: edit-distance similarity §3.4 (3). Code: thefuzz fuzz.ratio, computed only for names that share a trigram and whose lengths differ by at most 1.5 times, and kept only above 80 [code: Code/entity-normalization.ipynb cell c3ec7feb].
8. Missing pairs block merges. The code's average similarity returns 0 as soon as one pair between two clusters is missing from the above-80 list, so that pair of clusters cannot merge [code: Code/entity-normalization.ipynb cell e9a97b6f].
9. Name cleaning. The code also sorts each name's words alphabetically, drops names that start with a digit, drops the lone word "score", and keeps only names of at least two characters [code: Code/entity-normalization.ipynb cell 2fe80573041059b8]. The paper lists lemmatisation, stop-word removal and space deletion only §3.4 (3).
10. Abbreviation choice. Paper: similarity between each full name and "all entities in the article" §3.4 (2). Code: cosine similarity between the full-name vector and the vector of the paper's joined entity list [code: Code/entity-normalization.ipynb cell d2cb636bf0bfe1b].
11. Loss notation. Eqs. 4, 5 and 8 write each sum as logits times the log of the label term. The code uses standard cross-entropy on the logits and the gold labels eq. 4 eq. 5 eq. 8 ner-cascade.py ner-cascade.py. Follow the code.
12. Start and end heads. Eqs. 2 and 3 are identical. The code uses two separate linear layers eq. 2 eq. 3 ner-cascade.py.
13. Special tokens. The code tokenises without [CLS] or [SEP] tokens ner-cascade.py. The paper does not say.
14. Metric name. §3.2 calls P "accuracy" and §4.1 calls it "precision". The code computes exact-match micro precision, recall and F1 over (type, start, end) spans §3.2 §4.1 (1) ner-cascade.py.
15. Decoding. The paper does not describe decoding. The code pairs starts with ends, takes a majority type over subword tokens, revises boundaries with two rules, and falls back to the type sequence when nothing is found ner-cascade.py ner-cascade.py ner-cascade.py.
16. Released data is smaller. The README describes 500 training, 100 validation and 100 test sentences, plus 2493 augmented sentences [README, "Dataset Discription"]. Table 5 reports 2493, 311 and 311 tab. 5.
17. Augmentation code is absent. The repository has no script for the procedure in §3.3.
18. Figure code is absent. No code is shipped for figures 5 to 11, for the §4.2 counts, or for the years-to-threshold averages of §4.4 (2).
19. Citation version. The README cites an IPM 2024 article (doi 10.1016/j.ipm.2023.103574). The arXiv record is dated 2026-06-29. The materials here do not show whether the two versions differ.
20. Missing inputs. The fastText model (ftt.model) and normalized-ents.parquet are not in the repository. The normalisation notebook reads one input from ./dada/, which does not match ./data/, and two others from hard-coded Windows paths under F:\tmp\ [code: Code/entity-normalization.ipynb cell 370f3540e4ec576a]. The z-score notebook writes ent-type.txt to F:\tmp\ [code: Code/z-score_calculation .ipynb cell ee68d31f62481b8d], but reads it back from ./data/ in cell 8e76d01aa256fd07.

Pitfalls

• Eq. 10 is under-specified. The sums run over community C, but the divisor is N, the size of the whole network §3.5. The paper does not say how communities are found, and it does not say whether self-edges count. Your choice here sets every z-score, so record it.
• Recognition and merging errors reach the ranking. Normalisation precision is 91.30 on 1000 pairs, with no interval §4.2. The authors note that errors in recognition and normalisation affect the impact values §5.2. A name split or merged in error moves its own z-score and those of its neighbours.
• Thresholds and pruning shape the clusters. The 80-point pruning, the missing-pair rule and the merge schedule all change which clusters form entity-normalization.ipynb cell c3ec7feb [cell e9a97b6f]. The paper gives only the endpoints.
• Hidden name rules. Sorting words makes names order-insensitive, and the digit, length and "score" filters remove names entirely [code: Code/entity-normalization.ipynb cell 2fe80573041059b8]. Names under three characters are never compared by similarity entity-normalization.ipynb cell c3ec7feb, so they only merge if identical after cleaning.
• Frequency floor applied differently. Annual versus total counts change which entities survive and when they first appear. Choose one and say so.
• Small test split, single run. The test split has 311 sentences tab. 5, with one run per model and no variance. The augmentation gain is 0.51 F1 points (86.49 to 87.00) tab. 7. A single run cannot show that this gain exceeds seed-to-seed variation.
• Unclear retraining for SciERC and TDM. The paper does not state whether the tagger was retrained on those datasets' training splits, or whether augmentation was applied there §4.1 (2) tab. 8.
• Split and count mismatch. Table 4 gives 155 tool entities for 2815 sentences. Table 5 gives 554 tool entities across the 3115 sentences. The paper does not explain the gap. The Method, Dataset and Metric counts also differ between the two tables.
• Figure 11 disagrees with its text. The text gives 12.03 and 12.39 for the first two periods; the figure labels give 11.97 and 12.32 §4.4 (2) fig. 11. Pick one and say which.
• Right-censoring is not discussed. The data end in 2022 §3.1. The 2019-2021 cohort has about two to four years of observation, and the averages use only the high-impact entities §4.4 (2). The paper does not discuss this bias.
• Counting convention. An entity already above 2.5 in its first year counts as one year §4.4 (2). Averages and counts are therefore sensitive to the yearly granularity.
• Scope is three venues. The corpus is ACL, EMNLP and NAACL, 2000 to 2022. Other venues and earlier papers were not collected, so "NLP" here means those venues §5.2.
• Unknown conversion losses. The 17,783 articles are the ones that converted successfully. The total number of PDFs is not stated, so the loss rate is not known §3.1 (1).
• Names differ in form across outputs. For example "attention" in the code and "Attention Mechanism" in tab. 10 and the appendix; "Moses" and "MOSES"; "cross-entropy" and "Cross-Entropy". Match names case-insensitively, and check the appendix's period lists app., p. 42-43.
• Cost is not reported. The pairwise similarity step compares all name pairs that share a trigram entity-normalization.ipynb cell c3ec7feb. The paper gives no runtime or memory figures, and the normalisation needs a fastText model trained on the corpus §3.4 (2).

Lineage

• Builds on: SciBERT, for the encoder §3.2 (Beltagy et al., 2019; no arXiv id given in the paper). Cascaded binary tagging, after Wei et al. (2020) §2.2. The fastText embeddings are used for abbreviation expansion §3.4 (2).
• Compared against: SCIIE (Luan et al., 2018), PURE (Zhong and Chen, 2021), SpERT (Eberts and Ulges, arXiv:1909.07755), Hierarchical Transformer (Zaratiana et al., 2022), and the TDM data of Hou et al. (2021) tab. 8. For topic-based analysis of NLP: Mohammad (2019, arXiv:1911.03562) and Parmar et al. (2020) §4.5.
• Impact measure: co-occurrence z-scores in the spirit of Guimerà et al. (2007), with the threshold 2.5 taken from that work §4.3 (1).
• Superseded by: not known from this wiki.