---
kind: concept
slug: bibliometric-analysis
title: Bibliometric analysis
summary: >-
  Measuring what a research literature contains over time: which languages, methods, datasets, tools or team types appear in which venues, and how central or novel they become. Four variants: venue-level inclusion over resource classes, entity co-occurrence with a z-score, entity-pair novelty, and descriptive counts over database records. Each depends on how records are detected and merged.
tags: [bibliometrics, measurement, nlp-inclusion, scientific-entities, systematic-review, novelty]
papers: ["2004.09095", "2606.29836", "2112.15538", "2606.31058", "2201.06657"]
related: [multilingual-evaluation, named-entity-recognition, sequence-labeling, data-augmentation]
status: draft
updated_at: 2026-11-01
---

## Overview

The problem: a claim such as "this field has grown more inclusive" or "this
technique is now central" is only checkable once the literature has been
counted. The core idea is to detect a unit of interest in each paper (a
language, a method name, a dataset), tally it per venue or per year, and
compare the tallies. The first two cards below share a skeleton: a corpus of
papers grouped by venue and year, a mention per paper, and a statistic over
those mentions [arxiv:2004.09095 §4.2] [arxiv:2606.29836 §3.5]. The third
card works one step earlier, on database records, and reports counts
without a per-mention test [arxiv:2112.15538 §3] [arxiv:2112.15538 §4]. The
fourth card scores each paper rather than each venue-year: it measures how
unusual the pairs of entities in a paper are, and then relates that score to
the institutional type of its authors [arxiv:2606.31058 §3.3.2].

The main difference is the unit. One variant counts languages against an
external resource classification. Another counts named technologies and
scores them by how tightly they cluster in co-occurrence networks. A third
counts papers by their metadata (subject, journal, country, author) and maps
their keywords, and its corpus is chosen by a database search rather than by
a full-text tagger. The fourth uses entity pairs within each paper, not
entity counts across papers, as its signal.

## Variants

### Venue inclusion over resource classes

Languages are sorted into six classes from labelled-dataset and Wikipedia
counts. Each venue is then scored by the entropy of its language mentions
and by the inverse mean reciprocal rank of each class. The authors report
that class 0 ("Left-Behinds") is the least inclusive class in 10 of 11
venues [arxiv:2004.09095 tab. 4]. They also train a Skip-gram style entity
embedding on title and abstract, and use it to rank authors and languages
[arxiv:2004.09095 §5.1] [arxiv:2004.09095 tab. 5]. Card:
[Joshi et al., linguistic diversity in NLP venues](/papers/2004.09095).

The class cut-offs and the rule for deciding that a paper mentions a language
are not stated, so an implementation has to define both [arxiv:2004.09095
§2.3] [arxiv:2004.09095 §4.2].

### Entity co-occurrence and z-score impact

Full-text PDFs of ACL, EMNLP and NAACL papers (2000 to 2022) are converted to
text, and a cascade tagger finds method, dataset, metric and tool names. The
names are normalised by an abbreviation map, fastText-based expansion and
edit-distance clustering [arxiv:2606.29836 §3.4]. For each year, entities
that share a paper get an edge, and each entity gets a community-based
z-score [arxiv:2606.29836 §3.5]. The authors use 2.5 as the high-impact
threshold, which yields 179 entities, 130 of them methods [arxiv:2606.29836
tab. 9]. Card:
[Zhang et al., entity-centric technology development](/papers/2606.29836).

The tagger reaches F1 87.00 on the authors' own 311-sentence test split,
against 86.04 for SciBERT without the cascade and augmentation [arxiv:2606.29836
tab. 7]. That is an author-reported figure from one run per model
[arxiv:2606.29836 §4.1].

### Entity-pair novelty and team type

Full-text PDFs of main-track ACL, EMNLP and NAACL papers (1979 to 2021) are
converted to text, and 15,337 papers are kept after rule-based matching
[arxiv:2606.31058 §3.1] [arxiv:2606.31058 tab. 1]. A SciBERT and BiLSTM
cascade tags method, dataset, tool and metric entities, with F1 86.27 on the
authors' own annotated test data [arxiv:2606.31058 §3.3.1] [arxiv:2606.31058
tab. 5]. Each paper's embedded entities are compared pairwise. A pair counts
as atypical when its cosine distance falls in the corpus top 10%. A paper's
novelty is the share of its pairs that are atypical, and the top 10% of
papers by that share are labelled novel [arxiv:2606.31058 eq. 1]
[arxiv:2606.31058 eq. 2] [arxiv:2606.31058 §3.3.2].

Team type comes from author affiliations, reduced to academic-only,
industry-only or mixed teams, which leaves 14,812 papers [arxiv:2606.31058
§3.2]. In regressions with task and year fixed effects, academic-only and
mixed teams score higher than industry-only teams on the novel label and on
the continuous score [arxiv:2606.31058 tab. 8] [arxiv:2606.31058 tab. 9].
The authors report these as associations and say that causal relations were
not examined [arxiv:2606.31058 §6.2]. Card:
[Chen et al., team composition and novelty in NLP papers](/papers/2606.31058).

The cut-offs are not given as numbers, and the training hyperparameters for
the tagger are not stated [arxiv:2606.31058 §3.3.2] [arxiv:2606.31058 §3.3.1].
The score is a fraction over all pairs, and the regression controls do not
include the number of entities [arxiv:2606.31058 eq. 2] [arxiv:2606.31058
tab. 7].

### Database records and descriptive mapping

Here the corpus comes from a title search in Scopus and Web of Science. The
search was limited to 2012 to 2021, to peer-reviewed English papers, and then
to open-access journal articles with full text for the bibliometric set. Two
reviewers screened titles and abstracts, and only the final set was read in
full [arxiv:2112.15538 §3.1] [arxiv:2112.15538 §3.2] [arxiv:2112.15538
§3.3]. The search returned 16,209 Scopus records and 2,129 WOS records before
filtering, and the bibliometric set has 1,216 papers [arxiv:2112.15538 §3.1]
[arxiv:2112.15538 §3.3].

The counts are then made per metadata field: subject area, keyword
co-occurrence, year, journal, citations, country and author [arxiv:2112.15538
§4]. The authors draw the keyword network with VOSviewer and rank citations
with R-studio, and they do not state the software settings [arxiv:2112.15538
§4.2] [arxiv:2112.15538 §4.5]. Reported results include 259 papers from China
and 139 from the USA as the top two countries [arxiv:2112.15538 §4.6]. The
authors note that their growth in yearly publication counts from 2017 may not
reflect real contribution [arxiv:2112.15538 §4.3]. Card:
[Ahsan and Siddique, machine learning for disease diagnosis review](/papers/2112.15538).

A narrative survey on chatbots shows the same kind of protocol with a lighter
output. Its search used six sources, seeded with "chatbot" and seven derived
terms, and it read the abstracts of 316 articles from 2007 to 2021 after title
screening [arxiv:2201.06657 §3.1] [arxiv:2201.06657 §3.2.2] [arxiv:2201.06657
tab. 1]. Its only bibliometric outputs are the per-source counts and a
publication-trend figure, and that figure is the basis for its start year
[arxiv:2201.06657 §3.2.1] [arxiv:2201.06657 fig. 1]. It computes no
per-paper statistic. Card:
[Caldarini et al., literature survey of chatbots](/papers/2201.06657).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Question is about which languages or communities a venue includes | Venue inclusion over resource classes | The metric targets venue-level coverage and needs no entity tagger [arxiv:2004.09095 tab. 4] |
| Question is about which named methods, datasets or tools gain traction | Entity co-occurrence and z-score | Works at the level of named technologies, with a worked threshold of 2.5 [arxiv:2606.29836 §4.3 (1)] |
| Question is about whether unusual combinations of methods, datasets, tools and metrics go with novelty | Entity-pair novelty | Scores each paper on its entity pairs, with a per-paper share rather than a venue tally [arxiv:2606.31058 §3.3.2] |
| Question is about how team composition relates to novelty | Entity-pair novelty, read as association | Team type comes from affiliations, and the authors say the analysis is not causal [arxiv:2606.31058 §3.2] [arxiv:2606.31058 §6.2] |
| Only titles and abstracts are available | Venue inclusion (embedding uses title and abstract) | The embedding was trained on title and abstract only [arxiv:2004.09095 §5.1] |
| Full text is available and you can annotate a sample | Entity co-occurrence or entity-pair novelty | Both taggers were trained on annotated sentences from 50 papers, with reported F1 of 86 to 87 [arxiv:2606.29836 §3.1 (2)] [arxiv:2606.31058 §3.3.1] |
| Recent cohorts are the focus | Entity co-occurrence, with care | Data end in 2022 and the cohort averages are right-censored, which the paper does not discuss [arxiv:2606.29836 §4.4 (2)] |
| Question is about where a field publishes: venues, journals, countries or authors | Database records and descriptive mapping | Its metadata fields are the output, and it names no per-paper detector [arxiv:2112.15538 §4] |
| Only database search results are available, with no full text | Database records and descriptive mapping | The review screened on titles and abstracts and read full text only for the final set [arxiv:2112.15538 §3.3] |
| Recording how a review's search and screening were done | Database records and descriptive mapping | The chatbot survey reports its terms, window and per-source counts in one table [arxiv:2201.06657 tab. 1] |
| Results must be reproduced from the paper alone | None of the four as published | The venue variant depends on unstated cut-offs and an unreleased dataset [arxiv:2004.09095 app. A.2]. The entity variant depends on unstated community detection and its code has no licence file [arxiv:2606.29836 §3.5]. The novelty variant leaves its cut-offs and tagger training unstated [arxiv:2606.31058 §3.3.1] [arxiv:2606.31058 §3.3.2]. The database variant leaves its software settings unstated and does not say its record set is released [arxiv:2112.15538 §3.1] [arxiv:2112.15538 §4.2] |

## Implementation sketch

The shared core is the per-year mention count. Entity extraction and
normalisation come first and are not shown. Each variant then adds its own
statistic: entropy and inverse MRR for the venue variant, a z-score over
a co-occurrence network for the entity variant, or the share of atypical
pairs for the novelty variant.

```pseudocode
# Group papers by venue and year, then count mentions  [arxiv:2004.09095 §4.2]
# Input:  papers: list of (venue, year, set of normalised names)
# Output: shares[(venue, year)][name] = fraction of papers mentioning name

def mention_shares(papers):
    # Group papers by venue and year
    groups = group_by(papers, key=lambda p: (p.venue, p.year))
    shares = {}
    for key, group in groups.items():
        # Count how many papers in the group mention each name
        counts = count_over(group, lambda p: p.names)
        # Divide by the number of papers in the group  [arxiv:2004.09095 eq. 1]
        shares[key] = {n: c / len(group) for n, c in counts.items()}
    return shares
```

```pseudocode
# Yearly co-occurrence weights between entities  [arxiv:2606.29836 §3.5]
# Input:  papers of one year, each with a set of normalised entities
# Output: weights[a][b] = number of papers mentioning both a and b

def cooccurrence(papers_of_year):
    weights = defaultdict(lambda: defaultdict(int))
    for p in papers_of_year:
        # Every pair of entities in one paper adds one to their edge
        for a, b in combinations(p.entities, 2):
            weights[a][b] += 1
            weights[b][a] += 1
    # Entities in fewer than three papers are dropped  [arxiv:2606.29836 §4.3 (1)]
    return weights
```

```pseudocode
# Share of atypical entity pairs in one paper  [arxiv:2606.31058 eq. 2]
# Input:  vectors: embedding per entity; cutoff: corpus top-10% distance
# Output: score in [0, 1]

def novelty_score(vectors, cutoff):
    # Every unordered pair of entities in the paper  [arxiv:2606.31058 §3.3.2]
    # Whether duplicate names are merged first is not stated
    pairs = list(combinations(vectors, 2))
    atypical = 0
    for a, b in pairs:
        # Cosine distance between the two entity embeddings  [eq. 1]
        if 1 - cosine_similarity(a, b) >= cutoff:
            atypical += 1
    return atypical / len(pairs)
```

The entity variant then scores each entity with eq. 10 of
[arxiv:2606.29836 §3.5]. The paper does not define the communities, so choose
one and record it. The released code uses the whole yearly network instead
of a community [arxiv:2606.29836 Paper vs code, item 1].

The novelty variant has no published cut-off value. Set the distance cut-off
at the corpus 90th percentile of pair distances and the novelty cut-off at
the 90th percentile of scores, and record both [arxiv:2606.31058 §3.3.2].

The database variant needs no new statistic. Its counts are the same yearly
tallies as the first block, with a metadata field in place of an extracted
name. Record the count at each filter step, because the review's own funnel
does not fully add up [arxiv:2112.15538 §3.3].

## Contradictions

- **Trend in the embedding validation.** The text says the class-wise MRR
  falls from class 0 to class 5, except for class 2 [arxiv:2004.09095 §5.3].
  Table 5 rises from class 2 to class 3 at MRR(5) and MRR(10), and from
  class 4 to class 5 at three cut-offs [arxiv:2004.09095 tab. 5]. Only the
  first class clearly ranks highest.
- **Speaker share.** The text gives class 0 as 15% of all speakers
  [arxiv:2004.09095 §2.4]. The rounded totals in Table 1 give about 13%
  [arxiv:2004.09095 tab. 1].
- **Figure 11 and its text.** The text gives mean years to threshold of 12.03
  and 12.39 for the first two periods. The figure labels give 11.97 and 12.32
  [arxiv:2606.29836 §4.4 (2)] [arxiv:2606.29836 fig. 11].
- **Tool counts.** Table 4 gives 155 tool entities for 2815 sentences. Table 5
  gives 554 for 3115 sentences, and the paper does not explain the gap
  [arxiv:2606.29836 tab. 4] [arxiv:2606.29836 tab. 5].
- **Paper and released code.** The paper defines z-score with a community
  sum [arxiv:2606.29836 §3.5]. The released code does not, and it applies the
  mention floor to total counts where the paper says annual [arxiv:2606.29836
  Paper vs code, items 1 and 2].
- **Citation count for one paper.** The text gives 218 citations for Gray et
  al. (2013) and the table gives 248 [arxiv:2112.15538 §4.5] [arxiv:2112.15538
  tab. 1].
- **Screening funnel.** 1,216 papers go into the bibliometric set, but only
  1,192 are listed as screened, and the review does not say what removed the
  other 24 [arxiv:2112.15538 §3.3].
- **Model 5 coefficients.** The text quotes coefficients of 0.062 and 0.05
  for Model (5). Those are the Table 8 Model (1) values. Table 9 Model (5)
  gives 0.0207 for A&I and 0.0113 for A&A [arxiv:2606.31058 §4.2.2]
  [arxiv:2606.31058 tab. 8] [arxiv:2606.31058 tab. 9].
- **Sample size of the team split.** Tables 8 and 9 report N = 14,812. The
  first-author split sums to 14,809 and the three-way split to 14,811
  [arxiv:2606.31058 fig. 2] [arxiv:2606.31058 §4.1]. The paper does not
  explain the gap.
- **Scopus search.** The survey says Scopus cannot search by keyword, but its
  publication-trend figure gives Scopus counts for keyword queries
  [arxiv:2201.06657 §3.1] [arxiv:2201.06657 fig. 1].
- **Category count.** The methods text names four classification aspects,
  while the list of groups has five [arxiv:2201.06657 §3] [arxiv:2201.06657
  §3.2.2].

## Papers

- [Joshi et al., linguistic diversity in NLP venues](/papers/2004.09095): measures language resource classes and venue inclusion in NLP conferences.
- [Zhang et al., entity-centric technology development](/papers/2606.29836): tracks method, dataset, metric and tool entities with a co-occurrence z-score.
- [Chen et al., team composition and novelty in NLP papers](/papers/2606.31058): scores papers by unusual entity pairs and relates the score to author team type.
- [Ahsan and Siddique, machine learning for disease diagnosis review](/papers/2112.15538): a systematic review with a bibliometric mapping of 1,216 database records.
- [Caldarini et al., literature survey of chatbots](/papers/2201.06657): a narrative survey whose search and screening steps and per-source counts are recorded.