Bibliometric analysis

Overview

The problem: a claim such as "this field has grown more inclusive" or "this technique is now central" is only checkable once the literature has been counted. The core idea is to detect a unit of interest in each paper (a language, a method name, a dataset), tally it per venue or per year, and compare the tallies. The first two cards below share a skeleton: a corpus of papers grouped by venue and year, a mention per paper, and a statistic over those mentions 2004.09095 §4.2 2606.29836 §3.5. The third card works one step earlier, on database records, and reports counts without a per-mention test 2112.15538 §3 2112.15538 §4. The fourth card scores each paper rather than each venue-year: it measures how unusual the pairs of entities in a paper are, and then relates that score to the institutional type of its authors 2606.31058 §3.3.2.

The main difference is the unit. One variant counts languages against an external resource classification. Another counts named technologies and scores them by how tightly they cluster in co-occurrence networks. A third counts papers by their metadata (subject, journal, country, author) and maps their keywords, and its corpus is chosen by a database search rather than by a full-text tagger. The fourth uses entity pairs within each paper, not entity counts across papers, as its signal.

Variants

Venue inclusion over resource classes

Languages are sorted into six classes from labelled-dataset and Wikipedia counts. Each venue is then scored by the entropy of its language mentions and by the inverse mean reciprocal rank of each class. The authors report that class 0 ("Left-Behinds") is the least inclusive class in 10 of 11 venues 2004.09095 tab. 4. They also train a Skip-gram style entity embedding on title and abstract, and use it to rank authors and languages 2004.09095 §5.1 2004.09095 tab. 5. Card: Joshi et al., linguistic diversity in NLP venues.

The class cut-offs and the rule for deciding that a paper mentions a language are not stated, so an implementation has to define both [arxiv:2004.09095 §2.3] 2004.09095 §4.2.

Entity co-occurrence and z-score impact

Full-text PDFs of ACL, EMNLP and NAACL papers (2000 to 2022) are converted to text, and a cascade tagger finds method, dataset, metric and tool names. The names are normalised by an abbreviation map, fastText-based expansion and edit-distance clustering 2606.29836 §3.4. For each year, entities that share a paper get an edge, and each entity gets a community-based z-score 2606.29836 §3.5. The authors use 2.5 as the high-impact threshold, which yields 179 entities, 130 of them methods [arxiv:2606.29836 tab. 9]. Card: Zhang et al., entity-centric technology development.

The tagger reaches F1 87.00 on the authors' own 311-sentence test split, against 86.04 for SciBERT without the cascade and augmentation [arxiv:2606.29836 tab. 7]. That is an author-reported figure from one run per model 2606.29836 §4.1.

Entity-pair novelty and team type

Full-text PDFs of main-track ACL, EMNLP and NAACL papers (1979 to 2021) are converted to text, and 15,337 papers are kept after rule-based matching 2606.31058 §3.1 2606.31058 tab. 1. A SciBERT and BiLSTM cascade tags method, dataset, tool and metric entities, with F1 86.27 on the authors' own annotated test data 2606.31058 §3.3.1 [arxiv:2606.31058 tab. 5]. Each paper's embedded entities are compared pairwise. A pair counts as atypical when its cosine distance falls in the corpus top 10%. A paper's novelty is the share of its pairs that are atypical, and the top 10% of papers by that share are labelled novel 2606.31058 eq. 1 2606.31058 eq. 2 2606.31058 §3.3.2.

Team type comes from author affiliations, reduced to academic-only, industry-only or mixed teams, which leaves 14,812 papers [arxiv:2606.31058 §3.2]. In regressions with task and year fixed effects, academic-only and mixed teams score higher than industry-only teams on the novel label and on the continuous score 2606.31058 tab. 8 2606.31058 tab. 9. The authors report these as associations and say that causal relations were not examined 2606.31058 §6.2. Card: Chen et al., team composition and novelty in NLP papers.

The cut-offs are not given as numbers, and the training hyperparameters for the tagger are not stated 2606.31058 §3.3.2 2606.31058 §3.3.1. The score is a fraction over all pairs, and the regression controls do not include the number of entities 2606.31058 eq. 2 [arxiv:2606.31058 tab. 7].

Database records and descriptive mapping

Here the corpus comes from a title search in Scopus and Web of Science. The search was limited to 2012 to 2021, to peer-reviewed English papers, and then to open-access journal articles with full text for the bibliometric set. Two reviewers screened titles and abstracts, and only the final set was read in full 2112.15538 §3.1 2112.15538 §3.2 [arxiv:2112.15538 §3.3]. The search returned 16,209 Scopus records and 2,129 WOS records before filtering, and the bibliometric set has 1,216 papers 2112.15538 §3.1 2112.15538 §3.3.

The counts are then made per metadata field: subject area, keyword co-occurrence, year, journal, citations, country and author [arxiv:2112.15538 §4]. The authors draw the keyword network with VOSviewer and rank citations with R-studio, and they do not state the software settings [arxiv:2112.15538 §4.2] 2112.15538 §4.5. Reported results include 259 papers from China and 139 from the USA as the top two countries 2112.15538 §4.6. The authors note that their growth in yearly publication counts from 2017 may not reflect real contribution 2112.15538 §4.3. Card: Ahsan and Siddique, machine learning for disease diagnosis review.

A narrative survey on chatbots shows the same kind of protocol with a lighter output. Its search used six sources, seeded with "chatbot" and seven derived terms, and it read the abstracts of 316 articles from 2007 to 2021 after title screening 2201.06657 §3.1 2201.06657 §3.2.2 [arxiv:2201.06657 tab. 1]. Its only bibliometric outputs are the per-source counts and a publication-trend figure, and that figure is the basis for its start year 2201.06657 §3.2.1 2201.06657 fig. 1. It computes no per-paper statistic. Card: Caldarini et al., literature survey of chatbots.

When to use which

Implementation sketch

The shared core is the per-year mention count. Entity extraction and normalisation come first and are not shown. Each variant then adds its own statistic: entropy and inverse MRR for the venue variant, a z-score over a co-occurrence network for the entity variant, or the share of atypical pairs for the novelty variant.

# Group papers by venue and year, then count mentions  [arxiv:2004.09095 §4.2]
# Input:  papers: list of (venue, year, set of normalised names)
# Output: shares[(venue, year)][name] = fraction of papers mentioning name

def mention_shares(papers):
    # Group papers by venue and year
    groups = group_by(papers, key=lambda p: (p.venue, p.year))
    shares = {}
    for key, group in groups.items():
        # Count how many papers in the group mention each name
        counts = count_over(group, lambda p: p.names)
        # Divide by the number of papers in the group  [arxiv:2004.09095 eq. 1]
        shares[key] = {n: c / len(group) for n, c in counts.items()}
    return shares

# Yearly co-occurrence weights between entities  [arxiv:2606.29836 §3.5]
# Input:  papers of one year, each with a set of normalised entities
# Output: weights[a][b] = number of papers mentioning both a and b

def cooccurrence(papers_of_year):
    weights = defaultdict(lambda: defaultdict(int))
    for p in papers_of_year:
        # Every pair of entities in one paper adds one to their edge
        for a, b in combinations(p.entities, 2):
            weights[a][b] += 1
            weights[b][a] += 1
    # Entities in fewer than three papers are dropped  [arxiv:2606.29836 §4.3 (1)]
    return weights

# Share of atypical entity pairs in one paper  [arxiv:2606.31058 eq. 2]
# Input:  vectors: embedding per entity; cutoff: corpus top-10% distance
# Output: score in [0, 1]

def novelty_score(vectors, cutoff):
    # Every unordered pair of entities in the paper  [arxiv:2606.31058 §3.3.2]
    # Whether duplicate names are merged first is not stated
    pairs = list(combinations(vectors, 2))
    atypical = 0
    for a, b in pairs:
        # Cosine distance between the two entity embeddings  [eq. 1]
        if 1 - cosine_similarity(a, b) >= cutoff:
            atypical += 1
    return atypical / len(pairs)

The entity variant then scores each entity with eq. 10 of 2606.29836 §3.5. The paper does not define the communities, so choose one and record it. The released code uses the whole yearly network instead of a community 2606.29836 Paper vs code, item 1.

The novelty variant has no published cut-off value. Set the distance cut-off at the corpus 90th percentile of pair distances and the novelty cut-off at the 90th percentile of scores, and record both 2606.31058 §3.3.2.

The database variant needs no new statistic. Its counts are the same yearly tallies as the first block, with a metadata field in place of an extracted name. Record the count at each filter step, because the review's own funnel does not fully add up 2112.15538 §3.3.

Contradictions

• Trend in the embedding validation. The text says the class-wise MRR falls from class 0 to class 5, except for class 2 2004.09095 §5.3. Table 5 rises from class 2 to class 3 at MRR(5) and MRR(10), and from class 4 to class 5 at three cut-offs 2004.09095 tab. 5. Only the first class clearly ranks highest.
• Speaker share. The text gives class 0 as 15% of all speakers 2004.09095 §2.4. The rounded totals in Table 1 give about 13% 2004.09095 tab. 1.
• Figure 11 and its text. The text gives mean years to threshold of 12.03 and 12.39 for the first two periods. The figure labels give 11.97 and 12.32 2606.29836 §4.4 (2) 2606.29836 fig. 11.
• Tool counts. Table 4 gives 155 tool entities for 2815 sentences. Table 5 gives 554 for 3115 sentences, and the paper does not explain the gap 2606.29836 tab. 4 2606.29836 tab. 5.
• Paper and released code. The paper defines z-score with a community sum 2606.29836 §3.5. The released code does not, and it applies the mention floor to total counts where the paper says annual [arxiv:2606.29836 Paper vs code, items 1 and 2].
• Citation count for one paper. The text gives 218 citations for Gray et al. (2013) and the table gives 248 2112.15538 §4.5 [arxiv:2112.15538 tab. 1].
• Screening funnel. 1,216 papers go into the bibliometric set, but only 1,192 are listed as screened, and the review does not say what removed the other 24 2112.15538 §3.3.
• Model 5 coefficients. The text quotes coefficients of 0.062 and 0.05 for Model (5). Those are the Table 8 Model (1) values. Table 9 Model (5) gives 0.0207 for A&I and 0.0113 for A&A 2606.31058 §4.2.2 2606.31058 tab. 8 2606.31058 tab. 9.
• Sample size of the team split. Tables 8 and 9 report N = 14,812. The first-author split sums to 14,809 and the three-way split to 14,811 2606.31058 fig. 2 2606.31058 §4.1. The paper does not explain the gap.
• Scopus search. The survey says Scopus cannot search by keyword, but its publication-trend figure gives Scopus counts for keyword queries 2201.06657 §3.1 2201.06657 fig. 1.
• Category count. The methods text names four classification aspects, while the list of groups has five 2201.06657 §3 [arxiv:2201.06657 §3.2.2].

Papers

• Joshi et al., linguistic diversity in NLP venues: measures language resource classes and venue inclusion in NLP conferences.
• Zhang et al., entity-centric technology development: tracks method, dataset, metric and tool entities with a co-occurrence z-score.
• Chen et al., team composition and novelty in NLP papers: scores papers by unusual entity pairs and relates the score to author team type.
• Ahsan and Siddique, machine learning for disease diagnosis review: a systematic review with a bibliometric mapping of 1,216 database records.
• Caldarini et al., literature survey of chatbots: a narrative survey whose search and screening steps and per-source counts are recorded.