Delving into LLM-assisted writing in biomedical publications through excess vocabulary

In plain words

Nobody labels which scientific abstracts were written with a chatbot, so this paper looks for a trace in word counts instead. Some words became much more common in 2024 than their earlier trend predicted. Words such as "delves" and "showcasing" jumped after ChatGPT appeared, while earlier spikes came from topics such as Ebola or Covid. The authors compare each word's 2024 frequency with a projection made from 2021 and 2022, and they keep the words that rose and are mostly style words rather than topic words.

Adding up the frequency gap of a group of such words gives a floor on how many abstracts may have been touched by an LLM. The authors reach at least 13.5% of 2024 abstracts, with higher values in some fields, countries and journals. The number is a lower bound, because an abstract without any of these words is not counted. The method also cannot tell direct LLM use apart from people copying the words that LLMs favour.

TL;DR

The paper measures LLM-assisted writing in PubMed abstracts from 2010 to 2024 without training a detector. It flags words whose 2024 frequency is far above a linear projection from 2021 and 2022, annotates them by hand as content or style words, and combines two sets of style words into a single gap. The reported lower bound is 13.5% of 2024 abstracts, against about 7% for Covid-related words in 2021. All numbers are author-reported. The released code differs from the printed excess rule in one sign, listed under Paper vs code.

Problem

Earlier estimates of LLM use rely on detectors or on mixture models trained on labelled human and LLM texts. Those training sets can be biased, and detector results can be hard to inspect word by word §1. None of the earlier work compares LLM-driven vocabulary change with earlier shifts, such as those caused by the Covid-19 pandemic §1 §3. The authors want a method that needs no labelled corpus and that can be run on earlier years as well §1 §3.

Method

This is a measurement method, not a model. No model is trained or run. The pipeline counts word occurrences per abstract and year, selects excess words for 2024, annotates them, and forms group gaps. Equations are inline in the paper and are not numbered, so they are cited to their section.

1. Corpus §4.1 §4.2. Use the PubMed baseline snapshot from the start of 2025. Keep complete English abstracts of 250 to 4000 characters, and keep publication years 2010 to 2024. Remove about 100 regular-expression patterns of non-abstract text (copyright lines, editor notes, citation strings), and drop errata and retraction notices identified by title §4.2. Build a binary word-by-abstract matrix, and keep only words of at least four letters made of the 26 letters a to z §4.2.
2. Smoothed frequency §4.3. For a word with a abstracts in a year out of b abstracts in that year, the frequency is (a + 1) / (b + 1).
3. Excess test for 2024 §2.1 §4.3. Use only words whose frequency is at least 10^-4 in 2023 and in 2024. Project the 2024 frequency q as the 2022 frequency plus twice any rise from 2021 to 2022, never below the 2022 value. Then compute the gap delta = p - q and the ratio r = p / q. A word is excess if delta > 0.01, or if log10 r exceeds the bound given in the pseudocode below. The printed bound differs from the code; see Paper vs code.

# Excess test for one word in 2024  [§2.1] [§4.3]
# Input:  freq[y] smoothed frequency of the word in year y (floats)
# Output: True if the word is an excess word

def is_excess(freq):
    # Only words at or above the floor in 2023 and 2024  [§4.3]
    if freq[2023] < 1e-4 or freq[2024] < 1e-4:
        return False
    # Projection: never below the 2022 value  [§4.3]
    expected = freq[2022] + 2 * max(freq[2022] - freq[2021], 0)
    # Frequency gap and frequency ratio  [§2.1]
    gap = freq[2024] - expected
    ratio = freq[2024] / expected
    # Ratio bound; the sign follows the released code, see Paper vs code
    # [code: scripts/preprocess_and_count_utils.py#L423]
    ratio_bound = -(log10(2) / 4) * log10(freq[2024])
    return gap > 0.01 or log10(ratio) > ratio_bound

4. Manual annotation §2.1 §4.4. The authors annotated the 900 unique excess words from 2013 to 2024 as content or style, and gave each a part of speech. The annotators did not know the year in which each word was selected. Words whose use was ambiguous were left unlabelled §4.4.
5. Rare set §2.2 fig. 4. Take the style excess words whose 2024 frequency is below a threshold T. The authors choose T = 0.02, which gives the largest rare-set gap. The set has 291 words §2.2 fig. S6.
6. Common set §2.2. Ten style words, chosen by hand to maximise the gap: across, additionally, comprehensive, crucial, enhancing, exhibited, insights, notably, particularly, within §2.2.
7. Group gap §2.2. For a marker set G, the gap is Delta_G = P_G - Q_G, where P_G is the observed share of 2024 abstracts that contain at least one word of G, and Q_G is the expected share. The paper does not state how Q_G is projected. The pseudocode assumes the same projection as for single words, which is our reading. The two sets are disjoint, and the lower bound is the mean of their two gaps.

# Lower bound on the share of 2024 abstracts  [§2.2]
# Input:  papers[y] abstracts of year y; two disjoint marker sets
# Output: scalar, a lower bound on the LLM-processed fraction

def lower_bound(papers, rare_set, common_set):
    # Gap for each marker set  [§2.2]
    delta_rare = group_gap(papers, rare_set)
    delta_common = group_gap(papers, common_set)
    # group_gap is defined in Method step 7
    # Average of the two independent estimates  [§2.2]
    return (delta_rare + delta_common) / 2

8. Subgroups §2.3 §4.5. Apply the same two sets to subgroups of papers: 50 countries with the most papers, 100 journals with the most papers in 2024, 39 fields taken from prior work §4.1, four pooled journal groups, and first- and last-author gender. Compute a subgroup estimate only if it has at least 300 papers in each year from 2018 to 2023 §4.5. Gender is inferred from first names and covers about 55% of authors §4.1.
9. Interpretation §3. The gap is a lower bound. Abstracts that use no marker word are not counted, and the method cannot separate direct LLM use from the spread of LLM-favoured words into human writing §3.

Hyperparameters

Results

Author-reported, on the PubMed snapshot described in Method. The paper compares against no model; the baselines are the Covid-related word set and earlier work on LLM detection and mixture models §2.2 §3. The lower bound counts only abstracts with a marker word, so it is not a prevalence estimate.

The paper reports no confidence intervals and no robustness check on the selection of marker words §2.2 §3.

Paper vs code

Official code: berenslab/llm-excess-vocab, linked from the paper's data and code availability section §4.7. The repository LICENSE is MIT and names two of the authors. We cloned it at commit 53db991afc251782106cd817a1c3fa47a4d41781, read the preprocessing and excess scripts, and did not run them. We did not trace the gap computation in the figure notebook, and we did not verify the 2025 scripts against the paper's numbers.

Matches the paper:

• Smoothed frequency, projection and floor follow §4.3 preprocess_and_count_utils.py.
• Words are kept only if they are a to z and at least four letters §4.2 preprocess_and_count_utils.py.
• Vectoriser settings follow §4.2 preprocess_and_count_utils.py.
• Rare threshold 0.02 and the ten common words follow §2.2 02-preprocess-and-count.py preprocess_and_count_utils.py.
• Excess words are kept when the gap exceeds 0.01 or the ratio test passes §2.1 preprocess_and_count_utils.py.
• The annotation file holds 900 words: 462 content (51.3%) and 407 style (45.2%), matching §2.1 excess_words.csv.
• The top 50 countries and top 100 journals follow §4.5 preprocess_and_count_utils.py preprocess_and_count_utils.py.

Differs from the paper:

• Ratio bound sign. The paper prints the bound as (log10 2 / 4) times log10 p §2.1. The code computes log10 2 minus (log10 p + 4) times (log10 2 / 4) preprocess_and_count_utils.py, which equals minus (log10 2 / 4) times log10 p. The printed form is negative for every p below 1, so read literally it would accept a word at a ratio of about 0.6 when p = 10^-3, where the code requires about 1.68. This is our arithmetic.
• Frequency floor. The paper says frequencies must be above 10^-4 §4.3. The code keeps words at or above 10^-4 preprocess_and_count_utils.py.
• Style label. The rare and common sets use only words labelled exactly "style" 02-preprocess-and-count.py preprocess_and_count_utils.py. The annotation file holds 26 mixed labels (content/style, style/content, and similar), which the code excludes. The paper says ambiguous words are not labelled §4.4.
• Later scripts. The repository contains 2025 scripts (scripts/102-preprocess-and-count-2025.py, scripts/103-updated-figures-2025.ipynb) and a post-publication figure. The paper's numbers are from the earlier scripts. We did not check whether the 2025 scripts change any reported value.
• Gap computation. The group gaps Delta_G are computed in the figure notebook scripts/03-figures.ipynb. We did not trace how Q_G is projected, so the projection in Method step 7 is our reading.
• Subgroup minimum. We did not find the 300-paper minimum §4.5 in the scripts we read.

No discrepancies found in: the smoothing, the projection factor, the word filter, the vectoriser settings, the rare threshold, the common word list, and the top-50 and top-100 subgroup choices.

Pitfalls

• Printed ratio bound. Implement the code's sign, not the printed one. With the printed sign, a word at p = 10^-3 passes the ratio test at r of about 0.6, where the code requires about 1.68. We did not run the test, so the effect on the paper's word counts is unchecked.
• Lower-bound logic. The gap counts only abstracts with a marker word. An LLM-processed abstract with none of the words is missed, so the estimate is a floor. The paper says so §3. A rise caused by people adopting LLM-favoured words also counts as LLM use, which the paper cannot separate §3.
• In-sample selection. The rare threshold T is chosen to maximise the rare gap on the same 2024 data §2.2. The common set is hand-adjusted to maximise its gap §2.2. The paper reports no held-out check, so the reported lower bound is optimistic by construction. Re-run with a fixed threshold before reusing the number.
• Group projection is not stated. Q_G is the expected share of abstracts with any marker word, and the paper does not say how it is projected §2.2. Using the single-word projection is our assumption.
• Manual style labels. Style and content labels are the authors' judgement, made without knowing the year §4.4. Another annotator could move words in or out of the sets, and that moves every gap.
• Frequency ratio is unstable for rare words. For a word with tiny projected frequency, r can be very large, as with "zika" (r = 40.4 in 2017) and the Covid words (up to r above 1000) §2.1. The floor of 10^-4 limits this only in part.
• Subgroups are small. Several subgroup results rest on a few hundred papers per year, and the t-SNE cluster is about 400 papers §2.3 fig. S7. Gender is inferred from first names for about 55% of authors and is approximate §4.1.
• Vectoriser threshold. min_df 1e-6 is a fraction of documents in scikit-learn, which keeps words in about 15 abstracts out of 15.1 million. This is our reading of the library's convention, not a value the paper states.
• Corpus scope. The study covers PubMed English abstracts with a length filter §4.1. Full texts, other databases and other languages are not tested.
• Not validated against labels. The paper does not check the estimate against any labelled set of LLM-written abstracts. It argues against that approach §1 §3, but the lower bound has no external check here.
• Repository changes. The code has later scripts and a 2025 figure that the paper does not describe. Pin the commit in code_commit when you reuse it.

Lineage

• Builds on: the field mapping and PubMed parsing from prior work by González-Márquez et al. §4.1 §4.2 §4.5, and the excess-mortality idea from the Covid literature §1. Neither is linked here, since no card exists for them.
• Compared against: detector-based studies, mixture-model studies and marker-word lists §1 §3. The paper compares its marker words with examples from §3.
• Superseded by: not known from content in this wiki.