A Systematic Review of Natural Language Processing Applied to Radiology Reports

In plain words

Radiologists write short narrative reports after they read a scan. Those reports hold useful facts, such as where a tumour is or whether a finding has changed, but the facts are buried in free text. Software that reads text and pulls out structured facts is called natural language processing (NLP). This paper is a survey of published research that applies NLP to such reports.

The authors searched the literature in a documented way, removed off-topic papers with automatic rules and then by reading, and sorted what remained by the clinical job each study was for, the NLP techniques it used and how well it was reported. The picture they paint is that newer neural methods are spreading, but older hand-written rules and classic statistical classifiers are still common. Results are hard to compare, because most studies use their own data, and few studies publish their data or code.

TL;DR

A systematic review of NLP on radiology reports published from 2015 to October 2019, following PRISMA. Records were found by a Google Scholar query, screened by automated and manual steps, and coded on 21 variables. Rules and classic machine learning remain prevalent, deep learning rises from a small base, and results are hard to compare because datasets differ. The authors call for reporting standards, significance testing, shared code and common benchmarks. All numbers are the authors' own counts, except the row sums marked as ours in the Results table.

Problem

Earlier reviews of this area are narrower or older. The most cited one covered studies up to October 2014 and found 67 of them § Related Work. Another review covered only deep learning, with 10 relevant papers up to September 2019 § Related Work. The authors want an up-to-date survey across all NLP methods, which also tracks study properties such as data size, performance and annotation § Background. They argue that a synthesis helps identify gaps and avoid duplicated work § Abstract, and that reporting needs to improve § Discussion.

Method

The paper's sections are unnumbered, so this card cites them as [§ Section] (Subsection). The review follows PRISMA and its protocol is registered on protocols.io § Methods. The steps are:

1. Search. Publish or Perish queries Google Scholar for research articles in English, from January 2015 to October 2019, that match the term "radiology" or "radiologist", combined with any of five NLP terms (natural language, text mining, information extraction, document classification, word2vec), and excluding patents § Methods (Eligibility for Literature Inclusion and Search Strategy). A pilot search with narrower terms returned too few papers; its papers were kept for full-text screening § Methods (Eligibility for Literature Inclusion and Search Strategy).
2. Metadata enrichment. Steps run conditionally: a DOI match via Crossref; Semantic Scholar for metadata and abstract when a DOI matched; PubMed for an abstract when neither applied; arXiv for a preprint; Unpaywall for an open-access PDF; and PDF extraction when no separate abstract was found tab. 1.
3. Automated filtering. A record is removed when any of seven rules applies: non-English, "patent" in title or URL, year before 2015, review wording in the title or "this review" in the abstract, image keywords in title or abstract, no radiology keywords, or no NLP terminology in the abstract tab. 2.
4. Citation search. A snowball step follows forward citations of the Pons et al. review and of the papers it cites. Filtering and metadata enrichment run at each stage because the branching is large § Methods (Eligibility for Literature Inclusion and Search Strategy).
5. Title and abstract screening. Four reviewers (three NLP researchers and one epidemiologist) screened independently in Rayyan and discussed disagreements. Fleiss' kappa was 0.70 § Methods (Manual Review of Literature).
6. Full-text screening. A team of eight (six NLP researchers, two epidemiologists) reviewed each full text. Each was double-reviewed by an NLP researcher, and discrepancies were settled in meetings § Methods (Manual Review of Literature).
7. Exclusion criteria. Ten criteria apply, among them: case reports; published before 2015; non-English; processing radiology images; reviews, abstracts, comments, patents or editorials; no outcomes of interest; not radiology reports; no NLP methods; no full text; duplicates § Methods (Eligibility for Literature Inclusion and Search Strategy).
8. Extraction. Each included study is coded on 18 items, listed under § Methods (Data Extraction for Analysis). They cover the study's purpose and technical objective, its data source and period, the language and body region of the reports, scan type and disease area, and the sizes of the full, annotated and split datasets. They also record external validation, annotator details and agreement, the NLP technique, the best recall, precision and F1 reported, and whether data and code are shared.

The paper describes these steps in prose and in tab. 1 and tab. 2. The pseudocode below is our restatement of the order, not code from the paper.

# Screening order  [tab. 1] [tab. 2] [§ Methods] (Manual Review of Literature)
# Input:  records returned by the query (title, abstract, metadata)
# Output: records sent to full-text review

def screen(records):
    # 1. Add DOI, abstract, arXiv and open-access links  [tab. 1]
    records = [enrich_metadata(r) for r in records]

    # 2. Drop a record matching any automated filter  [tab. 2]
    records = [r for r in records if not any_filter_hit(r)]

    # 3. Title and abstract screening by four reviewers  [§ Methods]
    records = [r for r in records if title_abstract_ok(r)]

    # 4. Full-text review; each one double-checked  [§ Methods]
    return [r for r in records if fulltext_ok(r)]

Coding decisions are manual. Each study gets one primary clinical application category, and the authors note that many studies could fit more than one § Results (Clinical Application Categories) § Limitations of Study. The six categories are Diagnostic Surveillance, Disease Information and Classification, Quality and Compliance, Cohort and Epidemiology, Language Discovery and Knowledge Structure, and Technical NLP § Results (Clinical Application Categories). The paper starts from Pons et al.'s five categories. It finds no Clinical Support Services studies, adds Disease Information and Classification and Technical NLP (which Pons et al. did not have), and its Language Discovery category is broader than Pons' query-based retrieval § Results (Clinical Application Categories).

Figure 5 plots, per application category and NLP method, the mean (bar), error bars and median (asterisk) fig. 5. The authors state that heterogeneity makes direct method comparison hard, so they summarise instead of pooling § Results (NLP Performance and Evaluation Measures).

Hyperparameters

This is a review, so there are no model settings. The table lists the search and screening parameters that a repeat of the review would need.

Results

Author-reported counts over the 164 included studies, unless stated.

Machine learning methods were led by SVM (34 studies), logistic regression (23) and random forests (18). Deep learning was led by RNN variants (14) and CNNs (10) tab. 9. The authors consider deep learning less common in clinical use than rules and classic classifiers § Discussion (Clinical Applications and NLP Methods in Radiology).

The authors' four recommendations are clearer reporting of study properties, significance testing for method comparisons, shared data and code where possible, and common benchmark datasets § Discussion (Study Heterogeneity, a Need for Reporting Standards).

Paper vs code

No official code. The paper contains no repository link. Its own synthesis (study-level listing) is "available on request" § Background. For the code search, the paper's text and abstract were checked for links, and two GitHub repository searches ("radiology reports NLP systematic review" and "radiology NLP review") returned no repositories. The search does not rule out a repository that the authors never named.

Pitfalls

• Filter word lists are missing. The automated filters rely on image, radiology and NLP keyword lists that the paper does not give tab. 2. Repeating the filters means guessing the lists.
• The search itself is underspecified. The query date, the software version and the snowballing depth are not stated § Methods (Eligibility for Literature Inclusion and Search Strategy). Google Scholar results change over time, so counts will not match a rerun.
• Variable count disagrees. The abstract says 21 variables § Abstract. The extraction list in the Methods has 18 items § Methods (Data Extraction for Analysis). The paper does not reconcile the two.
• Annotation counts do not add up. Of the studies, 89 used two or more annotators, 75 gave no annotation details and one used a single annotator. That sums to 165, but the review includes 164 § Results (Annotation and Inter-Annotator Agreement).
• Data-size percentage. The text gives 14% of studies with fewer than 500 reports. tab. 11 shows 9 + 6 = 15 studies, which is 11.1% of 135. Use the table § Results (Data Sources and Availability).
• Method counts overlap. The rows of tab. 9 add to 103 machine-learning mentions for 74 machine-learning studies, because one study can name several algorithms. The per-algorithm counts are not mutually exclusive.
• Category counts tie. Disease Information and Classification and Diagnostic Surveillance both total 45 in tab. 8. The text describes Disease Information and Classification as the main focus § Results (Clinical Application Categories).
• Growth wording. The abstract says 2019 publications "almost triple" 2015. The counts are 22 and 55 over a 10-month 2019, a ratio of about 2.5 § Abstract § Results (General Characteristics).
• F1 figure without a count. The "17% above 0.85 F1" figure appears only in the abstract. The body gives means and medians only as a plot fig. 5, without numbers in the text.
• Heterogeneous reporting. Data sizes, splits and performance measures vary between studies, so reported values are not comparable. The authors therefore use summary views instead of pooled estimates § Results (NLP Performance and Evaluation Measures) fig. 5.
• Best-result reporting. The extracted best-reported results may reflect the top configuration in each paper, which is not the same as typical performance § Methods (Data Extraction for Analysis) § Results (NLP Performance and Evaluation Measures).
• Subjective coding. The categories were hard to agree on, and reviewers' experience may have shaped what they recorded § Limitations of Study.
• Minor text order. The text lists MRI (16) before X-ray (18), while tab. 3 has X-ray higher. Read the table § Results (General Characteristics).

Lineage

• Builds on: Pons et al., the earlier systematic review of this field, whose five application categories the authors adapt § Related Work.
• Compared against: Cai et al. (a narrative review of NLP in radiology), Kreimeyer et al. (NLP coding of clinical text), Spasic et al. (machine learning in clinical text), Wu et al. (deep learning in clinical NLP) and Sorin et al. (a deep-learning review of radiology NLP) § Related Work.
• Superseded by: not stated in this paper.