Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges

In plain words

Text generation is the part of language AI that writes sentences: replies, stories, summaries, translations, rewrites and answers. Research on it has grown so fast that papers are hard to compare, because each one uses different tasks, scores and assumptions.

This paper reads a large set of recent papers on the topic and sorts them. It groups the work into five jobs a text generator is asked to do. It then looks at how people measure whether the output is any good, from word-matching scores to asking another AI model or a human to judge. Finally it lists problems that show up in almost every job, such as bias, made-up facts, misuse, privacy and the cost of computing.

It is a map of the field rather than a new method. Its value is in showing where the measurements are weak and where open questions remain.

TL;DR

• A systematic review of text-generation papers from 2017 through December 2025, with 257 papers in the final set §2.4.
• The tasks are open-ended generation, summarization, translation, paraphrasing and question answering §3.
• The most used metric is BLEU, in 69 papers, and n-gram overlap metrics are judged weak for open-ended outputs §4 tab. 2.
• Nine challenges are named, from bias to computing §5 §6.

Problem

Generative tasks have a large answer space and often one reference text. Overlap-based scores can therefore penalise correct paraphrases §4, and the field lacks a shared view of its tasks, metrics and open problems §1. The review sets out to organise the literature systematically and to name the gaps §1. Its scope is text-to-text generation from natural-language input. Data-to-text and multimodal work (for example images and tables) is excluded §1 §2.3.

Method

The review follows the guidelines of Kitchenham and Charters and of Okoli §2. It has three stages: search and retrieval, automatic filtering, and manual assessment, followed by a refined selection fig. 2 tab. 1.

1. Search. Two primary terms, "text generation" and "machine-generated text", are combined with secondary terms §2.1. The queries run on Semantic Scholar, restricted to the Computer Science field, and the search reports 3107 publications from 1966 to 2025 §2.1.
2. Automatic filtering. Only work from 2017 onward, the year the Transformer was introduced, is kept §2.2. For each query and year, the top five papers by influential citations are kept §2.2. Until 2021 a paper also needs more than two influential citations; from 2022 to 2025 this threshold is dropped §2.2. This leaves 381 works §2.2.
3. Manual assessment. Titles and abstracts are read to remove work from other fields and multimodal work §2.3. Relevant papers are then read in full and tagged by aspect (for example dataset, metric) §2.3.
4. Refined selection. Papers cited by fully read papers are added (68) §2.4, and a targeted subfield search adds more (95) §2.4. The final set is 257 §2.4.

# Review pipeline  [§2]
# Input:  Semantic Scholar index, Computer Science field
# Output: reviewed papers (257 reported)

def build_review(primary_terms, secondary_terms):
    # Run every primary x secondary query  [§2.1]
    candidates = search_all(primary_terms, secondary_terms)
    # Keep works from 2017 onward  [§2.2]
    candidates = [p for p in candidates if p.year >= 2017]
    # Keep the 5 most influential papers per query and year  [§2.2]
    shortlist = top_k_per_query_year(candidates, k=5)
    # From 2022 on keep all; up to 2021 require > 2 influential citations
    # [§2.2]
    shortlist = [
        p for p in shortlist
        if p.year >= 2022 or p.influential_citations > 2
    ]
    # Screen titles and abstracts, then read relevant papers in full  [§2.3]
    relevant = manual_screen(shortlist)
    # Add papers cited by full reads (68) and subfield search finds (95)
    # [§2.4]
    return relevant + cited_by_full_reads + subfield_search

The review then reports, per task, its characteristics, sub-tasks, datasets and open challenges §3. It catalogues evaluation methods in three groups: model-free, model-based, and human evaluation, with LLM-as-a-judge discussed alongside human evaluation §4. It then discusses nine cross-task challenges: bias, reasoning, hallucinations, misuse, privacy, interpretability, transparency, datasets and computing §5 §6.

The paper contains no equations and no algorithm blocks.

Task taxonomy §3 fig. 1:

• Open-ended text generation, with sub-tasks open-domain generation, story generation and dialogue generation §3.2.
• Summarization, with single-document, multi-document and dialogue summarization §3.3.
• Translation, with sentence-level and document-level translation §3.4.
• Paraphrasing, with uncontrolled and controlled paraphrasing §3.5.
• Question answering, with internal-knowledge and external-knowledge grounded QA §3.6.

Evaluation families §4 tab. 2 tab. 3:

• Model-free metrics are rule-based (for example BLEU, ROUGE, METEOR) §4.
• Model-based metrics use learned models; they are split into trained metrics (for example BLEURT, BARTScore) and hybrid metrics (for example BERTScore, MoverScore) §4.
• Human evaluation covers labelling methods and agreement measures §4 tab. 3.
• LLM-as-a-judge uses an instruction-tuned model to give a decision or score §4.

Hyperparameters

These are the review's protocol parameters, not model hyperparameters. Values come from the paper unless marked as code.

Results

Author-reported counts of the review's own process. These are not benchmark results. Setup: Semantic Scholar search, Computer Science field, manual screening by the authors §2.

Yearly counts at each stage tab. 1:

Automated metrics and how many reviewed papers use them tab. 2. Counts are per paper and are not mutually exclusive:

Human evaluation methods tab. 3:

The captions of tables 2 and 3 say their counts cover the 2017–2023 subset after eligibility checks tab. 2 tab. 3.

Headline findings (author-reported) §6:

• Five tasks, nine cross-task challenges, and 16 automated metrics §4 §6.
• Model-based metrics are used in 28 paper mentions, most often alongside model-free metrics §4 tab. 2.
• Only a few studies report agreement statistics §4. Human evaluation is described as the most reliable approach for nuanced judgments, but costly and hard to reproduce §4.
• LLM-as-a-judge agrees well with humans on simple tasks, but shows position and length bias and does worse on complex tasks §4.
• Bias is split into exposure, allocation and representation bias §5. Hallucinations are split into intrinsic and extrinsic §5.
• Misuse is grouped into non-disclosed, misaligned and adversarial use §5.

Paper vs code

The paper links a repository for the retrieval pipeline §2. It is the authors' repository (first author's GitHub account, linked from the paper). It was cloned at commit 68ff67ab61540255a203d542a8c276b2165d050b. It has no LICENSE file, so it is cited only and nothing is quoted from it.

The repository contains the search script, the filtering and top-X extraction scripts, a setup file and documentation, but no input data. Its input/ folder holds only a placeholder file.

• Time window. The paper says the top 100 results are kept "per query and year" §2.1. The code queries without a year restriction (TIMEFRAME = None), so each query returns 100 papers across all years setup.py search.py. This matches the reported 1966–2025 range §2.1 and not the "per year" wording.
• Query count. The paper reports 30 queries §2.1. The code runs each anchor with 16 sub-keys plus the bare anchor, so 2 × 16 + 2 = 34 queries setup.py search.py.
• Influential citation threshold. The more-than-two threshold for papers up to 2021 §2.2 does not appear in the scripts. filter.py and extractTopX.py keep only the top X per year and sub-key by influentialCitationCount filter.py extractTopX.py.
• Per-subquery selection. Selection matches sub-keys by substring over a joined list of query strings, and anchors are included as well filter.py. A paper can therefore be selected under several sub-keys before deduplication. The paper describes top-5 per query and year §2.2.
• Reproducibility of counts. The scripts read input/allTime.csv filter.py and output/search/deduplicated_combined.csv diagnose_filtering.py. Neither file is in the repository, so the 3107, 381 and 257 counts cannot be re-derived from it.
• Search terms. The paper's text says 14 secondary terms and lists 16 §2.1. The code has 16 setup.py.

Pitfalls

• Date ranges in captions. Tables 2 and 3 say their counts cover 2017–2023 after eligibility checks tab. 2 tab. 3. The text and abstract cover 2017–2025 §1. Which years the counts use is therefore unclear.
• Labelling methods. The text says four labelling methods were found §4. Table 3 lists five tab. 3.
• Manual screening count. The number of papers that pass manual screening is not stated. 257 minus the 68 and 95 additions would imply 94, but the paper does not say so §2.4. Table 1 also has a "Prior" column (34 in the refined stage) whose meaning is not explained tab. 1.
• Search date. The review covers literature "until December 2025" §1. The date of the Semantic Scholar search is not stated. Its ranking can change over time, so the 3107 count may not repeat.
• Appendix not in the compiled paper. The source tree includes an appendix listing the Semantic Scholar influential-citation parameters §2.2. The main document does not \input it, so it is not in the compiled paper.
• Metric counts overlap. Counts are the number of papers using each metric. One paper can use several, so the counts do not add up to the number of papers tab. 2.
• Subjective selection. Search terms were chosen with ChatGPT-4 suggestions and consensus voting among authors §2.1. The paper says this step was subjective.
• Survey, not a method. There are no hyperparameters or training runs to reproduce. The paper gives no quantitative comparison between tasks beyond counts.

Lineage

• Methodology inspired by: Kitchenham and Charters (2007) and Okoli (2015) §2.
• Compared against (named in the review): Fatima et al. (2022), a survey of 90 publications from 2015–2021 §1. Goyal et al. (2023) §1. Tay et al. (2020) and Lin et al. (2021), who review Transformers §1. Ji et al. (2023), on hallucination §5. Also named: a 2018 survey covering tasks, applications and evaluation §1.
• Superseded by: not stated.