Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models

In plain words

This is a literature review written in spring 2023, a few months after ChatGPT launched. The authors collected every arXiv paper they could find that mentioned ChatGPT up to 1 April 2023, 194 of them, and asked two questions: how fast is this research growing, and what are people using ChatGPT for?

The first part is simple counting: papers per month, a word cloud of the most frequent words across all 194 papers, and how the papers spread over arXiv subject areas. Most of the work sat in language processing, with smaller clusters in education, medicine and other fields. A small set of scripts behind these plots is public.

The rest is a guided tour of what those papers report. ChatGPT was tried on math and physics exams, patient questions, classifying tweets, translating, writing and fixing code, pulling structured facts out of text, and judging the quality of machine translations. The recurring picture: often useful without any task-specific training, usually behind the best specialised systems, inconsistent between repeated runs, and prone to confident mistakes. The review closes with known weaknesses (outdated knowledge, bias, energy use, privacy, misuse) and where the authors think the field should go.

There is nothing to implement here. The value for an engineer is a dated map of early ChatGPT studies and of how a quick arXiv-trend analysis can be scripted.

TL;DR

A survey of ChatGPT-related research (GPT-3.5 and GPT-4) §1. Its own contribution is a bibliometric snapshot of 194 arXiv papers mentioning ChatGPT as of 1 April 2023: monthly and cumulative submission counts fig. 1, a word cloud fig. 2 and a distribution across fields fig. 3. The bulk of the paper summarises other works by application area §2.1, AI ethics §2.2, third-party evaluations §3, and lists limitations and future directions §4. All task numbers are re-reported from cited papers, not measured by the authors. Useful as a pointer list into early 2023 ChatGPT studies; verify any number against its original source.

Problem

ChatGPT research was growing quickly across many fields, and the authors set out to give an overview of what had been studied, where ChatGPT works and fails, and what ethical issues arise §1. They also wanted to show the trend quantitatively, and point to a dynamic webpage that tracks new submissions §1.

Method

There is no machine-learning method. The paper has two parts.

1. Bibliometric analysis of arXiv §1 fig. 1 fig. 2 fig. 3. The paper states only that 194 papers mentioned ChatGPT on arXiv as of 1 April 2023 and that it analysed their submission trend, a word cloud, and their distribution across fields §1. How papers were selected, how text was extracted and how fields were assigned is not stated in the paper. The authors' repository resolves this; the pipeline below is inferred from code only.

# Bibliometric pipeline, inferred from the authors' scripts

# Page through arXiv search results for the word "chatgpt",
# all-fields search, 50 hits per page, sorted by submission date
# [code: articles_info.py#L15-L27]
query = "chatgpt"
articles = []
for page in pages_until_empty(query, page_size=50):
    for hit in page:
        # Keep title, link, submission date, primary category
        # and cross-list categories  [code: articles_info.py#L31-L44]
        articles.append(parse_listing(hit))
save_table(articles, "articles.xlsx")

# Download each listed paper's PDF  [code: download_pdf.py#L9-L21]
for article in articles:
    download(pdf_url(article.link))

# Extract plain text per PDF, cut everything from the first
# "References" heading on, drop NLTK English stopwords
# [code: readpdf.py#L31-L48]
for pdf in pdfs:
    text = concatenate(extract_text(page) for page in pdf.pages)
    text = drop_from_first_match(text, "References")
    words = [w for w in tokenize(text) if lower(w) not in nltk_stopwords]
    save_text(words)

# One word cloud over all papers together: library stopwords
# plus a hand-written list, at most 70 words
# [code: txt2wordcloud.py#L32-L37]
all_text = concatenate(all saved texts)
stopwords = wordcloud_stopwords + custom_stopword_list
cloud = word_cloud(all_text, max_words=70, stopwords=stopwords)

# Trend plots: papers per month, cumulative papers per day,
# papers per year  [code: figure.py#L15-L24] [code: figure.py#L65-L66]
# [code: figure.py#L97]
# Field distribution: count by arXiv primary category
# [code: analysis.ipynb]

2. Narrative review of the collected literature, structured as in fig. 4:

• Applications §2.1: question answering in education, medicine and other fields §2.1.1; text classification §2.1.2; text generation, grouped by output length into phrases, sentences and paragraphs §2.1.3; code generation and repair §2.1.4; inference and reasoning §2.1.5; data visualisation, information extraction, quality assessment, data augmentation, multimodal fusion and prompting ("cueing") techniques §2.1.6; human–ChatGPT collaboration §2.1.7; integration into other systems §2.1.8; medical applications §2.1.9.
• AI ethics: plagiarism and cheating, political and moral leanings, regulation §2.2.
• Evaluation: comparisons with other models from cited benchmark studies §3.1, user feedback from a Twitter study §3.2, adverse effects on users §3.3.
• Limitations and future directions §4.1 §4.2.

The paper contains no equations, tables or algorithms.

Hyperparameters

Only the bibliometric analysis has settings. The paper states none of them beyond the cutoff date; values below are from code.

Results

The only measurements the authors made themselves are the bibliometric ones §1. Everything else in the table is re-reported by the survey from other papers, which this card has not checked against the originals.

Own analysis. 194 arXiv papers mentioned ChatGPT as of 1 April 2023 §1. Monthly submissions and the cumulative count grow over the period from 2022 to April 2023 fig. 1. The word cloud and field distribution are read by the authors as concentrated on natural language processing, with room for more work in education, medicine and history §1 fig. 2 fig. 3. The paper gives no per-month or per-field counts in text.

Re-reported from cited works (survey's wording condensed):

The survey's summary judgement is that ChatGPT's zero-shot performance is comparable to fine-tuned BERT-style and GPT-3.5 models but below SOTA, weak on named entity recognition, multi-hop and non-textual (math, temporal, spatial) reasoning, and low-resource languages §3.1.

Paper vs code

The repository (GPL-3.0, so no verbatim snippets here) contains the arXiv scraping, PDF download, text extraction, word cloud and plotting scripts, plus the static page of the dynamic tracker README.md. It covers fig. 1 to fig. 3; the narrative review has no code.

• The paper does not say how papers were selected; the code runs arXiv's all-fields search for "chatgpt" with no date bound articles_info.py. A rerun returns today's results, not the 194-paper snapshot, and the committed articles.xlsx at this commit was last updated after the paper (latest commit November 2023), so it is not guaranteed to be the paper's corpus.
• The paper speaks of "various fields" and reads them as application domains such as education and medicine §1 fig. 3; the notebook groups by arXiv primary category analysis.ipynb. Application domains appear only in the narrative review §2.1.
• fig. 1 shows monthly and cumulative daily counts; figure.py also draws a third panel, papers per year figure.py, which the caption does not mention.
• download_pdf.py writes to arxivChatGPT/ download_pdf.py but readpdf.py reads ./arXivChatGPT/ readpdf.py; on a case-sensitive file system the pipeline finds no PDFs unless the folder is renamed.

Pitfalls

• Secondary numbers only. Every task result comes from another paper, condensed by the survey §2 §3. Several summaries are garbled: the Tang et al. RE result ends mid-sentence ("from 75%") §2.1.6; the ICL-D3IE F1 scores quoted are for the GPT-3 variant, not ChatGPT §2.1.6; the GHOSTS result is given only as "scored impressively in 26% of cases" §2.1.1. Go to the cited work before reusing a number.
• "We use publicly available datasets" is misleading. §3.1 reads as if the authors ran evaluations; the numbers that follow are all from cited studies (refs. 48, 56, 70, 73). The same section calls ChatGPT "a qualified general-purpose model" while concluding it cannot beat SOTA §3.1.
• Corpus definition is loose. "Mentioning ChatGPT" is not defined in the paper §1; in code it is a metadata search, not a full-text search articles_info.py. False positives and misses are not discussed, and the corpus is not published as a frozen list in the paper.
• Fragile reference stripping. The regex deletes everything after the first match of "References", so a paper that uses the word earlier loses most of its body, and appendices after the bibliography are always dropped readpdf.py. The word cloud therefore reflects an unknown fraction of each paper.
• Word cloud has hand-tuned stopwords. Generic research words such as "data", "training" and "dataset" are removed by a custom list txt2wordcloud.py; the paper does not mention this, and it shapes which terms dominate fig. 2.
• Model naming. The paper uses "ChatGPT" for both GPT-3.5 and GPT-4 §1, and most surveyed studies do not say which version or date they tested. It also mentions an "open release" of GPT-4 §1 without saying what was released.
• Duplicated entries. Xia et al. (conversational program repair) is summarised in both §2.1.3 and §2.1.4, Noever et al. (ref. 37) in both §2.1.4 and §2.1.6, and Huang et al. (hate speech) in both §2.1.2 and §2.1.5; do not count them as separate evidence.
• Unsupported generalisations. Statements such as text classification struggling with rare or out-of-vocabulary words §2.1.2, training data being biased towards Python, C++ and Java §2.1.4, and knowledge ending in 2021 §4.1 carry no citation in the paper.
• Dated. The snapshot ends at 1 April 2023 §1, although this card follows arXiv v4 (August 2023), and the paper does not say whether works it discusses, such as ImpressionGPT and AD-AutoGPT §2.1.9, are in the 194-paper corpus. The repository README cites the journal version in Meta-Radiology README.md README.md.
• Licence of the code. The scripts are GPL-3.0; reuse them only under that licence.

Lineage

• Builds on: InstructGPT and RLHF as the basis of ChatGPT §1, and the 194 surveyed arXiv papers §1.
• Compared against: not applicable (survey). The evaluation section draws on Bang et al. (ref. 48, arXiv:2302.04023), Qin et al. (ref. 56, arXiv:2302.06476), Zhong et al. (ref. 70, arXiv:2302.10198) and Kocoń et al. (ref. 73, arXiv:2302.10724) §3.1.
• Superseded by: not known from content in this wiki.