ChatGPT and a New Academic Reality: Artificial Intelligence-Written Research Papers and the Ethics of the Large Language Models in Scholarly Publishing

In plain words

This is an essay, not a method paper. Written by information-science researchers in early 2023, a few months after ChatGPT came out, it asks what happens to academic publishing when a chatbot can produce a passable essay in under a minute.

It first explains, for readers without a machine-learning background, what AI, machine learning, language processing and chatbots are, and how ChatGPT is built: a pretrained language model, then fine-tuned on human-written answers, then tuned further using a reward model trained on human rankings of its answers. It then weighs the good (help with grammar, summaries, finding related work, translating or polishing non-native English) against the bad: biased training data, unclear authorship and copyright of generated text, plagiarism, essays without references, and the risk of making the "rich get richer" pattern in citations worse.

Its proposed remedies are policy ideas, not tested tools: a detector that compares a submitted paper with papers the AI generates from the same keywords, more encouragement of original research, and judging academics on the impact of their work rather than how many papers they publish. There are no experiments or measurements; the only artifact is a short ChatGPT-written essay in the appendix.

TL;DR

A non-empirical position paper on ChatGPT and GPT-3 in scholarly publishing p. 3. It offers a lay primer on AI, ML, NLP, chatbots and the ChatGPT training pipeline p. 3–6 fig. 1, lists benefits for editors, reviewers and authors p. 8, and discusses ethical issues: training-data bias, authorship, copyright, plagiarism, citation practice and the Matthew effect, and pressure from tenure evaluation p. 8–13. It proposes three responses: an AI-text detector, encouragement of creative research, and tenure reform p. 12–13. Nothing is implemented or evaluated. Relevant to an engineer only as an early survey of the concerns publishers raised in 2023; there is nothing to implement beyond a four-step detector sketch.

Problem

ChatGPT can draft essays and manuscripts quickly, and the authors argue that the ethical consequences for academia have not been worked through p. 3. Their concerns are that bias in web-scale training data could spread into published research and threaten scientific integrity, that ownership and copyright of generated text are unclear, that generated essays may lack references, and that pressure to publish many papers could push researchers towards using such tools p. 3 p. 9 p. 11–12. The paper assumes GPT-3 is the technology underneath ChatGPT p. 2.

Method

There is no method in the machine-learning sense: the paper is a narrative literature discussion. Sections are unnumbered, so citations below are to PDF pages of arXiv v2.

Structure of the argument.

1. Primer on AI, ML, NLP and chatbots, drawn from secondary sources p. 3–4.
2. Description of OpenAI, DALL-E and ChatGPT; GPT training described as unsupervised generative pretraining followed by supervised discriminative fine-tuning p. 5–6.
3. Fig. 1 describes the ChatGPT pipeline in three boxes: (1) collect demonstration data and fine-tune a pretrained GPT into a supervised fine-tuned (SFT) model; (2) collect comparison data, where labelers rank SFT outputs from best to worst, and train a reward model; (3) optimize a policy initialized from the SFT model against the reward model with PPO fig. 1. The paper gives no hyperparameters or source for this figure.
4. Limitations of GPT: misreading ambiguous, polysemous and compound terms (examples given: "construct", "digital immigrant"), high energy use, and spreading misinformation p. 7.
5. Comparison with BERT, RoBERTa and XLNet, asserted qualitatively (scale, versatility, data efficiency, human-like output, public availability) with no benchmark numbers p. 7–8.
6. Benefits: helping editors with repetitive tasks such as grammar fixes, suggesting solutions in review reports, better metadata, indexing and summaries, lay-language translation, query-based recommendation of studies, revising drafts "to improve clarity", and translating or fixing non-native English p. 8.
7. Ethical issues p. 9–13:
   • Bias in web-scale training data (gender, race, ethnicity, disability) that can carry into generated research p. 9.
   • Authorship: ownership is arguable when the user's input is substantial, unclear when the prompt is minimal or another person edits heavily; the authors suggest listing the model as a coauthor may be needed at minimum and contacting the developer p. 9.
   • Copyright: provenance of quoted material cannot be traced for models trained on web-scale data, so infringement risk may pass to the author p. 10.
   • Plagiarism: the authors argue it may not arise if original authors are credited; they note Science banned AI-generated text in its papers p. 10.
   • Citation practice: ChatGPT may suggest citations, but has produced essays with missing references (Appendix 1 cited as an example) p. 11.
   • Matthew effect: tools that rank by citation count may keep promoting highly cited, older work p. 11.
   • Publish-or-perish incentives may push researchers to use generated papers and further reduce innovation p. 12.
8. Proposed responses p. 12–13: an anti-ChatGPT detector (below); institutions and journals encouraging more creative research; changing tenure criteria toward impact and relevance rather than paper count and journal prestige.

Proposed detector. The paper presents this as a bullet list attributed to the idea of building software "similar to adblockers" p. 12. Restated as pseudocode:

Algorithm: keyword-regeneration detector (proposed, untested)  [p. 12]
Input: submitted paper P
1. Extract keywords K from P.
2. Use an AI model to generate a set of papers G from K,
   "following multiple paths" (sampling procedure not stated).
3. Compute similarity between P and each paper in G (measure not stated).
4. Flag P as possibly ChatGPT-generated if similarity indicates a match
   (threshold not stated).
Output: flag / no flag

No equations appear in the paper.

Hyperparameters

Not applicable: the paper reports no experiments, training or evaluation. For the proposed detector, every parameter is not stated.

Results

No quantitative results. The only artifact is Appendix 1, a five-paragraph essay on the value of AI that the authors say ChatGPT wrote with no human editing p. 23. The authors judge it indistinguishable from human writing and possibly above the average doctoral student, an impression with no evaluation behind it p. 7. They also note it contains no references p. 11. Other figures in the paper are quoted from secondary sources, such as ChatGPT passing one million users within a week of launch p. 5.

Paper vs code

No official code.

Pitfalls

• Not empirical. There are no measurements, and the comparison with BERT, RoBERTa and XLNet is assertion only p. 7–8. Do not quote it as evidence about model capability.
• Detector is a sketch. It names no keyword extractor, generator, sampling procedure, similarity measure or threshold p. 12. With sampled generation, regenerating "the same" paper from keywords is unlikely to produce near-duplicate text, so plain similarity may separate human and generated papers poorly; the paper does not discuss this or false positives on human papers about the same topic.
• Fig. 1 has no source. It describes the SFT → reward model → PPO pipeline without citing where it comes from or giving any setting fig. 1.
• Loose technical statements. The paper says GPT "has access to the entire Internet" p. 6, describes ChatGPT as using "regression language modeling" p. 12, and suggests ChatGPT uses citation counts to pick what to cite p. 11; none is supported by a technical source in the paper. Treat the primer as a lay summary, not a specification.
• Citation mismatches. Some in-text citations do not match the claims next to them: the Microsoft investment and OpenAI history are cited to the OpenAI Gym paper (Brockman et al., 2016) p. 5, and the remark that DALL-E and ChatGPT are OpenAI's best-known products is cited to the BERT paper (Devlin et al., 2018) p. 5. "Gilat & Cole, 2023" in the text appears as "Gilot, R., & Cole" in the references p. 8 p. 17. Check the original sources before reusing any of these citations.
• Dated model framing. The paper treats GPT-3 as the model behind ChatGPT and expects a "full version" with responses of thousands of words p. 2 p. 7. Claims about output length and missing references reflect ChatGPT as of early 2023.
• Appendix setup unstated. The model version, date and settings for the Appendix 1 essay are not given, so it cannot be reproduced p. 23.
• Venue. The title page states the paper was published in the Journal of the Association for Information Science and Technology p. 1; this card follows arXiv v2.

Lineage

• Builds on: secondary literature on ChatGPT and AI ethics, including Bender et al. 2021 ("stochastic parrots"), Dehouche 2021 on plagiarism with GPT-3, and Lund & Wang 2023 on ChatGPT in libraries p. 9 p. 10 p. 8. GPT pretraining is cited to Radford et al. 2018 p. 5.
• Compared against: BERT, RoBERTa and XLNet, qualitatively only p. 7.
• Superseded by: not known from content in this wiki.