wikillm

Paper · 2023-02-17 · cs.LG

How Generative AI models such as ChatGPT can be (Mis)Used in SPC Practice, Education, and Research? An Exploratory Study

Exploratory, qualitative study of ChatGPT (GPT-3.5, January 2023 preview) on statistical process control: code, explanations and idea generation for practice, teaching and research. The authors ran and judged each prompt by hand. Code often failed until fixed, some explanations and an EWMA simulation were wrong, and an invented dataset was described. No metric. Author-reported.

LLM evaluationPrompt engineeringProgram synthesisstatistical-process-controlpromptingeducation

arXivPDFCodeMarkdown9 min read

AI-compiled by claude-haiku-5-5 on 2026-11-22, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

This paper looks at statistical process control (SPC), whose tools include control charts. It asks whether a chatbot, ChatGPT, can help the people who work in that field: practitioners who write code and reports, teachers who build courses, and researchers who look for open questions.

The authors gave ChatGPT a set of questions in three groups (practice, learning and research). The questions asked for code, for explanations of terms, and for new material such as a framework or a syllabus. They ran the code themselves and checked the answers against textbooks and the research literature.

The chatbot did well on tasks with a clear, familiar answer, such as moving code from R to Python or explaining well-known methods. It did poorly on less common terms and on writing working code from scratch. It sometimes invented details, such as a dataset that does not exist. The authors conclude that its output must be checked by running the code and comparing it with known answers before anyone relies on it.

TL;DR

  • An exploratory, qualitative study of ChatGPT (GPT-3.5, the 30 January 2023 free research preview) on SPC tasks in practice, learning and research §4 §8.1.
  • Each prompt was run once in a new chat by hand. Generated code was pasted into R or Python and run; explanations were compared with textbooks §4. No accuracy score is reported.
  • Code for an X-bar chart failed as first written and needed argument fixes §5.1 fig. 4 fig. 5. A made-up dataset name was accepted and described §5.1 fig. 8.
  • The EWMA zero-state ARL simulation ran but gave wrong results; the authors list two main flaws §7.1 fig. 16.
  • Outputs vary between runs, and the model has since changed, so the findings are a snapshot. Every result must be checked before use.

Problem

The authors ask whether generative AI tools can be useful in statistical process control in three roles: practice, teaching and research §2. They also ask how to evaluate outputs that vary from run to run §8.2. The paper states that it does not cover ethical, legal or philosophical questions §2.

Method

Study design. The study is qualitative. Prompts are grouped by domain (practice, learning, research). Within each group they fall into three subgroups: code, explanation and knowledge generation §4 §8.1 fig. 3. The team of educators, authors and statisticians judged the responses §4.

Protocol for each prompt. The steps below are the paper's procedure, written as pseudocode. The wording is ours.

Pseudocode
# Procedure for one study prompt  [§4] [§5.1]
# Input: a prompt in one domain and one subgroup
# Output: a judged record of the reply and any fixes made by hand

def evaluate_prompt(prompt, domain):
    # Start a new chat for each subtopic so earlier context is not reused  [§4]
    reply = ask_chatbot(prompt, new_session=True)
    if reply_contains_code(reply):
        # The chatbot cannot run code, so the team runs it  [§4]
        result = run_in_r_or_python(reply)
        if result.has_error:
            # Team edits the code and records what was wrong  [§5.1]
            fixed = debug_by_hand(reply, result.error)
            record_fixes(fixed)
    else:
        # Explanations are compared with textbooks and literature  [§4]
        compare_with_references(reply, domain)
    return judge(reply)  # qualitative verdict, no score  [§4]

Repetition. Outputs for the same prompt differ in content and wording. The authors saw this when they re-ran a sample of prompts in the same order §8.1. The number of repeated runs per prompt is not stated.

Code tests. For the X-bar chart the authors checked the R qcc package source in releases 1.0, 2.0 and 2.7 to see whether the argument errors came from package versions §5.1. They judged that versions did not explain the failure §5.1.

Evaluation. Code is judged on whether it runs and on whether it gives the expected output. Explanations are judged against textbooks and literature. Knowledge creation is compared with existing material in each domain §4. The authors call their methods qualitative and say their own bias affects the judgments §4.

Interpretation of model behaviour. The authors attribute the made-up dataset example to the fine-tuning method (RLHF) and call the prompt an adversarial test. They say this is their estimate §5.1. The paper does not test that claim.

Hyperparameters

NameValueWhereNotes
ModelChatGPT, "Jan 30 Version (Free Research Preview)", GPT-3.5 engine§4 §8.1the only model used
InterfaceChat window, one new conversation per subtopic§4 §6.1
Sampling settings (temperature, top-p, max tokens)not stated—
Runs per promptnot stated; the paper reports one kept response per prompt§4 §8.1
Languages used to run codeR and Python§5.1 §6.1
R packages named in repliesqcc, ggplot2, dplyr§5.1
qcc versions checked by the authors1.0, 2.0, 2.7§5.1
X-bar test data20 subgroups of 4 observations, a sample of the qcc pistonrings data (first four of five observations for subgroups 1 to 20)fig. 7 practice_chatgpt_code_assessment.Rderived from the authors' repository, not stated in the text
Control-limit factor used in the fixed X-bar code0.729 (n = 4), applied to the mean rangefig. 7 practice_chatgpt_code_assessment.R
EWMA prompt parameterssmoothing constant 0.1, control-limit factor 3, normal data§7.1 fig. 16from the prompt
EWMA simulation settings in the replyn = 1000 steps, 10,000 runs, alpha 0.1, k 3ChatGPT_generated_code_01.pychosen by the chatbot, not the authors
Root-finding interval for cE (ARL 500)original c(0, 100); corrected c(0.1, 5)§7.1 fig. 18 ChatGPT_generated_code_04.r
Evaluation metricnone; qualitative judgement§4
Team size and roles per judgmentnot stated beyond the author list§4
Hardwarenot stated—

Results

Author-reported. The study has no metric. Each row gives what the authors saw and how they judged it.

Practice (§5)

PromptAuthors' findingSource
R code for an X-bar chart from a 20 by 4 CSVCode named "Python" in its label. Used qcc. Failed with an error about a logical argument. Setting the chart flag gave an individual chart. Changing type to xbar gave the right chart. The subgroup.size argument is not a qcc argument; the authors say it should be sizes§5.1 fig. 4 fig. 5
Second R approach with tidyverseWrong grouping (assumes one row per observation), an undefined value object, and error bars at 2 sigma while qcc uses 3 sigma. The authors' rewrite reproduces the qcc limits§5.1 fig. 6 fig. 7
Chart for a dataset name the authors made up, "bigfish"The chatbot loaded it with data() and described it, after confirming it is absent from qcc. The description is invented§5.1 fig. 8
Phase 1 vs Phase 2The reply gives the wrong purpose for each phase, and its summary is wrong§5.2 fig. 9
Zero-state ARLPartly correct. The "zero-state" qualifier is misread, so the reply describes the in-control ARL§5.2 fig. 9
Univariate, multivariate, profile monitoringPartly correct. Hotelling's T² is not mentioned; profile monitoring is limited to "complex" shapes§5.2 fig. 9
Framework, DMAIC template, explaining a signal to a managerReasonable as brainstorming. The manager reply lacks the detail a statistical consultant would ask for§5.3 fig. 10

Learning (§6)

PromptAuthors' findingSource
Explain uncommented R codeThorough and mostly correct. Does not say what the horizontal lines mean§6.1 fig. 11
Rewrite in base RRight approach. The row_number() call comes from dplyr and is wrong here; the authors replaced it with row.names(), after which the plot is correct§6.1 fig. 12
Rewrite in PythonRan without changes; the plot is correct§6.1 fig. 13
How many phases in SPC chartsAnswered "five phases". Not wrong, but not what an expert would say; the authors add measurement-system analysis, out-of-control actions and Phase 2 monitoring§6.2 fig. 14 fig. 15
Course syllabus, 15 weeks, undergraduateDescription, outcomes and topic list are usable with small changes. One topic ("multi-vari study") is unclear§6.3

Research (§7)

PromptAuthors' findingSource
Monte Carlo zero-state ARL for an EWMA chart (Python)Runs but is wrong. Authors' test runs gave 260.8, 255.8, 262.9 and 260.2. They state the correct value is near 842.1, 1035.1 or 371, depending on the standardisation used. Main flaws: run lengths cut at n = 1000 (about 40% of runs give zero), and a control limit built from the same sample's standard deviation§7.1 fig. 16 ChatGPT_generated_code_01.py
Explain an R ewma.arl functionNames the method as a Markov-chain approximation, close to the full answer§7.1 fig. 17
Find cE for ARL 500 with unirootFails with "f(lower) is NA". With interval c(0.1, 5) the authors get 2.8156§7.1 fig. 18 ChatGPT_generated_code_04.r
Practitioner-to-practitioner variabilityImprecise. Lists five method-level factors, which misses the intended meaning (repeated runs of one fixed procedure)§7.2 fig. 19
Open issues in SPC researchReasonable themes: big data and machine learning, real-time monitoring with full sampling, reducing normality assumptions, new domains§7.3 fig. 20

Discussion (§8). The authors' overall conclusions: the chatbot is useful for structured tasks such as code translation and explaining known concepts; it is weaker on niche terms and on code from scratch; its outputs must be validated §8.1. They suggest that it works as a brainstorming tool for knowledge creation §8.1.

Paper vs code

The paper links a GitHub repository in its Online Materials section § Online Materials: github.com/fmegahed/llm_expository. Its owner is the first author. It holds the authors' supplementary material, not an implementation of a method. Cloned at commit 4fb850c63749044ff2a115e3c4d201cca65648b3; the LICENSE file is MPL-2.0. The card quotes no code. The repository is cited by path and line only.

What the repository matches:

  • The Python EWMA simulation in the paper's listing matches the saved reply in ChatGPT_generated_code_01.py, with the same parameters and logic ChatGPT_generated_code_01.py fig. 16.
  • The authors' R check builds data.csv from the qcc pistonrings data, drops the fifth observation and keeps 20 subgroups, as the figure caption says practice_chatgpt_code_assessment.R fig. 7.
  • The saved reply to the second X-bar prompt uses two-sigma error bars and a value column that the wide data does not have, which matches the paper's criticism practice_chatgpt_code_assessment.R §5.1.
  • The authors' fixed ggplot code uses the 0.729 factor with the mean range, and the repository comment says the range method is used only to reproduce the qcc output practice_chatgpt_code_assessment.R.

What does not match or is missing:

  • The repository keeps the failing uniroot call with interval c(0, 100) ChatGPT_generated_code_04.r. The corrected interval c(0.1, 5) and its result 2.8156 are described in the paper but are not in the repository.
  • The base R and Python rewrites in §6.1 fig. 12 fig. 13, the learning explanations §6 and the practitioner explanations §5.2 have no saved transcript in the repository. The paper points to recordings for them.
  • The repository has no code for the ARL values 842.1, 1035.1 or 371, so those numbers cannot be checked from the repository.

The repository is MPL-2.0, a file-level copyleft licence. We copy nothing from it, so only paths and line numbers appear on this card.

Pitfalls

  • Not reproducible. The authors say the outputs cannot be reproduced because the chatbot is stochastic § Online Materials. The model was a preview from early 2023. Any later model may answer differently. Use the findings as a snapshot, and re-test with the model you plan to use.
  • One reply per prompt. The paper reports one kept reply per prompt. It does not state how many runs were tried, how replies were chosen, or whether the judges agreed. Treat the verdicts as the authors' judgement.
  • Counts disagree. The text says the open-issues reply has seven themes §7.3. The figure caption says six open issues fig. 20.
  • Figure numbers. The figure numbers above come from counting the figure environments in the TeX source, because the Online Materials list of links in the repository (Links.md) uses other numbering for some figures. Check against the PDF before citing a figure number outside this card.
  • The authors fixed the code. Where code failed, the authors edited it. The fixed versions are the authors' work, not the chatbot's. Use the fixed code only after you check it against the qcc output.
  • Packages and versions. The authors checked qcc releases 1.0, 2.0 and 2.7 and judged versions not to be the cause §5.1. Other versions were not tested.
  • Constant for the range estimate. The 0.729 constant is applied to the mean range. The authors say they do not recommend ranges for estimating the standard deviation, but used them to match the qcc output [code: practice_chatgpt_code_assessment.R#L135-L136].
  • EWMA reference values. The authors' corrected values (842.1, 1035.1, 371) are stated without the calculation or code §7.1. The card does not verify them. Also, the 2.8156 root is not reproduced in the repository.
  • Invented facts. The made-up dataset description was accepted and repeated by the chatbot §5.1. The same risk applies to any name or citation a chatbot gives: check it.
  • Training cutoff. The authors say the model "finished training in early 2022" §7.3. Recent methods may be missing or wrong.
  • Interpretation. The RLHF explanation for the "bigfish" reply is the authors' estimate §5.1, not a tested result.

Lineage

  • Builds on: none stated.
  • Compared against: none. The study compares the chatbot's answers with textbooks and the literature, not with other models.
  • Superseded by: not known from content in this wiki.

Source and license

Adapted from “How Generative AI models such as ChatGPT can be (Mis)Used in SPC Practice, Education, and Research? An Exploratory Study” by Fadel M. Megahed, Ying-Ju Chen, Joshua A. Ferris, Sven Knoth, L. Allison Jones-Farmer (arXiv:2302.10916), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/fmegahed/llm_expository at commit 4fb850c63749 (MPL-2.0). Any quoted code is shown under that license.