Understanding Critical Thinking in Generative Artificial Intelligence Use: Development, Validation, and Correlates of the Critical Thinking in AI Use Scale

In plain words

Chatbots can sound fluent and confident while being wrong, so people who use them need to check what they say. This paper asks what it means to be good at that checking and whether it can be measured. The authors define a disposition, "critical thinking in AI use", as the habit of checking where AI information comes from, wanting to understand how these systems work, and thinking about the wider effects of relying on them.

To measure it, the authors wrote a short questionnaire and tested it over six separate studies with different groups of people. The final version has 13 statements that people rate from strongly disagree to strongly agree. The statements fall into three groups: checking sources, wanting to understand AI, and thinking about its wider consequences. Higher scores went with more checking in a fact-checking task and with deeper reflection in a writing task. The scale is self-reported, so it measures how people describe their own habits, not how they behave in every situation.

TL;DR

• A 13-item, 5-point self-report scale with three subscales: Verification, Motivation to understand AI, and Reflection on responsible AI tab. 1.
• Psychometric evidence across six studies: EFA, higher-order CFA, sex invariance, convergent and discriminant validity, and test-retest reliability over one and two weeks §2–§6.
• Criterion evidence is limited to a GPT-4o fact-checking task and a writing task in Study 6. The CRT test showed no reliable association in Study 3 §4.2.2 §7.2.
• Author-reported throughout. Samples are Prolific or university pools; Study 6's chatbot task is the authors' own design §7.1.2 p. 48.

Problem

Generative AI outputs are fluent, can be wrong, and are often opaque. The authors argue that existing critical-thinking measures were built for pre-AI contexts and do not capture verifying AI outputs or understanding how models fail §1 p. 4. The paper therefore sets out to define a domain-specific construct, develop a measure, and examine its correlates and criterion validity §1.3 p. 13.

Method

Construct. Critical thinking in AI use is defined as reflective thinking about what to believe or do when interacting with AI outputs, adapted from Ennis's definition §1.1 p. 5. It has three dimensions: reflective judgement, verification, and epistemic motivation §1.1.2 p. 7. The authors frame it with the Heuristic-Systematic Model and dual-process theory §1.1.1 [pp. 5–6].

Study 1: item generation and content validation §2 [pp. 14–16].

• Generate 90 items, 18 per dimension, adapted from three existing critical-thinking measures, with wording changed to refer to AI §2.1.1 p. 14. Two extra dimensions (scepticism, openness) were included at generation, giving five dimensions §2.1.1 p. 14.
• Three subject-matter experts reviewed the items and kept 30 §2.1.1 p. 15.
• 24 naive judges sorted items into dimensions over four rounds. Two indices were used: proportion of substantive agreement (p_sa) and substantive validity coefficient (c_sv) §2.1.2 p. 15.
• Final content validity: p_sa = .96 and c_sv = .93 for the 27-item scale. Readability: Flesch–Kincaid grade 6.22, Reading Ease 74 §2.2 p. 16.

Study 2: exploratory factor analysis §3 [pp. 16–21].

• 270 Prolific participants answered the 27 items §3.1.1 p. 17.
• Parallel analysis, scree plot and BIC were used to choose the number of factors §3.1.3 p. 18.
• Extraction was principal axis factoring with oblimin rotation §3.1.3 p. 18.
• Items were removed iteratively if communality was below .35, primary loading was below .40, or a cross-loading was .35 or higher with a primary-minus-secondary gap below .20. No factor could drop below three items §3.1.3 [pp. 18–19].
• The solution with the lowest RMSEA was selected, and items lacking conceptual fit were then removed §3.1.3 p. 19.
• A three-factor, 17-item solution was retained, and four more items were removed, leaving 13 items §3.2 p. 21. The removed items are listed on p. 21.

Study 3: first psychometric validation §4 [pp. 22–34].

• Sample of 376 Prolific participants, sized for 99% power at df = 62 (183 for 80% power) §4.1.1 p. 22.
• Measures: the 13-item scale, the Critical Thinking Disposition Scale, Need for Cognition, Actively Open-minded Thinking, Cognitive Reflection Test-2, Attitudes Toward AI, AI Dependency, and the Big Five Inventory-2-XS §4.1.2 [pp. 23–25].
• Analysis: first-order CFA with no cross-loadings or correlated residuals, then a higher-order CFA. Estimator MLR. Thresholds: CFI/TLI ≥ .90 acceptable and ≥ .95 good, RMSEA ≤ .06, SRMR ≤ .08, standardised loading ≥ .50 salient, AVE ≥ .50 adequate §4.1.3 [pp. 25–26].
• Sex invariance: configural, metric, scalar and residual steps, with ΔCFI ≤ .01 as the criterion §4.1.3 p. 29.
• Discriminant validity: Fornell–Larcker, where √AVE of each comparator must exceed its correlation with the scale §4.1.3 [pp. 27–28].

Study 4: second psychometric validation §5 [pp. 37–44].

• 295 recruited, 290 analysed. Inclusion required some prior AI use. Sample 20–78 years, mean age 42.30 §5.1.1 p. 38.
• Adds AI attachment, social desirability, positive and negative affect, and self-reported frequency of AI use §5.1.2 [pp. 38–40].
• Same CFA and validity logic as Study 3 §5.1.3 p. 40.

Study 5: test–retest reliability §6 [pp. 45–47].

• 1-week sample: 92 university-pool participants recruited in Singapore. 2-week sample: 122 participants recruited through a Telegram channel who completed both sessions §6.1.1 p. 45.
• ICC(2,1) with absolute agreement, computed with a two-way random-effects model, plus Pearson r §6.1.2 p. 46.

Study 6: criterion validation §7 [pp. 47–56].

• 200 Prolific participants recruited, 191 analysed, targeting 159 for r = .22 at α = .05 two-tailed §7.1.1 p. 48.
• Fact-checking task. 12 statements (6 true, 6 false) through a GPT-4o chatbot embedded in Qualtrics. Each statement was assigned one of four source-link conditions (none, broken, working-irrelevant, working-relevant). Participants answered Correct, Incorrect or Unsure. Veracity accuracy is percentage correct §7.1.2 [pp. 48–50] fig. 6.
• Self-reported verification. One frequency item (0–4) and one methods item listing four categories plus free text §7.1.2 p. 50.
• Writing task. 5-minute reflection on AI's effects, scored 0–4 by GPT-4o with a rubric. Cross-checked against Claude Opus 4.1 §7.1.2 p. 51 §7.1.3 p. 53.
• Process indices (per participant, 0–100%): Evidence Extraction, Misread Rate, Evidence-Gap Resolution §7.1.3 p. 52–53 tab. 3.
• Analysis: OLS regression of each outcome on total scale score §7.1.3 p. 52.

Scoring (as printed on the scale; no reverse-scored items are marked in tab. 1).

# Subscale and total scores  [tab. 1]
# Input:  responses, a list of 13 Likert values (1-5), index 0 = item 1
# Output: verification, motivation, reflection, total

def score_scale(responses):
    # Verification: mean of items 1-5  [tab. 1]
    verification = mean(responses[0:5])
    # Motivation: mean of items 6-9  [tab. 1]
    motivation = mean(responses[5:9])
    # Reflection: mean of items 10-13  [tab. 1]
    reflection = mean(responses[9:13])
    # Total: mean of the three subscale means, not of all 13 items  [tab. 1]
    total = mean([verification, motivation, reflection])
    return verification, motivation, reflection, total

Final 13-item scale tab. 1 p. 20. Preamble: "Think about how you usually use AI tools (e.g., chatbots, image generators, voice assistants). For each statement, indicate how true it is of your typical behaviour using 1 (strongly disagree) to 5 (strongly agree)."

Hyperparameters

Psychometric settings and thresholds. Nothing here is a training hyperparameter.

Results

Author-reported. Setup: Prolific or university samples, self-report scale, GPT-4o chatbot for Study 6 §2–§7.

Scale-level results from the text. Study 3 reports higher-order loadings of .73, .66 and .68 for verification, motivation and reflection §4.2.1 p. 30. Study 3 reports first-order correlations of .48, .49 and .45 p. 29. Study 4 reports a higher-order model with χ²(62) = 144.22 p. 41. The Study 3 nomological model with convergent and discriminant comparators has χ²(134) = 569.29, CFI = .98, TLI = .98, RMSEA = .09, and SRMR = .08 §4.2.3 p. 33. The Study 4 equivalent has χ²(122) = 302.04, CFI = .94, TLI = .93, RMSEA = .07, and SRMR = .06 §5.2.2 p. 42.

Paper vs code

No official code. The paper's text, abstract and author note contain no repository link, and the arXiv comment carries none (the paper refers readers to a DOI). A GitHub repository search for "critical thinking in AI use" returned no results. Supplementary Materials (Tables S1–S14, Figures S1–S4) are referenced throughout but are not in the PDF available to this compilation.

Pitfalls

• Figure and text disagree on second-order loadings. The text gives .73, .66 and .68 for verification, motivation and reflection, with R² = .53, .44 and .46 §4.2.1 p. 30. Figure 1 shows .90, .68 and .77 for the same model fig. 1 p. 31. The R² values match the text figures. Use the text values, and check the figure against the supplementary tables before relying on either.
• "Fit well" versus the paper's own thresholds. Study 4's first-order CFA has RMSEA = .07 (90% CI .05–.08) §5.2.1 p. 41. The paper's stated acceptable threshold is RMSEA ≤ .06 §4.1.3 p. 26. The text calls the fit good anyway. The Study 3 nomological model has RMSEA = .09 §4.2.3 p. 33, which the authors describe as "some residual misfit".
• Study 2 item selection is partly judgement-based. The EFA trimming rules are stated §3.1.3 [pp. 18–19], but the final choice of the 17-item solution, and then the removal of four items for conceptual reasons, is not fully algorithmic p. 21. The 27-item pool and its item numbering are only in Supplementary Table S2, so the removed items cannot be mapped back to the 27-item numbering from the paper alone.
• Study 1 judges are undefined. The 24 naive judges were university students or recent graduates in Singapore. Demographics "were not formally collected" §2.1.2 p. 15. The three subject-matter experts are the paper's senior authors §2.1.1 p. 15.
• The total score is not a 13-item mean. Table 1 defines the total as the average of the three subscale means tab. 1 p. 20. A straight mean of the 13 items gives a different value because the subscales have 5, 4 and 4 items.
• Positive attitudes flip sign between studies. Study 3 reports r = −.13, p = .02 p. 33. Study 4 reports r = .14, p = .14 p. 42. The authors say the link may depend on sample or context p. 69.
• Criterion validity rests on one study. Study 3 found no reliable link with the CRT §4.2.2 p. 32. The authors attribute this to prior exposure to CRT items and to general intelligence loadings on CRT. Criterion validity therefore rests on Study 6 alone, which uses one GPT-4o chatbot and one writing prompt §7.1.2 [pp. 48–51].
• Effect sizes are small. Study 6's veracity-accuracy model explains R² = .06 §7.2 p. 54. The general discussion calls this a "moderate effect size" §8 p. 61. Use the R² values in tables.
• Multiple outcomes, no correction stated. Study 6 runs about six primary and several exploratory OLS regressions without stating a correction for multiple comparisons §7.1.3 [pp. 52–53]. Read the exploratory process indices as exploratory.
• Test–retest samples differ. The 1-week sample is a university subject pool (mean age 21.13) and the 2-week sample is a Telegram recruitment channel (mean age 24.15) §6.1.1 p. 45. The two intervals therefore differ in population as well as in time.
• Samples are mostly U.S. or Singapore-based and online. Study 2–4 samples are U.S. residents recruited via Prolific §3.1.1 p. 17 §4.1.1 p. 22. Study 4 required prior AI use §5.1.1 p. 38. Study 5 used Singapore-based participants. Study 6 did not state residency in the text.
• Cross-sectional self-report. Studies 3, 4 and 6 are cross-sectional. Study 6 measured the scale before the fact-checking task in the same session, and the authors state that the reverse causal reading is implausible given the trait-like nature of the construct p. 70. The writing-task reflection scores come from GPT-4o, cross-checked by Claude Opus 4.1 with ICC = .79 p. 54.
• The fact-checking statements and the chatbot's answers are not in the paper. The 48-item pool is in Supplementary Table S12, and the rubric in Supplementary Figure S4 §7.1.2 [pp. 49, 51]. An implementer replicating Study 6 needs these supplementary files.
• The Big Five measure is partly substituted. In Study 3, five of the 15 BFI-2-XS items were replaced with domain-consistent items from the 30-item BFI-2-S §4.1.2 p. 25. Personality subscale alphas are low (openness .64, extraversion .59) p. 25.

Lineage

• Builds on: Ennis (1985, 2011), Facione (1990a) and Sosu (2013) for the critical-thinking framework §1.1 [pp. 5–8]; Chaiken (1980) and Chen & Chaiken (1999) for the Heuristic-Systematic Model §1.1.1 p. 5. These are not arXiv papers and are not compiled here.
• Item sources: Critical Thinking Toolkit (Stupple et al., 2017), Critical Motivational Scale (Valenzuela et al., 2017), and Critical Thinking Disposition Scale (Sosu, 2013) §2.1.1 p. 14.
• Compared against: Critical Thinking Disposition Scale (Sosu, 2013), Need for Cognition (Cacioppo et al., 1984), Actively Open-minded Thinking (Stanovich & Toplak, 2023), Cognitive Reflection Test-2 (Thomson & Oppenheimer, 2016) §4.1.2 [pp. 23–24].
• Discriminant comparators: attitudes toward AI (Sindermann et al., 2021), Generative AI Dependency Scale (Goh et al., 2025), AI Attachment Scale (Kasturiratna & Hartanto, 2025) §4.1.2 [pp. 24–25] §5.1.2 [pp. 38–39].
• Superseded by: not stated.