---
kind: concept
slug: psychometric-scale-validation
title: Psychometric scale validation
summary: >-
  How to check that a self-report questionnaire measures what it claims: factor structure (CFA), reliability, convergent and discriminant validity, test-retest stability and method bias. Three author-reported validation studies show the checks and their gaps; two user studies use established scales with no such checks. None reports every check with statistics; method bias is the least covered.
tags: [psychometrics, questionnaire, validity, reliability, survey-research]
papers: ["2302.09319", "2401.12247", "2512.12413", "2410.18727", "2412.05024"]
related: [moderation-analysis, human-ai-collaboration, human-evaluation]
status: draft
updated_at: 2026-10-25
---

## Overview

A questionnaire turns a construct that cannot be observed directly, such as AI
literacy, trust in a chatbot or critical thinking about AI output, into a score
built from item answers. Before a study reads the score as the construct, it
checks five things: whether the items group the way the theory says (internal
structure), whether they agree with each other (reliability), whether items meant
for one construct converge and stay separate from other constructs (convergent and
discriminant validity), whether the score stays stable when the same people answer
again (test-retest), and whether the answers reflect the construct rather than a
response habit or a single survey session (method bias).

Three author-reported validation studies show these checks. One validates a 34-item AI literacy
and self-management questionnaire with a confirmatory factor analysis on 300 adults
[arxiv:2302.09319 §3.3]. Another uses 299 consumers and checks measurement with
reliability, loading, AVE and Fornell–Larcker statistics before it tests hypotheses
[arxiv:2401.12247 §3.3.1]. The third builds a 13-item critical-thinking scale over six
studies, with a separate exploratory and confirmatory sample, sex invariance,
test-retest reliability and a criterion test [arxiv:2512.12413 §3] [arxiv:2512.12413 §4.2.1]
[arxiv:2512.12413 §6.2] [arxiv:2512.12413 §7.2].

Two user studies use established scales to compare robot designs. Neither
validates its scale. One teleoperation study uses NASA-TLX and five items of the
NARS S1 subscale [arxiv:2410.18727 §IV]. One Pepper study uses the 18-item RoSAS
[arxiv:2412.05024 §5.1]. Both are covered under their own variant below.

None of the three validation studies reports all four checks with statistics. Method bias is the least
covered: the chatbot study names it without values, and the other two report none
[arxiv:2401.12247 §3.3.2].

## Variants

### Confirmatory factor analysis (CFA)

Fit a model that states which items load on which factors and how the factors
relate, then test how well it reproduces the observed correlations. The MAILS
study fits a two-level model: eight first-order factors load on AI Literacy and
AI Self-management, estimated with a robust Satorra–Bentler method in lavaan
[arxiv:2302.09319 §3.3] [arxiv:2302.09319 fig. 1]. The first version did not
fit. The authors then removed low-loading items (the number is not stated),
moved three items to a new factor, separated one factor from the second-level
construct and split the self-management factor in two. The revised model reports
CFI .926, TLI .920, RMSEA .057 and SRMR .079, described as good and acceptable,
while the chi-square is significant (886.87 on 513 df, p < .001) [arxiv:2302.09319 §3.3].
No cut-off values are given for these indices. The authors call the work
exploratory and ask for tests in independent samples [arxiv:2302.09319 §4.2].

The critical-thinking study fits a first-order CFA of its 13 items with robust
maximum likelihood (MLR) and fixes the latent variances to 1 for identification
[arxiv:2512.12413 §4.1.3] [arxiv:2512.12413 §4.2.1]. In its Study 3 sample of 376,
the fit is χ²(62) = 131.75, CFI 1.00, TLI 1.00, RMSEA .06 (90% CI .04–.07) and SRMR
.05, with standardised loadings of .80–.93, .81–.95 and .68–.89 for its three
subscales [arxiv:2512.12413 §4.2.1]. The same model in its Study 4 sample of 290
gives CFI .97, TLI .96, RMSEA .07 and SRMR .04 [arxiv:2512.12413 §5.2.1].

### Exploratory factor analysis, then a new sample for confirmation

Use one sample to find the structure and a different one to test it. The
critical-thinking study began with 27 candidate items. In a 270-person sample it ran
principal axis factoring with oblimin rotation, chose the number of factors with
parallel analysis, a scree plot and BIC, and removed items with communality below
.35, a primary loading below .40, or a cross-loading of .35 or more with a gap below
.20 [arxiv:2512.12413 §3.1.3]. It kept three factors and 17 items, then removed four
more on conceptual grounds, leaving 13 [arxiv:2512.12413 §3.2]. The confirmatory
work used a new Study 3 sample of 376 [arxiv:2512.12413 §4.1.1].

The MAILS study revised its model on the same 300 respondents that gave the first
fit, and it lists the revisions it made [arxiv:2302.09319 §3.3]. The authors name
this as a limit and ask for independent samples [arxiv:2302.09319 §4.2].

### Reliability and convergent validity with stated thresholds

Reflective constructs are checked with Cronbach's alpha, standardised item loadings
and average variance extracted (AVE). The chatbot study states its cut-offs: loadings
of at least 0.7, AVE of at least 0.5, and a variance inflation factor below 10
[arxiv:2401.12247 §3.3.1] [arxiv:2401.12247 tab. 2]. Its alphas run from .717
(disclosure, two items) to .951 (reliance), and its AVEs from .691 (resistance)
to .910 (reliance) [arxiv:2401.12247 tab. 2]. It does not give an alpha threshold
[arxiv:2401.12247 §3.3.1].

The critical-thinking study uses a lower loading bar: salience at .50 with narrow
confidence intervals, and AVE at .50 [arxiv:2512.12413 §4.1.3]. Its Study 3
alpha is .88 in total and .89, .90 and .80 for the subscales, with composite
reliability (ω) of .90, .90 and .81 and AVEs of .73, .77 and .59 [arxiv:2512.12413 §4.1.2]
[arxiv:2512.12413 §4.2.1]. Study 4 reports alpha .92 in total [arxiv:2512.12413 §5.1.2]. It states no alpha or
ω threshold [arxiv:2512.12413 §4.2.1]. Convergent evidence is the correlation with
measures expected to relate: a critical-thinking disposition scale at r = .70,
need for cognition at .36, and actively open-minded thinking at .20, all with
p < .01 or lower [arxiv:2512.12413 §4.2.3].

The MAILS study reports alpha only for its auxiliary scales: .82 to .88 for the
two attitude subscales, and above .81 for the willingness scales
[arxiv:2302.09319 §3.2.2] [arxiv:2302.09319 §3.2.3]. The card reports no
reliability coefficient for the 34 MAILS items or their facets.

### Discriminant validity

Items for different constructs should not be too strongly related. The chatbot
study applies the Fornell–Larcker test: the square root of each construct's AVE
should exceed its correlations with the other constructs. Its table lists these
square roots, for example 0.933 for friendliness [arxiv:2401.12247 tab. 3].

The critical-thinking study uses the same Fornell–Larcker logic against its comparators
[arxiv:2512.12413 §4.1.3], but its card gives no Fornell–Larcker values. It reports
the correlations with comparators instead, and expects them to be small for
attitudes and AI dependency. The results were r = −.01 for AI dependency, −.13 for
positive attitudes to AI and .11 for negative attitudes [arxiv:2512.12413 §4.2.3].

The MAILS study reports latent correlations of AI Literacy with AI Self-efficacy
(.93) and with AI self-competency (.86) [arxiv:2302.09319 tab. 2]. These are the
pairs where a discriminant test matters most, and the card does not report one for
them.

### Test-retest reliability

Stability over time is checked by giving the same scale to the same people twice.
The critical-thinking study computes ICC(2,1) with absolute agreement from a two-way
random-effects model, and reports Pearson r next to it [arxiv:2512.12413 §6.1.2]. With
a one-week gap (n = 92, a university subject pool) the total score gives r = .70 and
ICC = .70 (CI .58–.79). With a two-week gap (n = 122, recruited through a messaging
channel) it gives r = .65 and ICC = .65 (CI .53–.74) [arxiv:2512.12413 §6.2]. The two
samples differ in age and recruitment as well as in interval [arxiv:2512.12413 §6.1.1],
so the gap cannot be separated from the sample difference. The MAILS and chatbot
cards report no test-retest result.

### Criterion validity

Criterion validity asks whether the score predicts something outside the questionnaire.
The critical-thinking study tests this in one study only (Study 6, n = 191). Its
fact-checking task gave 12 statements through a GPT-4o chatbot, each with one of four
source-link conditions, and the score predicted accuracy with b = 6.91, R² = .06
[arxiv:2512.12413 §7.1.2] [arxiv:2512.12413 §7.2]. It also predicted a self-reported
reflection score (R² = .09) and an evidence-gap resolution index (R² = .08). Evidence
extraction was not significant, and the misread rate was marginal [arxiv:2512.12413 §7.2].
The reflection scores came from a writing task scored by GPT-4o and cross-checked with
Claude Opus 4.1 [arxiv:2512.12413 §7.1.2] [arxiv:2512.12413 §7.1.3]. The process
indices were analysed without a stated correction for multiple comparisons
[arxiv:2512.12413 §7.1.3]. The MAILS study tested no external criteria
[arxiv:2302.09319 §4.1].

### Method bias checks

Answers to a single survey, with one respondent rating every scale, can share a
common method variance. The chatbot study ran Harman's single-factor test and a
marker-variable test and reports both as passed, but gives no test values
[arxiv:2401.12247 §3.3.2]. The MAILS card records no check of this kind
[arxiv:2302.09319 §3]. The critical-thinking card reports no method-bias test. It
collected the scale before the fact-checking task in the same session, and the
authors argue the reverse reading is implausible for a trait-like construct
[arxiv:2512.12413 p. 70].

### Established scales used without a validation step

Two user studies rate robots with published scales and report no check of the
scale itself. The Pepper study uses RoSAS, 18 items in three factors of six, each
factor a sum from 6 to 30 [arxiv:2412.05024 §5.1]. It compares the two conditions
per factor with a t-test and reports p-values only. The card records no reliability
or factor analysis [arxiv:2412.05024 §5.2].

The teleoperation study uses the raw NASA-TLX on a 7-point scale in place of the
21-point scale, and five NARS S1 items [arxiv:2410.18727 §IV]. It treats the NARS
items as a proxy for trust [arxiv:2410.18727 §II-C] and reports no overall NASA-TLX
score [arxiv:2410.18727 §V-B]. Neither card reports a reliability coefficient for
the scales it uses.

## When to use which

| Situation | Use | Why | Source |
|---|---|---|---|
| New multi-factor scale built from a theory | Split the work: explore on one sample, then confirm on a new one | The critical-thinking study used 270 respondents for EFA and a new 376 for CFA [arxiv:2512.12413 §3.1.3] [arxiv:2512.12413 §4.1.1]. The MAILS model was revised on its only sample and the authors call it exploratory [arxiv:2302.09319 §3.3] [arxiv:2302.09319 §4.2] | [Card](/papers/2302.09319) [Card](/papers/2512.12413) |
| A new CFA on about 300 respondents with a revised model | Treat the revised model as a hypothesis; confirm it on a fresh sample | The first CFA did not fit and needed several changes [arxiv:2302.09319 §3.3] | [Card](/papers/2302.09319) |
| Constructs with three or so items, analysed as composites | Alpha or ω, loadings, AVE, VIF and Fornell–Larcker against stated thresholds | The chatbot study states a loading cut-off of 0.7 [arxiv:2401.12247 §3.3.1]. The critical-thinking study uses .50 for loadings [arxiv:2512.12413 §4.1.3]. Pick one, justify it, and report the values | [Card](/papers/2401.12247) [Card](/papers/2512.12413) |
| Two constructs expected to be close, such as AI literacy and self-efficacy | A discriminant test, not only alpha | The MAILS latent correlations are .93 and .86 and the card reports no discriminant test for them [arxiv:2302.09319 tab. 2]. The critical-thinking card gives no Fornell–Larcker values [arxiv:2512.12413 §4.2.3] | [Card](/papers/2302.09319) [Card](/papers/2512.12413) |
| The score must stay stable over time | Test-retest ICC(2,1) with the interval stated | The critical-thinking study gives ICC .70 at one week and .65 at two weeks, in different samples [arxiv:2512.12413 §6.2] | [Card](/papers/2512.12413) |
| One respondent answers all scales in one session | Harman's single-factor test and a marker variable, with the values printed | The chatbot study ran both tests but gave no statistics [arxiv:2401.12247 §3.3.2]. The critical-thinking study reports no such test | [Card](/papers/2401.12247) [Card](/papers/2512.12413) |
| You use a published scale in a new population, or change its format | Treat it as a new instrument: report its reliability in your sample, its scoring direction and any change to the response format | The teleoperation study switched NASA-TLX to a 7-point scale and read NARS items as trust, with no reliability reported [arxiv:2410.18727 §IV] [arxiv:2410.18727 §II-C]. The Pepper study reports t-tests only [arxiv:2412.05024 §5.2] | [Card](/papers/2410.18727) [Card](/papers/2412.05024) |
| You need evidence that the scale predicts behaviour or an outside measure | Collect external criteria; claim only what they show | The critical-thinking study has one criterion study with R² of .06 to .09 [arxiv:2512.12413 §7.2]. MAILS tested no external criteria [arxiv:2302.09319 §4.1] | [Card](/papers/2512.12413) [Card](/papers/2302.09319) |
| You publish your own validation | Report the fit cut-offs, the items removed, the scoring rule and the method-bias statistics | Each of these is missing from at least one paper. The critical-thinking study states a scoring rule (mean of subscale means) [arxiv:2512.12413 tab. 1]; MAILS does not [arxiv:2302.09319 §3.3] | Three cards |

## Implementation sketch

Language-agnostic pseudocode of the common core, with notes on where the three
studies differ. None of them gives a complete scoring rule for a total, so the sketch
stops at the checks.

```pseudocode
# Validation checks in the order the studies use them
# Input:  an exploration sample, a new confirmation sample, the model, the cut-offs
# Output: a report with each statistic, not only pass or fail

def validate_scale(explore, confirm, model, cut, retest):
    # Fix the exclusion rule before looking at results  [arxiv:2401.12247 §3.1]
    # Structure: EFA on one sample, then CFA on a new one  [arxiv:2512.12413 §3]
    items = efa_trim(explore, min_items_per_factor=3)  # PAF, oblimin  [§3.1.3]
    # CFA with a robust estimator; MAILS used robust Satorra-Bentler  [§3.3]
    cfa = fit_cfa(confirm, model, estimator="MLR")  # the critical-thinking choice
    # Fit cut-offs differ by paper; state yours before running  [§4.1.3]
    report(cfa.cfi, cfa.tli, cfa.rmsea, cfa.srmr)
    # Per construct: alpha or omega, loadings, AVE  [arxiv:2401.12247 §3.3.1]
    for c in model.constructs:
        alpha = cronbach_alpha(confirm[c.items])
        # No alpha threshold is stated in any of the three papers; record it
        loadings = standardised_loadings(c)
        check(min(loadings) >= cut.loading)  # 0.7 or 0.5 in the cards
        check(average_squared(loadings) >= cut.ave)  # 0.5 in both
    # Discriminant: sqrt(AVE) above the construct's correlations  [tab. 3]
    check_fornell_larcker(model.constructs)
    # Test-retest: ICC(2,1), absolute agreement  [arxiv:2512.12413 §6.1.2]
    report(icc_2_1(retest.first, retest.second))
    # Method bias: single-factor test and marker variable  [§3.3.2]
    report(harman_single_factor(confirm), marker_variable_test(confirm))
    return report
```

## Contradictions

- **Fit standards differ.** MAILS calls CFI .926, TLI .920 and RMSEA .057 good, and
  states no cut-offs [arxiv:2302.09319 §3.3]. The critical-thinking study calls CFI and
  TLI of .90 acceptable and .95 good, with RMSEA of .06 as its acceptable limit
  [arxiv:2512.12413 §4.1.3]. Read against those numbers, the MAILS values fall in the
  acceptable range; that comparison is this page's, not the authors'. The chatbot
  study runs no factor analysis and gives no fit values [arxiv:2401.12247 §3.3.1].
- **Fit called good above the stated limit.** The critical-thinking study's Study 4 CFA
  has RMSEA .07 against its own limit of .06, and the text still calls the fit good
  [arxiv:2512.12413 §5.2.1] [arxiv:2512.12413 §4.1.3]. The Study 3 nomological model
  has RMSEA .09, which the authors describe as some residual misfit [arxiv:2512.12413 §4.2.3].
- **Second-order loadings: text and figure.** The critical-thinking text gives
  .73, .66 and .68 for verification, motivation and reflection, while its Figure 1 shows
  .90, .68 and .77 for the same model [arxiv:2512.12413 §4.2.1] [arxiv:2512.12413 fig. 1].
  The paper's text values are the ones cited here; the figure is not resolved.
- **Loading threshold within one paper.** The chatbot study's text says all loadings
  exceed 0.7 [arxiv:2401.12247 §3.3.1], but its table lists trust item TRU1 at
  0.671 [arxiv:2401.12247 tab. 2]. Both statements are reported here; the paper
  does not resolve them.
- **Sign of an attitude correlation.** The critical-thinking study finds positive
  attitudes to AI at r = −.13 in Study 3 [arxiv:2512.12413 §4.2.3] and r = .14 in Study 4
  [arxiv:2512.12413 §5.2.2]. Its authors suggest the link depends on sample or context.
  The attitude result is significant in one study and reversed, and not significant, in the other.
- **NARS scoring direction.** The teleoperation card's figure caption says a low NARS
  score means more trust, while its text reads the higher-trust result for the Japanese
  group from the same items [arxiv:2410.18727 §V-C] [fig. 12]. Its table lists Japan
  with the lower score (P < 0.05). The paper does not resolve this; check the scale's
  key before comparing with other work.

## Papers

- [MAILS -- Meta AI Literacy Scale: Development and Testing of an AI Literacy Questionnaire Based on Well-Founded Competency Models and Psychological Change- and Meta-Competencies](/papers/2302.09319): a 34-item questionnaire validated with CFA, with post hoc revisions and no external criteria.
- [Exploring consumers response to text-based chatbots in e-commerce: The moderating role of task complexity and chatbot disclosure](/papers/2401.12247): a survey of 299 consumers whose measurement checks give alpha, AVE, loading, Fornell–Larcker and method-bias results.
- [Understanding Critical Thinking in Generative Artificial Intelligence Use: Development, Validation, and Correlates of the Critical Thinking in AI Use Scale](/papers/2512.12413): a 13-item scale validated over six studies with EFA, CFA, sex invariance, test-retest reliability and a criterion study.
- [Breaking Down the Barriers: Investigating Non-Expert User Experiences in Robotic Teleoperation in UK and Japan](/papers/2410.18727): a 74-person user study that uses NASA-TLX and NARS S1 items without validating them.
- [Talking Like One of Us: Effects of Using Regional Language in a Humanoid Social Robot](/papers/2412.05024): a 17-person within-subject study that uses RoSAS and reports t-tests only.