Psychometric scale validation

Overview

A questionnaire turns a construct that cannot be observed directly, such as AI literacy, trust in a chatbot or critical thinking about AI output, into a score built from item answers. Before a study reads the score as the construct, it checks five things: whether the items group the way the theory says (internal structure), whether they agree with each other (reliability), whether items meant for one construct converge and stay separate from other constructs (convergent and discriminant validity), whether the score stays stable when the same people answer again (test-retest), and whether the answers reflect the construct rather than a response habit or a single survey session (method bias).

Three author-reported validation studies show these checks. One validates a 34-item AI literacy and self-management questionnaire with a confirmatory factor analysis on 300 adults 2302.09319 §3.3. Another uses 299 consumers and checks measurement with reliability, loading, AVE and Fornell–Larcker statistics before it tests hypotheses 2401.12247 §3.3.1. The third builds a 13-item critical-thinking scale over six studies, with a separate exploratory and confirmatory sample, sex invariance, test-retest reliability and a criterion test 2512.12413 §3 2512.12413 §4.2.1 2512.12413 §6.2 2512.12413 §7.2.

Two user studies use established scales to compare robot designs. Neither validates its scale. One teleoperation study uses NASA-TLX and five items of the NARS S1 subscale 2410.18727 §IV. One Pepper study uses the 18-item RoSAS 2412.05024 §5.1. Both are covered under their own variant below.

None of the three validation studies reports all four checks with statistics. Method bias is the least covered: the chatbot study names it without values, and the other two report none 2401.12247 §3.3.2.

Variants

Confirmatory factor analysis (CFA)

Fit a model that states which items load on which factors and how the factors relate, then test how well it reproduces the observed correlations. The MAILS study fits a two-level model: eight first-order factors load on AI Literacy and AI Self-management, estimated with a robust Satorra–Bentler method in lavaan 2302.09319 §3.3 2302.09319 fig. 1. The first version did not fit. The authors then removed low-loading items (the number is not stated), moved three items to a new factor, separated one factor from the second-level construct and split the self-management factor in two. The revised model reports CFI .926, TLI .920, RMSEA .057 and SRMR .079, described as good and acceptable, while the chi-square is significant (886.87 on 513 df, p < .001) 2302.09319 §3.3. No cut-off values are given for these indices. The authors call the work exploratory and ask for tests in independent samples 2302.09319 §4.2.

The critical-thinking study fits a first-order CFA of its 13 items with robust maximum likelihood (MLR) and fixes the latent variances to 1 for identification 2512.12413 §4.1.3 2512.12413 §4.2.1. In its Study 3 sample of 376, the fit is χ²(62) = 131.75, CFI 1.00, TLI 1.00, RMSEA .06 (90% CI .04–.07) and SRMR .05, with standardised loadings of .80–.93, .81–.95 and .68–.89 for its three subscales 2512.12413 §4.2.1. The same model in its Study 4 sample of 290 gives CFI .97, TLI .96, RMSEA .07 and SRMR .04 2512.12413 §5.2.1.

Exploratory factor analysis, then a new sample for confirmation

Use one sample to find the structure and a different one to test it. The critical-thinking study began with 27 candidate items. In a 270-person sample it ran principal axis factoring with oblimin rotation, chose the number of factors with parallel analysis, a scree plot and BIC, and removed items with communality below .35, a primary loading below .40, or a cross-loading of .35 or more with a gap below .20 2512.12413 §3.1.3. It kept three factors and 17 items, then removed four more on conceptual grounds, leaving 13 2512.12413 §3.2. The confirmatory work used a new Study 3 sample of 376 2512.12413 §4.1.1.

The MAILS study revised its model on the same 300 respondents that gave the first fit, and it lists the revisions it made 2302.09319 §3.3. The authors name this as a limit and ask for independent samples 2302.09319 §4.2.

Reliability and convergent validity with stated thresholds

Reflective constructs are checked with Cronbach's alpha, standardised item loadings and average variance extracted (AVE). The chatbot study states its cut-offs: loadings of at least 0.7, AVE of at least 0.5, and a variance inflation factor below 10 2401.12247 §3.3.1 2401.12247 tab. 2. Its alphas run from .717 (disclosure, two items) to .951 (reliance), and its AVEs from .691 (resistance) to .910 (reliance) 2401.12247 tab. 2. It does not give an alpha threshold 2401.12247 §3.3.1.

The critical-thinking study uses a lower loading bar: salience at .50 with narrow confidence intervals, and AVE at .50 2512.12413 §4.1.3. Its Study 3 alpha is .88 in total and .89, .90 and .80 for the subscales, with composite reliability (ω) of .90, .90 and .81 and AVEs of .73, .77 and .59 2512.12413 §4.1.2 2512.12413 §4.2.1. Study 4 reports alpha .92 in total 2512.12413 §5.1.2. It states no alpha or ω threshold 2512.12413 §4.2.1. Convergent evidence is the correlation with measures expected to relate: a critical-thinking disposition scale at r = .70, need for cognition at .36, and actively open-minded thinking at .20, all with p < .01 or lower 2512.12413 §4.2.3.

The MAILS study reports alpha only for its auxiliary scales: .82 to .88 for the two attitude subscales, and above .81 for the willingness scales 2302.09319 §3.2.2 2302.09319 §3.2.3. The card reports no reliability coefficient for the 34 MAILS items or their facets.

Discriminant validity

Items for different constructs should not be too strongly related. The chatbot study applies the Fornell–Larcker test: the square root of each construct's AVE should exceed its correlations with the other constructs. Its table lists these square roots, for example 0.933 for friendliness 2401.12247 tab. 3.

The critical-thinking study uses the same Fornell–Larcker logic against its comparators 2512.12413 §4.1.3, but its card gives no Fornell–Larcker values. It reports the correlations with comparators instead, and expects them to be small for attitudes and AI dependency. The results were r = −.01 for AI dependency, −.13 for positive attitudes to AI and .11 for negative attitudes 2512.12413 §4.2.3.

The MAILS study reports latent correlations of AI Literacy with AI Self-efficacy (.93) and with AI self-competency (.86) 2302.09319 tab. 2. These are the pairs where a discriminant test matters most, and the card does not report one for them.

Test-retest reliability

Stability over time is checked by giving the same scale to the same people twice. The critical-thinking study computes ICC(2,1) with absolute agreement from a two-way random-effects model, and reports Pearson r next to it 2512.12413 §6.1.2. With a one-week gap (n = 92, a university subject pool) the total score gives r = .70 and ICC = .70 (CI .58–.79). With a two-week gap (n = 122, recruited through a messaging channel) it gives r = .65 and ICC = .65 (CI .53–.74) 2512.12413 §6.2. The two samples differ in age and recruitment as well as in interval 2512.12413 §6.1.1, so the gap cannot be separated from the sample difference. The MAILS and chatbot cards report no test-retest result.

Criterion validity

Criterion validity asks whether the score predicts something outside the questionnaire. The critical-thinking study tests this in one study only (Study 6, n = 191). Its fact-checking task gave 12 statements through a GPT-4o chatbot, each with one of four source-link conditions, and the score predicted accuracy with b = 6.91, R² = .06 2512.12413 §7.1.2 2512.12413 §7.2. It also predicted a self-reported reflection score (R² = .09) and an evidence-gap resolution index (R² = .08). Evidence extraction was not significant, and the misread rate was marginal 2512.12413 §7.2. The reflection scores came from a writing task scored by GPT-4o and cross-checked with Claude Opus 4.1 2512.12413 §7.1.2 2512.12413 §7.1.3. The process indices were analysed without a stated correction for multiple comparisons 2512.12413 §7.1.3. The MAILS study tested no external criteria 2302.09319 §4.1.

Method bias checks

Answers to a single survey, with one respondent rating every scale, can share a common method variance. The chatbot study ran Harman's single-factor test and a marker-variable test and reports both as passed, but gives no test values 2401.12247 §3.3.2. The MAILS card records no check of this kind 2302.09319 §3. The critical-thinking card reports no method-bias test. It collected the scale before the fact-checking task in the same session, and the authors argue the reverse reading is implausible for a trait-like construct 2512.12413 p. 70.

Established scales used without a validation step

Two user studies rate robots with published scales and report no check of the scale itself. The Pepper study uses RoSAS, 18 items in three factors of six, each factor a sum from 6 to 30 2412.05024 §5.1. It compares the two conditions per factor with a t-test and reports p-values only. The card records no reliability or factor analysis 2412.05024 §5.2.

The teleoperation study uses the raw NASA-TLX on a 7-point scale in place of the 21-point scale, and five NARS S1 items 2410.18727 §IV. It treats the NARS items as a proxy for trust 2410.18727 §II-C and reports no overall NASA-TLX score 2410.18727 §V-B. Neither card reports a reliability coefficient for the scales it uses.

When to use which

Implementation sketch

Language-agnostic pseudocode of the common core, with notes on where the three studies differ. None of them gives a complete scoring rule for a total, so the sketch stops at the checks.

# Validation checks in the order the studies use them
# Input:  an exploration sample, a new confirmation sample, the model, the cut-offs
# Output: a report with each statistic, not only pass or fail

def validate_scale(explore, confirm, model, cut, retest):
    # Fix the exclusion rule before looking at results  [arxiv:2401.12247 §3.1]
    # Structure: EFA on one sample, then CFA on a new one  [arxiv:2512.12413 §3]
    items = efa_trim(explore, min_items_per_factor=3)  # PAF, oblimin  [§3.1.3]
    # CFA with a robust estimator; MAILS used robust Satorra-Bentler  [§3.3]
    cfa = fit_cfa(confirm, model, estimator="MLR")  # the critical-thinking choice
    # Fit cut-offs differ by paper; state yours before running  [§4.1.3]
    report(cfa.cfi, cfa.tli, cfa.rmsea, cfa.srmr)
    # Per construct: alpha or omega, loadings, AVE  [arxiv:2401.12247 §3.3.1]
    for c in model.constructs:
        alpha = cronbach_alpha(confirm[c.items])
        # No alpha threshold is stated in any of the three papers; record it
        loadings = standardised_loadings(c)
        check(min(loadings) >= cut.loading)  # 0.7 or 0.5 in the cards
        check(average_squared(loadings) >= cut.ave)  # 0.5 in both
    # Discriminant: sqrt(AVE) above the construct's correlations  [tab. 3]
    check_fornell_larcker(model.constructs)
    # Test-retest: ICC(2,1), absolute agreement  [arxiv:2512.12413 §6.1.2]
    report(icc_2_1(retest.first, retest.second))
    # Method bias: single-factor test and marker variable  [§3.3.2]
    report(harman_single_factor(confirm), marker_variable_test(confirm))
    return report

Contradictions

• Fit standards differ. MAILS calls CFI .926, TLI .920 and RMSEA .057 good, and states no cut-offs 2302.09319 §3.3. The critical-thinking study calls CFI and TLI of .90 acceptable and .95 good, with RMSEA of .06 as its acceptable limit 2512.12413 §4.1.3. Read against those numbers, the MAILS values fall in the acceptable range; that comparison is this page's, not the authors'. The chatbot study runs no factor analysis and gives no fit values 2401.12247 §3.3.1.
• Fit called good above the stated limit. The critical-thinking study's Study 4 CFA has RMSEA .07 against its own limit of .06, and the text still calls the fit good 2512.12413 §5.2.1 2512.12413 §4.1.3. The Study 3 nomological model has RMSEA .09, which the authors describe as some residual misfit 2512.12413 §4.2.3.
• Second-order loadings: text and figure. The critical-thinking text gives .73, .66 and .68 for verification, motivation and reflection, while its Figure 1 shows .90, .68 and .77 for the same model 2512.12413 §4.2.1 2512.12413 fig. 1. The paper's text values are the ones cited here; the figure is not resolved.
• Loading threshold within one paper. The chatbot study's text says all loadings exceed 0.7 2401.12247 §3.3.1, but its table lists trust item TRU1 at 0.671 2401.12247 tab. 2. Both statements are reported here; the paper does not resolve them.
• Sign of an attitude correlation. The critical-thinking study finds positive attitudes to AI at r = −.13 in Study 3 2512.12413 §4.2.3 and r = .14 in Study 4 2512.12413 §5.2.2. Its authors suggest the link depends on sample or context. The attitude result is significant in one study and reversed, and not significant, in the other.
• NARS scoring direction. The teleoperation card's figure caption says a low NARS score means more trust, while its text reads the higher-trust result for the Japanese group from the same items 2410.18727 §V-C fig. 12. Its table lists Japan with the lower score (P < 0.05). The paper does not resolve this; check the scale's key before comparing with other work.

Papers

• MAILS -- Meta AI Literacy Scale: Development and Testing of an AI Literacy Questionnaire Based on Well-Founded Competency Models and Psychological Change- and Meta-Competencies: a 34-item questionnaire validated with CFA, with post hoc revisions and no external criteria.
• Exploring consumers response to text-based chatbots in e-commerce: The moderating role of task complexity and chatbot disclosure: a survey of 299 consumers whose measurement checks give alpha, AVE, loading, Fornell–Larcker and method-bias results.
• Understanding Critical Thinking in Generative Artificial Intelligence Use: Development, Validation, and Correlates of the Critical Thinking in AI Use Scale: a 13-item scale validated over six studies with EFA, CFA, sex invariance, test-retest reliability and a criterion study.
• Breaking Down the Barriers: Investigating Non-Expert User Experiences in Robotic Teleoperation in UK and Japan: a 74-person user study that uses NASA-TLX and NARS S1 items without validating them.
• Talking Like One of Us: Effects of Using Regional Language in a Humanoid Social Robot: a 17-person within-subject study that uses RoSAS and reports t-tests only.