MAILS -- Meta AI Literacy Scale: Development and Testing of an AI Literacy Questionnaire Based on Well-Founded Competency Models and Psychological Change- and Meta-Competencies

In plain words

Many people now use AI at work, but there is no widely used way to measure how well someone can handle it. The authors wrote a questionnaire that asks people to rate, on a scale from 0 to 10, how well they think they can do things with AI: use it, understand what it is, spot it, judge its ethics, and build it. They also ask about a second group of skills: keeping up with change, solving problems with AI, and not being pushed around by it or by one's own emotions.

The questionnaire is built from earlier models of what AI literacy consists of, so its parts can be used on their own depending on what you need to know. The authors gave it to 300 adults online and checked whether the answers fell into the groups they expected. The groups fitted reasonably well, but the authors changed the structure after seeing the data, and it has not yet been checked against real behaviour or against other tests.

TL;DR

• MAILS is a German-language self-report questionnaire for AI literacy with 34 items on a 0 to 10 scale app. A §3.2.1.
• Its AI literacy part has four facets (use and apply, know and understand, detect, ethics) plus Create AI as a separate construct; its self-management part has self-efficacy (problem solving, learning) and self-competency (persuasion literacy, emotion regulation) fig. 2 app. A.
• Confirmatory factor analysis on 300 adults gave acceptable to good fit for the modified model, but the model was changed after seeing the data and has not been tested on new samples or against outside criteria §3.3 §4.1.

Problem

Existing AI literacy scales were, according to the authors, mostly built for one educational intervention, were not validated, or did not separate facets of AI literacy. They also did not include psychological competencies such as emotion regulation or problem solving with AI §2.1 §1. The authors argue that a self-assessment grounded in competency models, and modular enough for professional use, is needed to predict and support the use of AI at work §2.2 §2.3.

Method

The authors built an item pool, validated its factor structure, and then reported the revised model.

1. Theoretical base. AI literacy facets come from the conceptualisation of Ng and colleagues [33], which the authors map onto Bloom's taxonomy [7] §2. Self- assessment is grounded in the theory of planned behaviour [1] and in self-efficacy [5]; the psychological competencies are AI self-management: problem solving, learning, and emotion regulation, plus recognising AI influence §2.2 §2.3.
2. Item generation. For each domain, items were written, then discussed, rephrased, rejected and finalised by the research team. From 84 items, 68 were kept for analysis: 12 on AI self-management and 56 on the AI literacy facets §3.2.1. Each item states a specific ability; respondents rate it on an 11-point scale from 0 (hardly or not at all) to 10 (very well or almost perfectly) §3.2.1.
3. Auxiliary measures. Attitude toward AI was measured with a German translation of the General Attitude towards AI Scale (GAAIS), 20 items on 5-point Likert, split into positive and negative subscales [39] §3.2.2. Willingness to use technology was measured with a short scale (acceptance, competence, control; four items each, 5-point) [31] §3.2.3. Big Five traits and demographics were also collected, but the paper reports no results for them §3.
4. Fieldwork. Online survey (SoSci-Survey) with Prolific recruitment in early November 2022; 300 German-speaking adults; completion time 16:05 minutes on average (SD 5:46) §3 §3.1.
5. First CFA (conceptual model). Confirmatory factor analysis with all items. The conceptual model has eight first-order factors (Know & understand AI, Use & apply AI, Evaluate & create AI, AI ethics, AI problem solving, AI learning, AI persuasion literacy, AI emotion regulation), which load on two second-order factors: AI Literacy and AI Self-management. Each first-order factor has three to fifteen variables §3.3 fig. 1. The first model did not fit, so it was revised.
6. Revisions §3.3 fig. 2:
   1. Items with low factor loadings were removed (the number is not stated).
   2. Three items from Know & understand AI were moved to a factor named Detect AI.
   3. The first-level factor Create AI was left out of the second-level factor AI Literacy, and is modelled as a correlated separate factor.
   4. The second-level factor AI Self-management was split into AI Self-efficacy (problem solving and learning) and AI Self-competency (persuasion literacy and emotion regulation).
7. Second CFA (modified model). Fit reported in §3.3 and fig. 2.
8. Third CFA (with attitude and willingness). Attitude and willingness subscales added as latent variables; all their items load significantly §3.3. Latent correlations are reported in tab. 2.

Estimation used lavaan 0.6-12 in R with robust Satorra-Bentler estimation [38] §3.3.

The final item set is listed in app. A, in six groups: Use & Apply AI (6 items), Know & Understand AI (6), Detect AI (3), AI Ethics (3), Create AI (4), AI Problem Solving (3), AI Learning (3), AI Persuasion Literacy (3), AI Emotion Regulation (3). The appendix totals 34 items, of which 22 belong to AI Literacy and 12 to AI Self-management app. A. The paper does not state the scoring rule (for example, how items combine into a facet score).

# Final facet structure of MAILS  [app. A] [fig. 2]
# Input:  one 0-10 rating per item (34 items)  [§3.2.1]
# Output: facet membership; scoring rule not stated in the paper

# Item numbers are the order of the items in [app. A], 1 to 34.
facets = {
    "use_and_apply_ai": [1, 2, 3, 4, 5, 6],
    "know_and_understand_ai": [7, 8, 9, 10, 11, 12],
    "detect_ai": [13, 14, 15],
    "ai_ethics": [16, 17, 18],
    "create_ai": [19, 20, 21, 22],
    "ai_problem_solving": [23, 24, 25],
    "ai_learning": [26, 27, 28],
    "ai_persuasion_literacy": [29, 30, 31],
    "ai_emotion_regulation": [32, 33, 34],
}

# Second-order structure  [fig. 2] [tab. 2]
# AI Literacy: use_and_apply_ai, know_and_understand_ai, detect_ai, ai_ethics
# Create AI: separate factor, correlated with AI Literacy (r = .49)
# AI Self-efficacy: ai_problem_solving, ai_learning
# AI Self-competency: ai_persuasion_literacy, ai_emotion_regulation

Hyperparameters

The paper has no training hyperparameters. The table lists the estimation and design settings that are stated.

Results

Author-reported. Setup: 300 German-speaking adults (mean age 32.13 years, SD 11.66, range 18 to 72; 48.33% female, 50.67% male, 1% diverse; 77% in Germany, 7% in Austria) §3.1. No baseline or competing scale was validated against this sample §4.1.

Model fit, from §3.3:

Standardised loadings and correlations shown in fig. 2:

Latent correlations of AI Literacy (AIL), Create AI, AI self-competency (AISC) and AI self-efficacy (AISE) with the auxiliary scales tab. 2:

Note: the cells of tab. 2 are read from the PDF. Significance markers (* p < .05, ** p < .01, *** p < .001) are omitted here; see the original table for them. Values are as printed and were not checked against any raw data.

Descriptive statistics for attitude and willingness subscales tab. 1: positive attitude mean 3.60 (SD 0.57); negative attitude 2.74 (SD 0.67); acceptance 3.56 (SD 0.94); competence 4.83 (SD 0.82); control 3.85 (SD 0.68). Reliability: Cronbach's α .88 (positive attitude), .82 (negative attitude), all willingness scales above .81 §3.2.2 §3.2.3.

Paper vs code

No official code. The paper gives no repository link and no project page. Searched the GitHub repository API for "MAILS Meta AI Literacy Scale" and "AI literacy questionnaire Carolus": one hit, a browser-based self-assessment by a third party, which is not an author repository and was not used. No code was cloned.

Pitfalls

• Post hoc model. The final model is the result of several changes made after the first fit was poor: removing items, moving three items to a new factor, dropping one second-level link, and splitting a factor §3.3. The authors call the work exploratory and say future research should test the structure in independent samples §4.2 §4.1. Implement the item grouping as a hypothesis, not as a validated fact.
• Fit criteria are not stated. Chi-square is significant in both models (p < .001). The authors describe CFI .926 and RMSEA .057 as good §3.3. They do not state the cut-off values used.
• Items removed but not listed. The number of items removed for low loading is not stated §3.3, and the paper does not say that the 34 items in app. A are the final set. The pool dropped from 84 to 68 before the CFA §3.2.1; the reader must infer the final count from the appendix.
• Scoring is undefined. No rule combines items into facet scores or a total. Pseudocode above gives membership only. Any score you compute is your own choice.
• Facet naming differs. The abstract says "Understand AI" and "Evaluate and create AI"; the body and appendix use "Know & understand AI" and "Create AI". The appendix shows no "Evaluate" facet, and the text says the Evaluate & create AI items loaded on Know & understand AI instead §4 §4.2.
• Citation years. The appendix cites Ng et al., 2022, but the reference list gives 2021 for the same conceptualisation [33] app. A.
• Missing data and item type. Missing-data handling and whether the 0 to 10 items were treated as continuous are not stated §3.3.
• Unreported measures. Big Five and demographic data were collected §3. The paper reports only the sample demographics, not Big Five results.
• Translation. The GAAIS items were translated by the authors into German; the translation is not provided and the English scale is cited only by reference [39] §3.2.2.
• Sample limits. German speakers in Germany and Austria, recruited online, with a 1% share of diverse gender identity. The authors state that the sample is specific and that no external criteria were tested §4.1.
• Create AI is modelled separately. The authors found it correlates with AI Literacy at r = .49 but does not load on it, which contradicts the Ng conceptualisation §4 fig. 2. Do not assume a total AI literacy score includes Create AI.
• Survey length and time. The mean completion time was 16:05 minutes (SD 5:46) for 300 respondents, which is the only effort figure given §3.
• Table 2 has no caption in the PDF. It is referred to as "Table 2" in the text §3.3; the values are as printed on page 9 p. 9.

Lineage

• Builds on: the AI literacy conceptualisation of Ng and colleagues [33] (four domains: know and understand, use and apply, evaluate and create, ethics), Bloom's taxonomy [7], the theory of planned behaviour [1], and the digital interaction literacy model of Carolus et al. [8].
• Compared against: an AI readiness scale for medical students [20], and an AI literacy scale with 12 items [45]. The paper states that neither bases its conceptualisation on the competency models used here §2.1.
• Superseded by: not known from content already in this wiki.