Differentiating Between Human-Written and AI-Generated Texts Using Automatically Extracted Linguistic Features

In plain words

Many people now ask whether a piece of writing came from a person or from a chatbot. This small study took ten short essays on everyday topics. Five were written by people for an English language exam, and five were written by ChatGPT from the same kind of prompt, at about the same length. Instead of judging the essays by reading them, the author ran each one through a web tool that counts features of the text: its sounds, its word types, its sentence parts and how difficult its words are.

The ChatGPT essays looked fluent, but the counts differed. They used more nouns, more of certain consonant sounds and more difficult words, and fewer pronouns and function words. The author suggests that such differences could help detect AI text or guide how models are trained. The study is small and exploratory, so it is a starting point for larger work, not a general rule.

TL;DR

• Ten essays: five authentic IELTS writing task samples and five GPT-4 essays prompted to match them §2.1 p. 5.
• Each text was profiled by the Open Brain AI web platform into 94 measures (22 phonological, 15 morphological, 44 syntactic, 13 lexical) §2.1 p. 6.
• Univariate Wilcoxon rank-sum tests with Hedges' g, then one LASSO model per linguistic domain §2.2 p. 6.
• Author-reported differences are large in effect size but rest on five texts per group. No code or data is released.

Problem

Detecting AI-written text with black-box scores is hard to explain, and published comparisons of human and ChatGPT writing are few §1 p. 1–4. The paper asks which phonological, morphological, syntactic and lexical features separate the two kinds of text, using measures that can be read off directly §1.2 p. 4. The author calls the study exploratory and not exhaustive §1.2 p. 4.

Method

The study has five steps. The texts are normalised lightly, then profiled by the platform's linguistic module. Features are kept if they are counts or normalised frequencies and appear in at least 80% of texts §2.1 p. 5–6. Each kept feature is tested univariately. Finally, a LASSO model is fitted per domain with AI or human source as the binary outcome §2.2 p. 6.

1. Corpus. Five human essays are IELTS task samples written by professional English teachers, aimed at B2–C1 level. Five essays are generated by ChatGPT (GPT-4, March 2024 version) through the web interface with default platform settings. Each generated text targets the length of its human counterpart. The prompts ask for academic English and structure typical of IELTS Band 8 §2.1 p. 5.
2. Normalisation. Unicode is normalised to UTF-8; whitespace is trimmed and collapsed; smart quotes become straight quotes and em/en dashes become hyphens; HTML tags and copy-paste artifacts are removed. Words, punctuation and case are kept §2.1 p. 5.
3. Profiling. Each text goes through the Open Brain AI beta web platform's linguistic profiling module (readability, phonology, morphology, syntax and lexis). The platform runs its own NLP pipeline (tokenisation, lemmatisation, POS tagging, parsing). The paper uses the per-document table as output without changes §2.1 p. 5–6.
4. Feature filter. Keep counts or normalised frequencies of constituents (words, syllables, clauses or phrases). Keep features present in at least 80% of texts. Readability indices are analysed descriptively §2.1 p. 6.
5. Univariate tests. For each feature, a Wilcoxon rank-sum test (W, p) and Hedges' g with 95% CI. Holm correction is applied "when appropriate" §2.2 p. 6.
6. Multivariate selection. One LASSO model per domain (phonology, morphology, syntax, lexicon). The outcome is AI versus human. The model is fitted with glmnet, family binomial, α = 1, on unscaled features. The penalty is the λ that minimises cross-validated classification error (λ_min). Non-zero coefficients at λ_min are the selected features. Positive weights mean more typical of AI text, negative weights more typical of human text §2.2 p. 6.

# Input: five human essays, five generated essays (plain text)
# Output: univariate tests and one LASSO model per linguistic domain

def analyse(human_texts, ai_texts):
    # 1. Light, content-preserving normalisation  [§2.1] [p. 5]
    texts = [normalise(t) for t in human_texts + ai_texts]

    # 2. One row per text, one column per measure  [§2.1] [p. 5–6]
    #    The paper does not state how ratios are normalised.
    table = platform_profile(texts)

    # 3. Keep count or normalised measures in >= 80% of texts  [§2.1] [p. 6]
    kept = [m for m in table.measures if share_present(m) >= 0.8]

    # 4. Per measure: rank-sum test and Hedges' g  [§2.2] [p. 6]
    for m in kept:
        p, W = wilcoxon_rank_sum(m.human, m.ai)
        # Sign convention inferred from the Results text: positive = AI higher
        # [§3] [p. 7]
        g, ci = hedges_g(m.ai, m.human)

    # 5. Holm correction, scope not stated  [§2.2] [p. 6]
    p_adjusted = holm(all_p_values)

    # 6. One binomial LASSO per domain, alpha = 1, lambda at minimum CV error
    #    [§2.2] [p. 6]
    for domain in [phonology, morphology, syntax, lexicon]:
        coefs = lasso_cv(domain.measures, label=is_ai)
        selected = [m for m in domain.measures if coefs[m] != 0]

    return tests, selected

Hyperparameters

The paper has no training hyperparameters. The table lists the study's settings and what is not stated.

Results

Author-reported. Setup: 5 authentic IELTS task samples and 5 GPT-4 essays, measured by the Open Brain AI platform, tested with Wilcoxon rank-sum and per-domain LASSO §2.1–2.2 p. 5–6. Table 2 is reproduced in full; p-values are as printed and the sign of g is as printed. The Results text reads positive g as higher in AI text (see Pitfalls).

Univariate Wilcoxon results, all 35 tested features tab. 2 p. 8–9:

Readability indices, descriptive only, columns human then AI tab. 1 p. 7–8:

LASSO coefficients at λ_min, one model per domain, original scale tab. 3 p. 11–12. The column is headed "p-value" in the paper but holds coefficients. Features not listed as non-zero have coefficient 0.

Headline reading the author gives: the human texts have more adjectives, adpositions, auxiliaries and pronouns, and the AI texts have more coordinating conjunctions, nouns, difficult words and content words §3 p. 7; §4 p. 12–13.

Paper vs code

No official code. The paper gives no code link, no project page, and no data repository. Data is "available on request" and not public, because the human essays are under copyright and licence [data availability statement] p. 15. The platform, Open Brain AI, is a web service §2.1 p. 5. The raw arXiv metadata has no comments field. GitHub repository searches through the API were run on the title words and the author name, on the author's lab ("Georgiou Phonetic Lab"), on the platform ("Open Brain AI Themistocleous") and on the topic ("ChatGPT essays linguistic features human AI"). Each returned zero repositories.

Pitfalls

• Five texts per group. The sample is five human and five generated texts (n = 5 per group in each test). The paper calls the sample limited §2.1 p. 6 and exploratory §1.2 p. 4. Treat the effect sizes as descriptive, not as population estimates.
• Generator prompt confound. The prompt asks for IELTS Band 8 style, while the human essays are described as B2–C1 §2.1 p. 5. Differences may reflect the prompt's target register as well as the model.
• Only one model and one configuration. The study uses GPT-4 (March 2024) through the web interface with defaults. No other model, sampling setting or version is tested §2.1 p. 5.
• Feature counts do not add up. 94 measures are extracted and Table 2 tests 35 of them (13 phonology, 10 morphology, 9 syntax, 3 lexicon). The filter that removed the rest is not shown §2.1 p. 6 tab. 2 p. 8–9. "9201 observations" §2.1 p. 6 does not match 94 measures × 10 texts = 940; the paper does not explain the count.
• Normalisation unknown. The paper says features are counts or normalised frequencies, but not which denominator (words, syllables, clauses) each one uses §2.1 p. 6. Implementers need the platform's definitions to reproduce the values.
• Orthographic, not acoustic, phonology. The phonological measures are inferred from spelling through grapheme-to-phoneme conversion, not from audio. The paper says this may bias the results for irregular or reduced spellings §4 p. 12.
• Sign convention. Table 2 gives Hedges' g, but the sign direction is stated only in the Results text, which describes the AI text as having more difficult and content words (difficult words g = 2.90) §3 p. 7. Read positive g as higher in AI text, and treat that as inferred from the text.
• p-value and CI disagree in places. Several rows have p < 0.05 while the 95% CI includes 0 (e.g. object of preposition p = 0.0465, CI −2.93 to 0.03; voiceless p = 0.0367, CI 0.00 to 2.98; prepositional modifier p = 0.0367, CI −3.22 to 0.16). The p-value comes from the Wilcoxon test and the CI from effsize, so they are not expected to match tab. 2 p. 8–9.
• Holm scope and threshold unclear. Holm correction is applied "when appropriate", and the significance level is never stated. Whether Table 2's p-values are adjusted is not stated §2.2 p. 6.
• Primary stress: significant in LASSO, not in tests. Primary stress has p = 0.2963 and g = −1.00 tab. 2 p. 8, yet it has the largest-magnitude LASSO coefficient tab. 3 p. 11. The Discussion says it "did not differ significantly" but is the most influential phonological predictor §4 p. 12.
• LASSO coefficients are unscaled. All variables are used on their original scale §2.2 p. 6. Coefficient magnitudes (for example −269.2 and 0.017) therefore depend on each measure's units and cannot be compared directly.
• LASSO detail missing. The λ value, the number of folds, the random seed and the coding of the AI/human outcome are not stated. With ten texts, the selected set can change with the fold split. Rerunning the analysis may give different non-zero features.
• Table 3 header. The column is labelled "p-value" but holds LASSO coefficients tab. 3 p. 11.
• Readability is not tested. Table 1 has no p-values. The Results text says "we used statistical analyses" for the comparisons as a whole §3 p. 7, while §2.1 says readability is analysed descriptively §2.1 p. 6.
• Detection limits. The author warns that surface cues may shift with newer models and decoding strategies, and that detectors built on them risk false positives and misuse §5 p. 14. Do not use these features as a classifier without validation on new data.

Lineage

• Builds on: stylometry and computational linguistics traditions §1 p. 4. The paper extends earlier part-of-speech, lexical-density and cohesion work by adding phonology-adjacent measures and nonparametric tests §1 p. 4.
• Related work discussed: Alexander et al. [10], Herbold et al. [11], Cai et al. [12] and Liao et al. [13] on human versus AI writing §1 p. 2; DetectGPT [14] and Fast-DetectGPT [15] on likelihood-based detection §1 p. 3.
• Compared against: §4 p. 12–13 compares the findings with Alexander et al. and Liao et al.
• Superseded by: not stated.