Paper · 2019-07-26 · cs.CL
Supervised and Unsupervised Neural Approaches to Text Readability
Scores text readability two ways on four English and Slovenian corpora. Unsupervised: RSRS ranks per-word language-model surprisal, weights it by rank and averages it. Supervised: BiLSTM, HAN and BERT classifiers. Author-reported: RSRS beats traditional formulas on Slovenian and OneStopEnglish, not WeeBit or Newsela; BERT is best on WeeBit, HAN on OneStopEnglish and Newsela by accuracy.
Language model surprisalText classificationLong short-term memory (LSTM)Transfer learningreadabilitylanguage-modelstext-classification
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Readability tools estimate how hard a text is to read, usually for a given age or learner level. The classic formulas count simple things such as sentence length and long words. They are cheap, but they miss meaning and context, and they can give different answers on different kinds of text.
This paper tries two ways to use neural networks. In the first, a language model reads the text one word at a time and gets a "surprise" score for each word: how unexpected the word was. The paper's score then gives more weight to the most surprising words in each sentence, because one hard word can make a sentence hard. This needs no labelled data, only text from the domain. In the second, a neural classifier is trained on texts labelled with their level and learns to predict the level of a new text.
The authors test both on four collections in English and Slovenian. The results depend on the collection: the new score wins on some, and the simple sentence-length baseline wins on others.
TL;DR
- Unsupervised, with no labelled data needed §1: per-word negative log-likelihood from a language model, sorted and weighted by the square root of rank, averaged per sentence and then per text (RSRS) §4.2.2 eq. 3.
- The language models are a TCN, a character-CNN + LSTM model, and BERT; English uses Wikipedia-based training corpora, Slovenian uses KRES-balanced §4.1 §4.3.1.
- Supervised: BiLSTM, hierarchical attention network (HAN) and fine-tuned BERT, all on the same four labelled corpora §5.1.
- Results are author-reported. Best supervised accuracy: BERT on WeeBit, HAN on OneStopEnglish and Newsela, BiLSTM on Slovenian SB; BERT has the best QWK on WeeBit and Slovenian SB tab. 5.
- Unsupervised best per corpus: GFI/FKGL on WeeBit (0.544), TCN RSRS-simple on OneStopEnglish (0.615), ASL on Newsela (0.906), TCN RSRS-balanced on Slovenian SB (0.789) tab. 3.
- No official code found. Several details (WNLL formula, LM context, QWK weights) are ambiguous; see Pitfalls.
Problem
- Classic formulas use only shallow counts. Their cross-genre and cross-language transfer is weak §2.1.1 §3.2.
- Cohesion and discourse features have not beaten shallow formulas in past work §2.1.2.
- Earlier language-model readability features used n-gram models trained per readability class. Neural language models were not used as readability predictors §2.1.5 §4.1.
- Supervised feature engineering needs many hand-made features and labelled data, which are scarce, especially for Slovenian §1 §2.2.
- Earlier neural readability studies mostly used one corpus, most often WeeBit, so their results do not compare across datasets §2.3.
Method
The paper has two tracks, both run on the same four corpora §3.1. The unsupervised track asks whether language-model statistics alone can estimate readability, and proposes RSRS. The supervised track compares three neural classifiers with the state of the art.
Unsupervised track
- Language models. TCN, a character-CNN + LSTM model (called RLM), and BERT are the three architectures §4.1. For English, TCN and RLM are trained on three Wikipedia-based corpora (Wiki-normal, Wiki-simple, Wiki-balanced). The pretrained uncased bert-base model is used as is §4.3.1 tab. 1. For Slovenian, TCN and RLM are trained only on KRES-balanced; the pretrained CroSloEngual BERT is used §4.3.1.
- Per-word probability. For each word, take the probability the language model gives to the word that actually occurs, from the softmax §4.2.2.
- Word negative log-likelihood. Compute WNLL with the binary cross-entropy form written in §4.2.2. See Pitfalls: it is not clear whether the sum runs over the vocabulary.
- Sentence split. Split each text into sentences with NLTK's default sentence tokenizer §4.2.2.
- Rank and weight. Sort the words of a sentence by WNLL, ascending. Weight the word at rank i by the square root of i. Words out of the vocabulary get the weight doubled §4.2.2 eq. 3.
- Average. The sentence score is the weighted sum divided by the sentence length S eq. 3. The text score is the mean of its sentence scores §4.2.2.
- Perplexity baseline. The language model's perplexity on each text is also used as a measure, computed from the negative log-likelihood eq. 1 eq. 2 §4.1.
# RSRS for one text [§4.2.2] [eq. 3]
# Input: text (str), lm (next-word model), vocab (set of known words)
# Output: float, higher = harder to read
def rsrs_text(text, lm, vocab):
# Split into sentences with NLTK's default tokenizer [§4.2.2]
sentences = nltk_sent_tokenize(text)
# Score each sentence, then average over the text [§4.2.2]
scores = [rsrs_sentence(s, lm, vocab) for s in sentences]
return mean(scores)
def rsrs_sentence(words, lm, vocab):
# Probability of each actual word in its context [§4.2.2]
p_true = lm.prob_of_actual_word(words) # (S,)
# WNLL, reading A: plain negative log-probability [§4.2.2]
wnll = -log(p_true) # (S,)
# Sort ascending by WNLL, keep the out-of-vocabulary flags [§4.2.2]
order = argsort(wnll)
sorted_wnll = wnll[order]
is_oov = [w not in vocab for w in words]
oov_sorted = [is_oov[k] for k in order]
total = 0.0
for i in range(1, len(words) + 1):
# Square-root-of-rank weight, doubled for OOV words [§4.2.2]
weight = sqrt(i) * (2 if oov_sorted[i - 1] else 1)
total += weight * sorted_wnll[i - 1]
# Divide by the sentence length S [eq. 3]
return total / len(words)Supervised track
- Three classifiers §5.1.
- BiLSTM: a bidirectional LSTM over the document. Max pooling and mean pooling of its output are concatenated and fed to a linear layer. Follows the classification design of Conneau et al. §5.1.
- HAN: word-level and sentence-level bidirectional LSTMs, each with attention, following Yang et al. §5.1.
- BERT: pretrained encoder (12 layers, width 768, 12 heads) with a linear head, fine-tuned on the whole model §5.1.
- Splits §5.1.
- Newsela and Slovenian SB: shuffle, then a stratified 80/10/10 train/validation/test split.
- WeeBit and OneStopEnglish: five-fold stratified cross-validation; within each fold, 80/10/10.
- Training.
- BiLSTM and HAN: up to 100 epochs; the epoch with the best validation score is kept and used on the test set §5.1.
- BERT: three epochs on the training set; the final model is tested. No validation-based selection is stated §5.1.
- Grid search over the validation sets for BiLSTM and HAN §5.1. The values chosen are in Hyperparameters.
- Metrics §5.1.
- Accuracy; weighted precision, recall and F1 (scikit-learn, average = "weighted").
- Quadratic weighted kappa (QWK) for ordinal labels eq. 4. The paper defines the weights as the absolute label distance |c_i − c_j|.
Baseline formulas, computed on each document, with W = words, S = sentences, Sy = syllables, C = characters §2.1.1 (unnumbered in the source):
\textrm{GFI} = 0.4\left(\frac{W}{S} + 100\,\frac{\textrm{long words}}{S}\right) §2.1.1
Long words are those longer than 7 characters. Higher GFI means harder.
\textrm{FRE} = 206.835 - 1.015\frac{W}{S} - 84.6\frac{Sy}{W} §2.1.1
Higher FRE means easier. The sign is reversed when reading correlations (see Pitfalls).
\textrm{FKGL} = 0.39\frac{W}{S} + 11.8\frac{Sy}{W} - 15.59 §2.1.1
\textrm{ARI} = 4.71\frac{C}{W} + 0.5\frac{W}{S} - 21.43 §2.1.1
\textrm{DCRF} = 0.1579\left(\frac{\textit{difficult words}}{W}\cdot 100\right) + 0.0496\frac{W}{S} §2.1.1
Difficult words are those not in a 3000-word list. When the list is not used, words longer than 7 characters count as difficult §2.1.1.
\textrm{SMOG} = 1.0430\sqrt{\textit{polysyllables}\cdot\frac{30}{S}} + 3.1291 §2.1.1
Polysyllables are words with three or more syllables §2.1.1.
ASL, average sentence length, is the simplest baseline §4.3.1.
Equations from the source:
\textrm{PPL} = 2^{-\frac{1}{N}\sum_{i=1}^{N}\log_2 m(w_i)} eq. 1
\textrm{NLL} = -\sum_{i=1}^{n}\log P(w_i \mid w_{1:i-1}) eq. 2
\textrm{RSRS} = \frac{\sum_{i=1}^{S}\sqrt{i}\cdot\textrm{WNLL}(i)}{S} eq. 3
\textrm{QWK} = 1 - \frac{\sum_{i}\sum_{j} w_{ij}\,x_{ij}}{\sum_{i}\sum_{j} w_{ij}\,m_{ij}} eq. 4
Pearson correlation ρ between each measure and the gold labels, computed per corpus §4.3.1.
Figures, described in words. fig. 1 is a worked RSRS example for one sentence. fig. 2 to fig. 5 are confusion matrices for BERT, HAN and BiLSTM on WeeBit, OneStopEnglish, Newsela and Slovenian SB.
Hyperparameters
Datasets §3.1 tab. 1:
| Corpus | Language | Units | Labels | Documents | Average length in source | Split used |
|---|---|---|---|---|---|---|
| WeeBit | English | articles | age 7-8, 8-9, 9-10, 10-14, 14-16 (600 each) | 3,000 | 193.26 tokens per doc tab. 1 | 5-fold stratified CV |
| OneStopEnglish | English | aligned texts | beginner, intermediate, advanced (189 each) | 567 | 677.23 tokens per doc tab. 1 | 5-fold stratified CV |
| Newsela | English | aligned simplifications | 2nd to 12th grade (imbalanced; 11 levels) | 9,565 | 760.30 tokens per doc tab. 1 | one 80/10/10 split |
| Slovenian SB | Slovenian | chunks of 25 sentences §3.1 | 9 primary and 4 secondary grades (13 levels) | 11,930 | 304.85 tokens per doc tab. 1 | one 80/10/10 split |
Language model training corpora §3.1 §4.3.1 tab. 1:
| Corpus | Content | Documents | Tokens |
|---|---|---|---|
| Wiki-normal | random English Wikipedia articles | 130,000 | 10,719,878 |
| Wiki-simple | random Simple English Wikipedia articles | 130,000 | 10,933,710 |
| Wiki-balanced | 65,000 Wikipedia + 65,000 Simple Wikipedia | 130,000 | 10,847,108 |
| KRES-balanced | Slovene magazines for children, teenagers and adults | about 2.4 million words | 2,402,263 |
Hyperparameters for the supervised models §5.1. The bold values were chosen by grid search on validation data.
| Name | Model | Value | Where | Notes |
|---|---|---|---|---|
| batch size | BiLSTM | 8, 16, 32 | §5.1 | |
| learning rate | BiLSTM | 5e-5, 1e-4, 2e-4, 4e-4, 8e-4 | §5.1 | |
| word embedding size | BiLSTM | 100, 200, 400 | §5.1 | |
| LSTM layer size | BiLSTM | 128, 256 | §5.1 | |
| number of LSTM layers | BiLSTM | 1, 2, 3, 4 | §5.1 | |
| dropout after each LSTM layer | BiLSTM | 0.2, 0.3, 0.4 | §5.1 | |
| max epochs | BiLSTM | 100 | §5.1 | best validation epoch kept |
| batch size | HAN | 8, 16, 32 | §5.1 | |
| learning rate | HAN | 5e-5, 1e-4, 2e-4, 4e-4, 8e-4 | §5.1 | |
| word embedding size | HAN | 100, 200, 400 | §5.1 | |
| sentence embedding size | HAN | 100, 200, 400 | §5.1 | |
| max epochs | HAN | 100 | §5.1 | best validation epoch kept |
| layers, hidden size of LSTMs, dropout | HAN | not stated | §5.1 | |
| learning rate | BERT | 2e-5 ("default") | §5.1 | |
| epochs | BERT | 3 | §5.1 | |
| max input length | BERT | 512 byte-pair tokens | §5.1 | longer texts are truncated; method not stated |
| batch size, optimizer, warm-up, weight decay | BERT | not stated | §5.1 | |
| pretrained model | BERT (English) | bert-base-uncased | §4.3.1 §5.1 | |
| pretrained model | BERT (Slovenian) | CroSloEngual BERT | §4.3.1 §5.1 | |
| training hyperparameters | TCN, RLM (language models) | not stated | §4.3.1 | sizes, learning rate, epochs not given |
| sentence splitter | RSRS | NLTK default | §4.2.2 | |
| rank weight | RSRS | square root of rank | §4.2.2 | |
| OOV weight multiplier | RSRS | 2 | §4.2.2 | |
| chunk length | Slovenian SB | 25 sentences | §3.1 | chosen to fit the 512-token limit |
| split | all supervised | 80/10/10, or 5-fold stratified CV | §5.1 | |
| label distance | QWK | |c_i − c_j| between ordinal labels | §5.1 | see Pitfalls |
The same configuration is used on every corpus; no per-corpus tuning of classifiers is stated §5.1. The text says the grid was run on validation sets, but does not say whether once or per corpus.
Results
Author-reported. Setup: four corpora §3.1. Unsupervised: Pearson ρ against gold labels, baselines are the traditional formulas and ASL §4.3.1 tab. 3. Supervised: accuracy, weighted F1 and QWK, on the test splits; baselines are published numbers from other papers §5.2 tab. 5.
Unsupervised, Pearson ρ, larger is better except FRE tab. 3:
| Measure | WeeBit | OneStopEnglish | Newsela | Slovenian SB |
|---|---|---|---|---|
| RLM perplexity-balanced | -0.082 | 0.405 | 0.512 | 0.303 |
| RLM perplexity-simple | -0.115 | 0.420 | 0.470 | / |
| RLM perplexity-normal | -0.127 | 0.283 | 0.341 | / |
| TCN perplexity-balanced | 0.034 | 0.476 | 0.537 | 0.173 |
| TCN perplexity-simple | 0.025 | 0.518 | 0.566 | / |
| TCN perplexity-normal | -0.015 | 0.303 | 0.250 | / |
| BERT perplexity | -0.123 | -0.162 | -0.673 | -0.563 |
| RLM RSRS-balanced | 0.497 | 0.551 | 0.890 | 0.732 |
| RLM RSRS-simple | 0.506 | 0.569 | 0.893 | / |
| RLM RSRS-normal | 0.490 | 0.536 | 0.886 | / |
| TCN RSRS-balanced | 0.393 | 0.601 | 0.894 | 0.789 |
| TCN RSRS-simple | 0.385 | 0.615 | 0.894 | / |
| TCN RSRS-normal | 0.348 | 0.582 | 0.886 | / |
| BERT RSRS | 0.279 | 0.384 | 0.674 | 0.126 |
| GFI | 0.544 | 0.550 | 0.849 | 0.730 |
| FRE | -0.433 | -0.485 | -0.775 | -0.614 |
| FKGL | 0.544 | 0.533 | 0.865 | 0.697 |
| ARI | 0.488 | 0.520 | 0.875 | 0.658 |
| DCRF | 0.420 | 0.496 | 0.735 | 0.686 |
| SMOG | 0.456 | 0.498 | 0.813 | 0.770 |
| ASL | 0.508 | 0.498 | 0.906 | 0.683 |
Bold marks the best per corpus in the source. "/" means not run tab. 3.
Averaged ranks tab. 4: the paper finds RSRS-simple and RSRS-balanced the most consistent across corpora (RLM RSRS-simple is 4th on all three English corpora). TCN RSRS-balanced is 1st on Slovenian SB and 2nd on OneStopEnglish and Newsela, but 11th on WeeBit. Traditional GFI is 1st on WeeBit but 10th on Newsela.
Supervised, test split tab. 5:
| Measure / Dataset | WeeBit | OneStopEnglish | Newsela | Slovenian SB |
|---|---|---|---|---|
| Filighera et al. (2019) accuracy | 0.8130 | - | - | - |
| Xia et al. (2016) accuracy | 0.8030 | - | - | - |
| SVM-BF (Deutsch et al., 2020) F1 | 0.8381 | - | 0.7627 | - |
| SVM-HF (Deutsch et al., 2020) F1 | - | - | 0.8014 | - |
| Vajjala et al. (2018) accuracy | - | 0.7813 | - | - |
| BERT accuracy | 0.8573 | 0.6738 | 0.7573 | 0.4545 |
| BERT precision | 0.8658 | 0.7395 | 0.7510 | 0.4736 |
| BERT recall | 0.8573 | 0.6738 | 0.7573 | 0.4545 |
| BERT F1 | 0.8581 | 0.6772 | 0.7514 | 0.4157 |
| BERT QWK | 0.9527 | 0.7077 | 0.9789 | 0.8855 |
| HAN accuracy | 0.7520 | 0.7872 | 0.8138 | 0.4887 |
| HAN precision | 0.7534 | 0.7977 | 0.8147 | 0.4866 |
| HAN recall | 0.7520 | 0.7872 | 0.8138 | 0.4887 |
| HAN F1 | 0.7520 | 0.7888 | 0.8101 | 0.4847 |
| HAN QWK | 0.8860 | 0.8245 | 0.9835 | 0.8070 |
| BiLSTM accuracy | 0.7743 | 0.6875 | 0.7111 | 0.5277 |
| BiLSTM precision | 0.7802 | 0.7177 | 0.6910 | 0.5239 |
| BiLSTM recall | 0.7743 | 0.6875 | 0.7111 | 0.5277 |
| BiLSTM F1 | 0.7750 | 0.6920 | 0.6985 | 0.5219 |
| BiLSTM QWK | 0.9060 | 0.7230 | 0.9628 | 0.7980 |
Baseline rows are the best number each cited paper reports. Filighera et al.'s WeeBit figure is on the development set; their test accuracy is 74.4% §5.2 (footnote). Deutsch et al. report SVMs over BERT features (SVM-BF) and over HAN features (SVM-HF) §5.2.
Key findings the paper reports §5.2 §6:
- BERT is clearly best on WeeBit (accuracy about 8 points above BiLSTM).
- HAN is best on OneStopEnglish and Newsela. The paper ties BERT's weak showing there to truncation of long documents.
- The gains over feature-engineered baselines are "not substantial", and the best architecture depends on the dataset §5.2 §6.
Paper vs code
No official code.
Searched: (1) the arXiv metadata and the LaTeX source, which contain no code link; (2) the authors' GitHub account matejMartinc, 25 public repositories. Its only readability repository, lasar_readability, belongs to a different paper (the SSAR 1.0 Slovenian dataset) and computes classical formulas, not these neural models; (3) the EMBEDDIA organisation, which has no readability repository; (4) GitHub repository search for "neural readability assessment language model", "readability RSRS", and "readability bert tcn". The only RSRS-named hits are third-party projects, such as VedantS5/document-readability-analyzer, which are not the authors'.
No code is recorded in this card, so none of the paper's hyperparameters can be checked against an implementation.
Pitfalls
- WNLL formula. The paper writes a binary cross-entropy for each word, with one-hot y_t and softmax y_p §4.2.2. It is not stated whether the (1 − y_t) log(1 − y_p) term is summed over all vocabulary words. If it is, the score includes an extra term from all non-target words. The card's pseudocode uses plain −log p, one reading. Implement one reading, state it, and check it against the correlations in tab. 3.
- LM context is not stated. It is not clear whether the language model sees earlier sentences of the text, or only the sentence itself, and whether the context is reset at each sentence boundary §4.2.2.
- BERT probability for RSRS is not stated. BERT is trained as a masked model §4.1, so a per-word probability needs a masking scheme (left-to-right or masked-token). Neither is given for the RSRS or perplexity values §4.1 tab. 3.
- Out-of-vocabulary words are not defined. The vocabulary of each model is not stated, and BERT's subword tokenizer makes "OOV" unclear. The doubling is only defined for OOV words §4.2.2.
- Sign convention. For FRE, a larger negative correlation means better; for all other measures a larger positive correlation means better §4.3.1. The RSRS score itself is higher for harder text.
- QWK weights. The paper calls the metric quadratic but defines the weight as |c_i − c_j| §5.1. The standard quadratic weight is (c_i − c_j)². The paper also does not define the expected matrix m_ij beyond "ground truth"; the usual construction uses the outer product of the marginal counts. Pick one and report it.
- Text and table disagree on Slovenian SB BERT. §5.2 says BERT gets "about 45.45%" F1 and "41.57%" accuracy. Table 5 gives accuracy 0.4545 and F1 0.4157. This card uses the table.
- Slovenian SB size. §3.1 says 3,639,665 words; Table 1 gives 3,636,843. The card reports the table.
- Units. Table 1 says "tokens" per document, while the text says "words". Document lengths in the text (193, 677, 760, 305) match Table 1, but the unit should be checked for the length-based analysis.
- Split leakage. Newsela and OneStopEnglish are parallel: the same story appears in several classes §3.2. The random shuffle is not stated to group by source article, so versions of one article may fall in both train and test. This could inflate the results on these two corpora.
- Truncation. BERT reads at most 512 byte-pair tokens §5.1. Average texts in OneStopEnglish and Newsela are longer than that tab. 1. The truncation side (head or tail) and any other handling are not stated.
- Grid search scope. The text says the grid was run on validation sets and the same configuration was used for all corpora §5.1. It is not clear whether the bold values came from one grid run or several.
- Language-model training is not specified. The sizes, learning rates, epochs and batch sizes of TCN and RLM are not stated §4.3.1. The unsupervised results cannot be reproduced from the paper alone.
- Variance and seeds. No standard deviations, run counts or random seeds are reported for any result tab. 3 tab. 5.
- Cross-dataset claims. The authors chose not to run cross-corpus supervised experiments §3.2. Cross-corpus and cross-language transfer is tested only for the unsupervised measure.
- Old baselines. The 93.3% accuracy of Vajjala et al. (2012) was questioned by Xia et al. (2016) §2.2. The card does not include that number.
- Cost. BERT fine-tuning and LM training costs (hardware, time) are not stated.
Lineage
- Builds on: character-level CNN + LSTM language model (Kim et al.) §4.1; TCN (Bai et al.) §4.1; BERT (Devlin et al.) §4.1; BiLSTM classifier (Conneau et al.) §5.1; HAN (Yang et al.) §5.1.
- Compared against: traditional formulas GFI, FRE, FKGL, ARI, DCRF, SMOG and ASL §2.1.1 tab. 3; supervised baselines of Vajjala et al. (2012, 2018), Xia et al. (2016), Filighera et al. (2019) and Deutsch et al. (2020) §2.2 §2.3 tab. 5.
- Earlier language-model readability features: n-gram models per class in Schwarm et al. (2005), Petersen et al. (2009) and Xia et al. (2016), which this paper replaces with neural models §2.1.5.
Source and license
Adapted from “Supervised and Unsupervised Neural Approaches to Text Readability” by Matej Martinc, Senja Pollak, Marko Robnik-Šikonja (arXiv:1907.11779), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.