Neural Grammatical Error Correction for Romanian

In plain words

Grammatical error correction (GEC) takes a sentence with mistakes and returns a corrected version. Most research has used English. For other languages there is often only a spellchecker with simple rules, and few example pairs of wrong and right sentences. This paper builds the first such set of pairs for Romanian: about ten thousand sentences, mostly phrases and sentences from Romanian TV and radio that a national audiovisual council had corrected, plus sentences with the kinds of mistakes people make when writing.

The authors first train a small sequence-to-sequence model, a Transformer, on these pairs alone. That is the baseline. A larger Transformer is first trained on about ten million Romanian Wikipedia sentences into which errors were added automatically, and then trained further on the real pairs. The error-adding step uses a spellchecker to replace words with its suggestions, plus a few random deletions, insertions and swaps. The larger model with this pretraining scores clearly better than the baseline. The authors also adapted the ERRANT scorer, which compares the edits a system makes with the edits in the reference, to Romanian.

TL;DR

• First Romanian GEC corpus: 10,119 sentence pairs in three groups, split 70/15/15 §3 tab. 1.
• Baseline: a small Transformer trained only on the gold pairs, F0.5 44.38 tab. 5.
• Best: a base-size Transformer pretrained on synthetic errors, then fine-tuned, F0.5 53.76 tab. 5.
• Scores are author-reported. The paper gives one number per row and no variance.
• Code is Apache-2.0 at TeodorMihai/RoGEC. It contains training and decoding, but not the evaluation scripts or the Romanian ERRANT adaptation.

Problem

Romanian GEC lacks a labelled corpus, and the available tools are mostly spellcheckers and simple rules §1. The paper's low-resource setting follows the BEA 2019 low-resource track. That track allowed only its official annotated pairs, but generated training data is permitted §1 §2. The goal is a corpus, a baseline, and a way to add data without human labels §1.

Method

The paper has no numbered equations and no numbered algorithms. Its formulas are given in words, and the one metric is described in prose §4.2.

1. Gold corpus. Sentences and phrases are split into three groups: NAC-phrases (no verb), NAC-sentences (spoken well-formed sentences) and W-sentences (written mistakes) §3. Each file is split 70/15/15 into train, dev and test §3 processings_cna.py. The paper does not state which file is which group. The file names suggest added.csv is W-sentences, but this is an inference.
2. ERRANT for Romanian. The German simplified ERRANT §3.1 was adapted to use UD POS tags only. The German-specific TRUNC tag and the CONTR category were removed. Spelling errors use the Hunspell Romanian dictionary. POS tags come from a Romanian spaCy model inside ReaderBench §3.1. The adapted scorer is not in the code repository [README.md#L29-L32].
3. Synthetic corpus. About 10 million sentences were filtered from Romanian Wikipedia §3.2 using the rules in Hyperparameters. Each sentence is then corrupted with the procedure below §3.2.
4. Models. Transformer-tiny is trained on the gold pairs only. Transformer-base is pretrained on the synthetic pairs, then continued on the gold pairs by one of two strategies: retraining (optimizer state reset) or fine-tuning (optimizer state kept). No layer is frozen §4.1.
5. Decoding. Beam search with 8 beams. An optional 5-gram Kneser-Ney KenLM re-ranks the final beams §4.1.
6. Evaluation. ERRANT extracts edits. Precision and recall are computed per edit, and F0.5 weights precision twice as much as recall §4.2.

Corruption of one sentence, as described in §3.2 and implemented in synthetic_generate/generate.py:

# Synthetic error injection for one clean sentence  [§3.2]
# Input:  tokens (list of str), a clean sentence
# Output: corrupted tokens (list of str)

def corrupt(tokens, spell_suggestions, sample_corpus_word):
    # Draw the error rate, clipped at zero  [§3.2]
    # [code: synthetic_generate/generate.py#L78-L79]
    error_rate = max(normal(mean=0.15, std=0.2), 0.0)
    # Scale by sentence length; code counts alphabetic tokens  [§3.2]
    # [code: synthetic_generate/generate.py#L84-L85]
    n_word_changes = round(error_rate * count_alpha_words(tokens))
    for _ in range(n_word_changes):
        # Pick a word. Paper: without replacement. Code re-draws each
        # time, so a word can be hit twice  [§3.2]
        # [code: synthetic_generate/generate.py#L87-L91]
        position = random_alpha_position(tokens)
        draw = uniform(0.0, 1.0)
        if draw < 0.7:
            # Replace with one of the top 20 spellchecker suggestions  [§3.2]
            # [code: synthetic_generate/generate.py#L96-L105]
            suggestions = spell_suggestions(tokens[position])
            tokens[position] = pick_suggestion(suggestions)
        elif draw < 0.8:
            # Delete the word  [§3.2]
            # [code: synthetic_generate/generate.py#L106-L108]
            del tokens[position]
        elif draw < 0.9:
            # Insert a corpus word. Paper: after the word; code: before it
            # [§3.2]  [code: synthetic_generate/generate.py#L109-L111]
            tokens.insert(position, sample_corpus_word())
        else:
            # Swap with the next token  [§3.2]
            # [code: synthetic_generate/generate.py#L112-L115]
            swap_with_next(tokens, position)
    # Character-level pass: 10% of the words, same operation mix  [§3.2]
    # [code: synthetic_generate/generate.py#L118-L153]
    apply_character_edits(tokens, fraction=0.1)
    return tokens

Training pairs are written as (corrupted, clean) with the clean sentence as the target. Decoding follows the procedure below, as in transformer.py §4.1 transformer.py.

# Beam search with optional LM re-ranking  [§4.1] [§5]
# Input:  source (word-piece ids), model, lm (5-gram KenLM), use_lm flag
# Output: one corrected string

def correct(source, model, lm, use_lm, normalise_lm):
    # Keep 8 hypotheses per step; code stops at 768 steps  [§4.1]
    # [code: transformer.py#L86, L92, L269-L297]
    beams = beam_search(model, source, beam_width=8, max_steps=768)
    candidates = []
    for beam in beams:
        # Drop start and end tokens  [code: transformer.py#L181-L187]
        text = detokenise(beam.ids)
        score = beam.log_prob
        if use_lm:
            # LM score of the candidate text  [§4.1] [code: transformer.py#L188]
            lm_score = lm.score(text)
            if normalise_lm:
                # Paper: normalised LM term at the same weight as the beam.
                # Code: fixed factor 10, and length is the total step count
                # [§4.1]  [code: transformer.py#L190-L191, L299-L301]
                score += 10 * lm_score / beam.length
            else:
                # Weight fixed to 1, not grid-searched  [§5]
                # [code: transformer.py#L93, L193]
                score += 1.0 * lm_score
        candidates.append((score, text))
    # Highest score wins  [code: transformer.py#L200]
    return sorted(candidates, key=first_item, reverse=True)[0][1]

Learning-rate schedule. The paper says only that a warm-up schedule is used, and gives no numbers §4.1. The values below are from the code:

# Warm-up then inverse-square-root decay  [§4.1: warm-up only]
# [code: transformer/transformer_scheduler.py#L3-L16]
def learning_rate(step, d_model, warmup_steps=4000):
    # step must be at least 1; the formula is undefined at step 0
    inverse_sqrt = step ** -0.5
    warmup_term = step * warmup_steps ** -1.5
    return d_model ** -0.5 * min(inverse_sqrt, warmup_term)

Hyperparameters

Values are the paper's unless marked "code". "Not stated" means the paper does not give the value.

Two training strategies, both with all layers trainable §4.1:

• Retraining: optimizer state and learning-rate schedule are reset before training on the gold pairs §4.1. Code: a new Adam is built, so its step counter restarts transformer.py. Weights are kept.
• Fine-tuning: optimizer state is kept §4.1. Code: the default, reset_opt=False transformer.py.

Results

Author-reported. Setup: RONACC, 10,119 pairs tab. 1; baseline is Transformer-tiny §5; metric is ERRANT-based F0.5 §4.2. The paper does not say which split tab. 5 and tab. 6 use. The repository's decoding script runs on the test files [run_experiments.sh#L7], so the test split is likely, but this is an inference.

Corpus statistics, from tab. 1:

Main results, from tab. 5:

Per group, from tab. 6. Decoding mode is not stated for this table:

What the paper says about these numbers:

• Fine-tuning beats retraining by about 8 F0.5 points, mainly through higher precision §5.
• LM re-ranking lowers scores for every model except the synthetic-only one. The authors cite the small LM corpus and a fixed weight §5.
• The gain is largest on W-sentences, which the authors link to the similarity between written mistakes and Wikipedia text. NAC-sentences gains least §5 tab. 6.
• Qualitative examples are in tab. 7. The model fixes spelling errors, the Romanian multiple-'i' plural and article issue, and some hyphen errors. It fails on non-idiomatic word choice, on diacritics, and on English words used in the text §5 §6.

Paper vs code

Code: TeodorMihai/RoGEC at commit 4f9a209b1981d3adfe50c2a4c557d3ecf6a3c0ea, Apache-2.0 [LICENSE]. The paper's footnote links this repository §1. The README cites the same three authors [README.md#L37-L39].

• Base width. The paper gives embedding size 512 tab. 4 and about 94M parameters §4.1. The code's default is d_model=768 transformer.py, and the README says the fine-tuned model is 768 wide [README.md#L23]. A hand count from the code's architecture gives about 94.5M for width 512 and about 156M for width 768. So the paper's numbers match 512, and the released model uses 768. Which width produced tab. 5 is not stated.
• Tiny config. Matches tab. 4 exactly transformer.py. A hand count gives about 1.6M parameters, which supports "approx. 1.5 mil" §4.1 over "approx. 15k" in the same section. The 15k figure is too small for the 2k vocabulary alone, since the embedding tables are about 0.5M.
• Dev set for pretraining. Paper: dev is 3% of the synthetic corpus §4.1. Code: dev comes from a separate file, dataset_file_dev, defaulting to the gold dev set transformer.py. The 3% split exists only in the branch with separate=False, and that branch defaults to train_dev_split=1.0, which leaves the dev set empty transformer.py dataset.py. The 3% split cannot be reproduced from the shipped defaults.
• Stopping rule. Paper: train until dev loss stops falling substantially §4.1. Code: no stopping rule. Training runs for epochs, default 500 transformer.py. A checkpoint is saved every two epochs, keeping five transformer.py. Dev loss is only logged transformer.py. Decoding restores the latest checkpoint, not the best on dev transformer.py.
• Retraining. Paper: retraining resets the optimizer and its learning rate §4.1. Code: consistent. It builds a new Adam with the schedule transformer.py. The weights are kept.
• LM normalisation. Paper: normalised LM term at the same weight as the beam probability §4.1. Code: adds 10 * weight * lm / length transformer.py. The factor 10 is not in the paper. length is len(out), the number of decoding steps, which is the same for all beams in a run transformer.py. So the division only rescales the LM term per sentence.
• Beam normalisation. Paper: "length-normalized beam search" §4.1. Code: the penalty ranks beams during search beam_search.py transformer.py. The final choice uses the raw beam log-probability when the LM is off transformer.py. So length normalisation can change which beams survive, but not the final pick unless the LM is on.
• LM score base. Code: kenlm.score is combined with natural-log beam scores without conversion transformer.py. KenLM's score returns log10 values by library convention. This was not checked by running the code.
• Synthetic corruption. Several steps differ from §3.2 (see Pitfalls for the full list): repeated word picks generate.py, word count over alphabetic tokens only generate.py, insertion before the word generate.py, and clipping at zero generate.py.
• Sentence filters. Paper: more than 8 words §3.2. Code: more than 8 characters filter_sent.py. The bracket rule checks parity, not balance filter_sent.py. Paper's "equal number" becomes "even count", so unbalanced pairs pass.
• Filter script. filter_sent.py never assigns args, so its __main__ block fails with a NameError filter_sent.py. The filter cannot be run as shipped.
• ERRANT and evaluation. The paper says all resources are in the repository §1. The Romanian ERRANT adaptation is not in the repository: the README only says to use errant -lang ro [README.md#L29-L32]. No F0.5 or scoring script is in the repository.
• Training text. Model input is raw text. ERRANT input is NLTK word-tokenised processings_cna.py processings_cna.py. The paper does not say which form the model sees.
• Split reproducibility. Splits use train_test_split with no seed processings_cna.py. Re-running gives different splits.
• BERT. The code has a BERT encoder path transformer.py. The paper says BERT was not used in the final experiments §6.
• Decoding script. run_experiments.sh calls transformer_bert.py in the top-level directory, but that file is under transformer/ [run_experiments.sh#L7] transformer_bert.py. The README uses transformer.py [README.md#L22]. The script also reads checkpoints from a gs:// bucket when use_bucket is set transformer.py.

Checked and consistent with the paper: the dropout rates, head counts and filter sizes in tab. 4 transformer.py. The loss ignores padding (id 0) transformer.py. The layers are post-LayerNorm encoder_layer.py encoder_layer.py. The paper says "default architecture", which is the post-LN design. Embeddings are not shared between encoder, decoder and output transformer.py, matching §4.1.

Pitfalls

• Width. The paper's table and parameter count say 512. The code default and README say 768. The 768 width gives about 66% more parameters (156M vs 94M). Pick one on purpose and say which.
• Tiny parameter count. The paper says both ~1.5M and ~15k. The table implies ~1.6M.
• Metric definition. F0.5 is defined only in words §4.2. The numbers in tab. 5 and tab. 6 match the standard F-beta formula with beta 0.5. I checked four rows by hand: tiny overall, fine-tuned NAC-phrases, fine-tuned NAC-sentences and fine-tuned W-sentences. Use ERRANT's scorer, since it is not in this repo.
• One run per row. No seeds, variance or confidence intervals are reported §5. The paper gives no evidence of how much re-runs vary, so small gaps between rows are hard to interpret.
• Split for results. tab. 5 and tab. 6 do not say which split they use.
• Stopping and checkpoint. Results depend on the checkpoint chosen. The code uses the last checkpoint, saved every two epochs, not the best dev one transformer.py.
• Silent truncation. Sentences longer than 512 word-pieces are cut, not dropped. The length check happens after padding and trimming, so it never fires dataset.py dataset.py dataset.py. A long target loses its end token. The paper does not say how long sentences are handled.
• File order. The training and dev files alternate lines: the even line is the clean target and the odd line is the corrupted source dataset.py dataset.py. Swapping the order trains the model in the wrong direction.
• Shared tokenizer. The word-piece vocabulary is built from the training file of the first run, then reloaded for later runs dataset.py. The paper says it is built from the synthetic corpus. The code does not check which corpus was used.
• Learning-rate restart. Retraining starts a new Adam, so the schedule restarts at step 0 transformer.py. This was not checked by running the code.
• Length normalisation. The GNMT-style penalty uses the ratio ((5 + length) / 6) to the power 0.6 beam_search.py transformer.py. The paper gives no formula. Its effect is limited when the LM is off (see Paper vs code).
• Generator needs a spellchecker. The abstract says the method "requires only a POS tagger". The generator uses Aspell for substitutions §3.2 generate.py generate.py, and a ReaderBench tokenizer generate.py. The POS tagger is used for ERRANT, not for generation.
• Repeated word picks. The paper says words are sampled without replacement. The code redraws from the current word list each time, so the same word can be corrupted twice generate.py.
• Corruption rate basis. The paper multiplies by sentence length §3.2 step 2. The code multiplies by the count of alphabetic tokens, so punctuation and numbers are excluded generate.py. The sampled rate is also clipped at zero, which the paper does not mention generate.py.
• Insertion position. The paper inserts a random word after the chosen word. The code inserts before it generate.py. Insertions draw from up to 200,000 observed lowercase tokens generate.py generate.py. The paper does not say where the word comes from.
• Character-level edits. Character edits use only lowercase letters, including the Romanian ones generate.py generate.py. Capital letters are never inserted.
• Confusion-set rule. The code drops the original word from the suggestion list when present, then takes the first 20 generate.py. The paper says only "top 20". The Aspell version is not stated.
• Filter level. The paper does not say at which level the ratio rules apply. The code computes the diacritics and other-character ratios per line, before sentence splitting filter_sent.py. Also, the abbreviation list has a missing comma that joins two entries, ' a.' 'pg.' filter_sent.py.
• Post-LN. The code uses post-LayerNorm encoder_layer.py. Pre-LN variants will not reproduce the code's behaviour.
• Group names. Mapping from file names to paper groups is an inference (see Method, step 1).
• Table 6 decoding. The decoding mode is not stated.
• Compute. The paper states only "a few hours" on a GTX 1050 for the baseline §5. The base model's hardware and training time are not stated, although TPUs are acknowledged.

Lineage

• Builds on: Grundkiewicz et al. (2019), a neural GEC system for the BEA 2019 low-resource track. This paper follows its synthetic-error method and its pretrain-then-fine-tune recipe §1 §2 §3.2.
• Builds on: Vaswani et al. (2017), the Transformer architecture used for both models §4.
• Builds on: Bryant et al. (2017), the ERRANT scorer, and Boyd et al. (2018), the German simplified version adapted here §3.1 §4.2.
• Compared against: the paper's own tiny baseline. It mentions Kaneko et al. (2020), which uses BERT features for GEC, and reports that its own BERT attempt performed poorly §2 §6.
• Builds on: Junczys-Dowmunt et al. (2018), for the error-rate balance and source-embedding dropout the authors suggest for future work §3.2 §6.