KLUE: Korean Language Understanding Evaluation

In plain words

Most language models are judged by a set of test tasks. Those sets exist for English, and for a few other languages, but Korean had no single standard suite that covered many kinds of language understanding. KLUE is that suite. It bundles eight tasks: sorting news headlines by topic, rating how similar two sentences are, deciding whether one sentence follows from another, finding names of people and places, finding how two entities are related, working out the grammar of a sentence, answering questions from a passage, and tracking what a user wants across a task-oriented dialogue.

The authors wanted the data to be usable by anyone. They built each task from text whose license allows reuse, and they filtered out biased, hateful and personal content. Along with the data they published two Korean pretrained language models and a fine-tuning recipe for each task, so that a team can reproduce a baseline and compare against it.

The main finding is that a larger model does better on most tasks, and that models trained on Korean text usually beat multilingual ones of similar size. The authors also found that removing personal details from the pretraining text cost little accuracy.

TL;DR

KLUE is a Korean benchmark of eight NLU tasks with per-task metrics, released under CC BY-SA. Its baselines fine-tune pretrained encoders with a small task head. Test labels are withheld, so the reported test scores come from a leaderboard submission. KLUE-RoBERTa-large gives the best test scores on most tasks, but the paper's text overstates this (see Pitfalls).

Problem

Korean had no unified NLU benchmark, unlike English (GLUE, SuperGLUE) and several other languages §1. Existing Korean datasets were often translations of English benchmarks, which can miss Korean-specific features such as agglutinative morphology and miss social context §1.1, §2. The authors also state that they want the benchmark usable without licence restrictions and free of social bias, toxic content and personal data §1.1. A benchmark only helps if its labels are unambiguous, so the paper documents annotation guidelines and their revisions §1.1.

Method

The paper has two parts: a benchmark (data and metrics) and baselines (pretrained models and fine-tuning).

Part 1: benchmark construction

1. Choose tasks that cover different aspects of understanding and avoid redundancy §1.1.
2. Collect source corpora that are openly licensed or not copyrighted. The paper lists ten sources, among them Yonhap News headlines, Wikipedia, Wikinews, Wikitree, policy news, ParaKQC, Airbnb reviews, the NAVER Sentiment Movie Corpus, The Korea Economy Daily and Acrofan news §1.1. Corpus selection and preprocessing are in §2.1–§2.4.
3. Filter sources for noisy, toxic or biased text and for personal information, using rules and classifiers before annotation §1.1.
4. Write annotation guidelines, revise them over several rounds, and drop examples that annotators do not agree on §1.1.
5. Split each task's data as described per task in §3.1–§3.8. Splits follow publication date (TC), a distribution-balancing rule (STS), class balance (NLI), unseen entities in test (NER, RE), or counterfactual goals and unseen database rows (DST).
6. Score each task with the metric in tab. 1. The paper deliberately does not average scores across tasks §5.3.

Part 2: baselines

7. Start from a pretrained encoder. Each task adds a small head on the encoder's output, as described in §5.1.
8. Fine-tune all weights with AdamW under the grid in §5.2, and keep the configuration with the best development score §5.2.
9. Pretrain KLUE-BERT and KLUE-RoBERTa on about 62.65 GB of Korean text, using morpheme-aware subword tokenization §4.1.

Task overview tab. 1; split details per task in §3.1–§3.8

Tab. 1 gives NER as "6, 12" classes without saying what the 12 are; the code uses 12 BIO tags plus O (see Paper vs code). The MRC row conflicts with §3.7 (see Pitfalls).

Task heads and losses §5.1

• Single-sentence classification (TC, RE): the final hidden state of the first token is mapped linearly to the label count K, trained with cross-entropy. RE adds four marker tokens (<subj>, </subj>, <obj>, </obj>) and expands the embedding matrix.
• Sentence-pair regression (STS): the same first-token state is mapped to one real number, trained with mean squared error.
• Sentence-pair classification (NLI): three logits, cross-entropy.
• Slot-value prediction (DST): a TRADE-style encoder-decoder in which the encoder is the pretrained model instead of a GRU. The first token's state starts the decoder. A slot-gate classifier has two extra labels, yes and no. The generator and gate losses are summed.
• Sequence tagging (NER): each token's final state is mapped to 12 values, one per named-entity category, with cross-entropy summed over tokens. Subword labels must be aligned to characters.
• Span prediction (MRC): each passage token gets two logits (start and end), cross-entropy. For unanswerable questions the first token is the target for both start and end.
• Dependency parsing (DP): each word is represented by the concatenation of its first and last subword states, optionally with a part-of-speech embedding. A biaffine attention predicts the head and a bilinear classifier predicts the arc type. The paper follows a cited parser for this design §5.1.

# Fine-tune one pretrained encoder on one KLUE task  [§5.1] [§5.2]
# Input:  train_set, dev_set, one configuration from the grid
# Output: checkpoint with the best dev score

def fine_tune_task(train_set, dev_set, config):
    # Pretrained encoder plus a new task head, both trained  [§5.1]
    model = encoder + task_head(config.task)
    # AdamW, learning rate and weight decay from the grid  [§5.2]
    optimizer = adamw(model, config.learning_rate, config.weight_decay)
    # Linear schedule; warmup is a fraction of total steps  [§5.2]
    scheduler = linear_warmup(optimizer, config.warmup_ratio)
    for epoch in range(config.epochs):
        for batch in batches(train_set, config.batch_size):
            # Task loss: cross-entropy, MSE or span loss  [§5.1]
            loss = task_loss(model, batch)
            # Clip gradient norm to 1.0 in the released code  [code: run_klue.py#L56]
            clip_gradients(model, max_norm=1.0)
            optimizer.step()
            scheduler.step()
            optimizer.zero_grad()
    # Select on dev, never on test  [§5.2]
    return best_checkpoint_by_dev_score(dev_set)

Pretraining (summary)

• Two architectures: BERT (static whole-word masking, next-sentence prediction) and RoBERTa (dynamic whole-word masking) §4.1.
• Tokenization: text is split into morphemes with Mecab-ko, then byte-pair encoding (BPE) with a 32k vocabulary is learned on the morphemes. At inference only the BPE model is used, so no morphological analyser is needed §4.1.
• Personal data: 16 types detected with regular expressions, replaced with generated values, pseudonymizing 1.2% of the corpus §4.1.
• Bias and hate speech were deliberately not filtered out of the pretraining corpus, with reasons given §4.1.

Hyperparameters

Pretraining §4.1, pretraining-details table

Other pretraining settings: sequences up to 512 tokens. Other settings "same as the original configurations" of BERT and RoBERTa. Warmup, Adam betas and epsilon for pretraining are not stated in the paper §4.1.

Pretraining corpus §4.1, pretraining-corpus table

For CC-100-Kor and NEWSCRAWL, sentences under 200 characters are dropped. Sentences that appear in the benchmark tasks are removed using BM25 similarity §4.1.

Fine-tuning search space, as stated in the paper §5.2

Optimizer: AdamW §5.2. Selection: best dev score §5.2. Number of random seeds: not stated.

Values the released code uses instead of the search run_all.sh. Each row is one task. Rows without "large" cover the small and base models in the script's loop; rows marked "large" apply to klue/roberta-large. Settings not listed use the script defaults (epochs 4, weight decay 0, no warmup, learning rate 5e-5).

Other fixed values in the script: weight decay 0 for TC, STS and NLI; 0.01 for DST; gradient clipping 1.0 for STS, NLI and DST; early-stopping patience 100000 for TC, MRC and DST (effectively off), 10000 for RE and DP, and the default of 5 for STS, NLI and NER run_all.sh; code: run_klue.py.

Code-only defaults (not in the paper): AdamW epsilon 1e-8 lightning_base.py; linear learning-rate schedule lightning_base.py; random seed 42 run_klue.py; gradient clip 1.0 run_klue.py.

Results

Author-reported. Setup: eight tasks, each scored on the KLUE test set through the authors' leaderboard §5.3, app. A. Models are BASE size unless marked LARGE. Fine-tuning hardware is not stated. Each row is one pretrained model. Bold values in the source table mark the best per column across all models; underlined values mark the best among BASE models. In this card the best value per column is marked in the text, not in the table.

Test set §5.3

Column keys: F1 is macro F1 except where noted. STS R^P is Pearson correlation. NER F1^E is entity-level macro F1; F1^C is character-level macro F1. RE F1^mic is micro F1 without the no-relation class. DP UAS and LAS are the attachment scores. MRC EM is exact match; ROUGE is ROUGE-W. DST JGA is joint goal accuracy; F1^S is slot-value micro F1 §3.1–§3.8, §5.3.

Headline reading (author-reported): KLUE-RoBERTa-large has the highest test score in STS (both), NLI, RE (both), DP UAS, MRC (both) and DST (both). XLM-R-large is highest on TC and DP LAS, and KoELECTRA-base on NER entity F1 §5.3. Among BASE models, the paper names KLUE-BERT best on TC and DST, KLUE-RoBERTa-base on RE and MRC, and KoELECTRA-base on STS and NLI §5.3. See Pitfalls for the paper's own overstatement of the large-model result.

Development set app. A

Same column layout as the test table above. Dev scores are for reference; test scores need submission.

Pretraining ablations §5.4 (KLUE-RoBERTa BASE trained on the MODU corpus only; the evaluation split is not stated in §5.4)

Tokenizer statistics on the Wikipedia corpus with 32k vocabularies §5.4: BPE has fertility 2.073, 57.8% continued words and UNK ratio 0.011; morpheme-based subword has fertility 2.468, 76.5% continued words and UNK ratio 0.009.

Paper vs code

Code: the KLUE-benchmark organization's KLUE-baseline repository at commit 8a03c9447e4c225e806877a84242aea11258c790 (Apache-2.0). The paper links the same organization's repository in its ethics section §6, which is why it is treated as official. The paper itself doesn't name KLUE-baseline.

Checked against the code: the paper's search grid, the per-task heads and the DST loss all match the code, except for the items below.

1. Hyperparameters are fixed per model in the script, not searched. The paper says the best configuration is chosen from a grid §5.2. run_all.sh hard-codes one setting per task and model size run_all.sh. The script can therefore not reproduce a search, and the grid values in the paper do not all appear there.
2. Epochs outside the stated grid. DP-large trains for 15 epochs run_all.sh; the paper's grid is {3, 4, 5, 10} §5.2.
3. Sequence length differs from the paper's stated values. The paper says 128 for all tasks except MRC and DST §5.2. The script uses 256 for RE and DP run_all.sh and 510 for NER run_all.sh. MRC and DST use 510 for the RoBERTa models, but 512 for the BERT-base MRC run run_all.sh.
4. Dependency parser has LSTM layers not described in §5.1. The code puts a bidirectional LSTM encoder with dropout 0.33 on top of the pretrained encoder dependency_parsing.py and an ELU projection before the biaffine and bilinear heads dependency_parsing.py. The paper's §5.1 describes only the heads and cites a parser for the design, so the LSTMs are implied at most.
5. DP metric names do not match the standard attachment score. The paper defines UAS and LAS as the fraction of correctly attached heads and labels §3.6. The code computes macro and micro F1 over the head labels and the arc-type labels functional.py. The script reports uas_macro_f1 and las_macro_f1 run_all.sh. These are not the same numbers as the paper's UAS and LAS unless the paper also used F1; the paper does not say which.
6. NER output size. The paper says the tagging head outputs 12 values §5.1. The label set in code has 12 BIO tags plus O, that is 13 labels klue_ner.py.
7. NER character F1 excludes O. The code's character macro F1 is computed over the entity labels only functional.py. The paper says only "character-level macro-F1" §3.4, tab. 1.
8. STS F1 threshold. The paper says the similarity scores are binarized "as in paraphrase detection" but does not give the threshold §3.2. The code uses 3 functional.py.
9. Dependency parsing training versus validation. During training the code uses gold head indices; at validation time it uses argmax predictions dependency_parsing.py. The paper does not state which is used at test time.
10. MRC scoring against several gold answers. The code takes the maximum EM and ROUGE-W over all gold answers for each question utils.py. The paper does not say how multiple references are handled.
11. Early stopping. Default patience is 5 epochs run_klue.py but the script sets 100000 for most tasks, which effectively disables it run_all.sh. The checkpoint saved is the one with the best validation metric run_klue.py. The paper does not describe early stopping.
12. Data version. The script pins KLUE data v1.1 run_all.sh. The paper does not name a data version.
13. Random seed and weight decay on biases and LayerNorm. The code uses seed 42 run_klue.py and excludes bias and LayerNorm weights from weight decay lightning_base.py. Neither appears in the paper.
14. Pretrained checkpoints are not in this repository. The script loads the models by Hugging Face names such as klue/bert-base run_all.sh. The morpheme-based tokenizer (§4.1) is not implemented in the repository. The repository does not include the pretraining code.
15. Test scores cannot be reproduced locally. The README states that test sets are not public and that scores come from submitting a model to the authors' system README.md.

No discrepancies were checked for the pretraining code, because none is in the repository.

Pitfalls

• Table 1 disagrees with §3.7 for MRC. The overview table lists about 12k train, 8k dev and 9k test examples tab. 1. The detailed split table in §3.7 gives 17,554 train, 5,841 dev and 5,918 test, totalling 29,313 §3.7. Use the §3.7 figures.
• Table 1 and §3.1 disagree on the TC size. The text says 70,000 headlines were kept §3.1. The split counts in the §3.1 table sum to 63,892 (45,678 train, 9,107 dev, 9,107 test). The paper does not explain the difference.
• The paper overstates its own results. The text says KLUE-RoBERTa-large beats all other models on every task except NER §5.3. In the test table, XLM-R-large scores higher on TC (86.06 vs 85.69) and DP LAS (88.70 vs 88.36) §5.3. The claim is true for most tasks, not all.
• Test scores need submission. Test labels are not released. Submissions are limited to one per day to slow saturation of the benchmark app. A. Any comparison using test scores depends on the leaderboard state at the time.
• Dev scores are not test scores. The paper picks each model's configuration using dev §5.2. Dev numbers are in app. A for reference only.
• Hyperparameter search is not reproducible from the paper alone. The grid §5.2 is given as sets, but the number of random seeds, warmup and weight decay for pretraining, and the search budget are not stated. The script uses fixed values, not the grid (see Paper vs code).
• Pretraining learning rate. The paper says it lowered the learning rate because its batch (2,048) is smaller than RoBERTa's 8k, then fixed it at 1e-4 for both models §4.1. The warmup schedule and optimizer betas are not stated.
• NER label count. §5.1 says the output is 12-dimensional, while the BIO scheme over six entity types plus O gives 13 labels (see Paper vs code).
• Metric definitions are partly loose. DP "UAS/LAS" in the paper is not the same computation as the code's macro F1 variants (Paper vs code 5). NER character F1 excludes O in code but the paper does not say so (Paper vs code 7).
• Morpheme analysis runs only when building the vocabulary. Mecab-ko is used for the vocabulary and corpus; at inference the BPE model alone is used §4.1. The paper's ablation shows BPE trails badly on NER entity F1 (68.75 vs 84.84) and DST JGA (21.38 vs 47.14) §5.4. The ablation does not say whether BPE with a morpheme analyser at inference would close the gap.
• Pseudonymization is cheap but not free. The ablation shows small drops; the largest are NLI accuracy (81.60 vs 78.50) and RE micro F1 (65.25 vs 62.79) §5.4.
• Licensing of the data. The benchmark data is released under CC BY-SA §6. Redistributed copies must keep that license. The paper does not say whether models fine-tuned on KLUE count as derivatives; check before commercial use.
• Bias is kept in pretraining. The authors did not filter bias or hate speech from the pretraining corpus §4.1. Pseudonymization covers only personal identifiers.
• Long MRC and DST inputs. MRC and DST use a 512-token limit in the paper §5.2; several RoBERTa runs in the script use 510 run_all.sh. Two-token differences matter if you compare exact figures.

Lineage

• Builds on: BERT and RoBERTa architectures and their pretraining objectives, used as KLUE-BERT and KLUE-RoBERTa §4.1. TRADE for DST §5.1. A cited biaffine-style dependency parser for DP §5.1. A cited entity-marker method for RE §5.1.
• Compared against: mBERT, XLM-R (BASE and LARGE), KR-BERT and KoELECTRA §4.2.
• Superseded by: not stated in this wiki.