wikillm

Paper · 2026-11-03 · cs.CL

HERIT: Enabling Global Access to Hanja-to-English Historical Translation by Mitigating Temporal Bias

HERIT fine-tunes Qwen3 32B to translate Hanja historical archives into English. Gemini-2.5-Flash writes about 315K pseudo-labels using retrieved few-shot examples, a glossary and FusioN draft fusion, and a two-stage fine-tune trains on them before the human-translated data. Author-reported: BLEU 38.43 against 36.93 for the human-data-only model, and an expert score of 3.93 against 3.22.

Neural machine translationRetrieval-augmented generationDomain adaptationFew-shot example selectiontranslationhistorical-documentshanjapseudo-labelingretrieval-augmented-generationtemporal-bias

arXivPDFCodeMarkdown11 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Korea kept long official records written in Hanja. Only a small part of these records has a human English translation, and most of that part covers one king's reign. A model trained on it does well on that reign and worse on the others.

The authors use a large language model to make extra practice translations from the Hanja text plus a human Korean translation. Each draft is written with help from similar human-translated examples, a list of names and terms, and a second model step that merges several drafts into one. The result is a large pile of imperfect English labels. A model is first trained on those labels and then on the scarce human translations, and the authors say this makes it work across reigns. They then used the final model to translate the parts of the archives that have no English yet.

TL;DR

  • Hanja-to-English translation model (Qwen3 32B, fine-tuned) for Joseon-era archives. §4.3
  • Pseudo-labels for about 315K documents come from Gemini-2.5-Flash with retrieval-augmented few-shot prompts, a glossary, Korean input and FusioN draft fusion. §4.2 app. C.1 app. C.2
  • Two-stage training: pseudo-labels first, then human-translated data. §4.3 app. B
  • Author-reported gains on a 1,000-document human-translated test set and on an expert evaluation. Results are single-run, with no variance reported. §5.2 §5.3
  • As of this compilation the repository holds only a README, and no model or training code is public. See Paper vs code.

Problem

Expert English translation of the Annals of the Joseon Dynasty (AJD) is slow and costly; the authors cite a 22-year, about 20 million USD project §1. Human English pairs are scarce and concentrated in King Sejong's reign: about 17.7K Hanja-Korean-English triples exist, and the training split holds 15.7K of them §3. Off-the-shelf Chinese-English systems fail for three reasons the paper names: Chinese models use Simplified characters where Korean Hanja uses Traditional forms; Hanja words often differ in meaning from Chinese ones; and proper nouns are transliterated with Chinese readings rather than Korean ones §1. The paper's stated goal is a method that generalises across reigns rather than fitting the temporal bias of the human data §1 §6.1.

Method

The pipeline has three parts: choose a pseudo-labelling LLM, build pseudo-labels, and train and run the translator.

Backbone choice §4.1 tab. 2. Base-prompt translation on the test set was compared across models. Gemini-2.5-Flash was chosen for cost-effectiveness; Gemini-2.5-Pro scores higher but costs more §4.1. Turning on reasoning raised Gemini-2.5-Flash and lowered Sonnet-4.5 in the base-prompt test tab. 2.

Inputs to the pseudo-labeller §4.2 fig. 1. Three additions go into each prompt, on top of the Hanja text:

  • the human Korean translation of the same document, which the authors use to compensate for weak LLM Hanja reading §4.2 app. E;
  • glossary terms appended when they match the source exactly; the glossary has 25K entries from NIKH lexicons §3 §4.2;
  • few-shot demonstrations as (Hanja, Korean, English) triples, retrieved from the human-translated training split D^train by an embedding retriever §4.2.

Proposer and Fusor §4.2 fig. 1. The Proposer writes several English drafts. The Fusor reads the source and all drafts and writes one final translation. This follows the FusioN approach of Khairi et al. §2 §4.2. The selected settings are 64 Proposer shots, 12 Fusor shots and five drafts §4.2 app. C.1. Prompts are described in app. E fig. 8 fig. 9 fig. 10.

Pseudo-label set D^aug §3 §4.2. Documents over 800 characters are dropped §3. The paper gives about 315K resulting pairs §3 and 317.4K pairs processed §4.2, and about $3,000 of API cost §4.2. King Sejong's reign is excluded because human data already covers it §4.2.

Training §4.3 app. B. Qwen3 32B is fine-tuned in two stages, one epoch each. Stage 1 teaches the model the meaning and sound patterns of Hanja, names included. Stage 2 uses D^train to match human style §4.3. Training took 15.1 h and 1.2 h on eight H200 GPUs §4.3. The paper says mixing both sets in one run underperforms the two-stage schedule, and it attributes this to the size imbalance §5.2.

Inference §4.3 §5.2. The final model translates Hanja to English directly, with no retrieval and no Korean input. The paper says this avoids the context growth that retrieval adds, which it puts at over 8K tokens and 9x training cost for the retrieval-based alternative §5.2. Documents longer than the limit are split into chunks § Limitations.

Test-set decontamination §3 app. A. For each test item, the authors take the 10 closest D^train items by BM25 and compute normalised character-level Levenshtein similarity. Items above 0.9 are removed. The threshold is justified with two examples app. A tab. 7. The paper reports removing 83 test items, three held-out items and one item from the human-evaluation subset §3.

Pseudocode
# Pseudo-label construction (retrieval-augmented FusioN)  [§4.2] [fig. 1]
# Input:  docs: list of (hanja, korean); d_train: list of (hanja, korean, en)
#         glossary: list of (hanja, english)
# Output: d_aug: list of (hanja, pseudo_english)

def build_pseudo_labels(docs, d_train, glossary):
    # 1. Index D_train with the embedding retriever  [§4.2]
    index = embed_index(d_train)
    d_aug = []
    for hanja, korean in docs:
        # 2. Skip documents over 800 characters  [§3] [§ Limitations]
        if len(hanja) > 800:
            continue
        # 3. Add glossary terms that match the source exactly  [§4.2]
        terms = glossary_matches(hanja, glossary)
        # 4. Proposer shots: top 64, query = Korean text  [§4.2] [app. C.1]
        shots = top_k(index, query=korean, k=64)
        # 5. Five drafts from the Proposer, temperature 0.3  [§4.2]
        drafts = [
            proposer(hanja, korean, terms, shots, temperature=0.3)
            for _ in range(5)
        ]
        # 6. Fusor shots: top 12, query = Hanja plus drafts  [§4.2] [§5.1]
        fusion_shots = top_k(index, query=hanja_plus(hanja, drafts), k=12)
        # 7. Fuse all drafts into one English output  [§4.2] [app. C.2]
        english = fusor(hanja, korean, terms, fusion_shots, drafts)
        d_aug.append((hanja, english))
    return d_aug
Pseudocode
# Two-stage fine-tuning of Qwen3 32B  [§4.3] [app. B]
# Input:  d_aug (hanja, english); d_train (hanja, english)
# Output: translation model

def train_two_stage(base, d_aug, d_train):
    # 1. Stage 1 on pseudo-labels: 1 epoch, lr 1e-5, batch 64  [app. B]
    model = finetune(base, d_aug, lr=1e-5, batch_size=64, epochs=1)
    # 2. Stage 2 on human translations: 1 epoch, lr 5e-6, batch 16  [app. B]
    model = finetune(model, d_train, lr=5e-6, batch_size=16, epochs=1)
    # 3. PagedAdamW 8-bit with FSDP on eight GPUs  [app. B]
    return model
Pseudocode
# Full-archive translation  [§4.3] [§5.2] [§ Limitations]
# Input:  untranslated Hanja documents; limit = 800 Hanja characters
# Output: English translation per document

def translate_archive(model, docs, limit):
    outputs = []
    for hanja in docs:
        # 1. Split documents over the limit into chunks; size not stated
        for chunk in split_if_longer_than(hanja, limit):
            # 2. Hanja only: no retrieval, no Korean input  [§5.2]
            outputs.append(model(chunk))
    return outputs

Hyperparameters

NameValueWhereNotes
Pseudo-label LLM (Proposer and Fusor)Gemini-2.5-Flash§4.2Chosen for cost §4.1
Proposer few-shot count K64§4.2 app. C.1Retrieved from D^train
Fusor few-shot count K12§4.2 app. C.1Retrieved from D^train
Drafts per document5§4.2 app. C.1Fused into one output
Sampling temperature0.3§4.2 app. C.1Text ties it to the Proposer drafts; Fusor temperature not stated
RetrieverGemini-Embedding§4.2Version not stated in the paper
Proposer retrieval queryKorean text of the document§4.2
Fusor retrieval queryHanja plus English drafts§4.2 §5.1Concatenation per §5.1
Glossary25K entries; appended on exact match§3 §4.2
Document length cap800 Hanja characters; 2,048 tokens§ LimitationsHardware limit; about 6.7% of training data filtered
D^aug sizeabout 315K pairs§3 §4.2See Pitfalls: counts differ
DecontaminationBM25 top 10; normalised Levenshtein; cut-off 0.9§3 app. AStrict or non-strict cut-off not stated
Stage 1 learning rate, 32B1e-5app. B
Stage 2 learning rate, 32B5e-6app. B
Stage 1 learning rate, 8B2e-5app. B
Stage 2 learning rate, 8B1e-5app. B
Batch size64 (stage 1); 16 (stage 2)app. B
Epochs1 per stageapp. B
OptimiserPagedAdamW 8-bitapp. B
Distributed trainingFSDP on 8 NVIDIA H200 GPUsapp. B
Learning-rate schedule and warm-upnot stated
Weight decaynot stated
Full or parameter-efficient fine-tuningnot stated
Training sequence lengthnot stated beyond the 2,048-token limit§ Limitations
Decoding settings for the final modelnot stated
Chunk size for long inference inputsnot stated§ Limitations
Evaluation split sizesD^valid 1,000; D^test 1,000; D^test_NT 2,080; human subset 200§3Before decontamination (83, 3 and 1 items removed)

Results

Author-reported. Setup: Hanja-to-English, test set D^test of 1,000 human-translated documents from the AJD §3. Metrics are BLEU, METEOR and chrF++ as stated in the paper §4.1. No seeds, variance or significance tests are reported.

SettingMetricBaselineThis paperSource
Qwen3 32B, two-stage D^aug to D^train, vs D^train onlyBLEU36.93038.432tab. 4
Same pairchrF++61.58362.525tab. 4
Same pairMETEOR0.6130.626tab. 4
Qwen3 8B, two-stage (no 8B baseline reported)BLEUnot reported37.849tab. 4
Pivot-MT (Hanja to Korean to English), same training settingsBLEU35.53938.432tab. 4
H2KE, BLEU from its original paperBLEU24.6238.432tab. 4
Pseudo-label pipeline, Gemini-2.5-Flash, with auxiliary inputsBLEU15.720 (base prompt)38.462tab. 3
LLM judge (Sonnet-4.6, 1 to 10), D^test_NT, Qwen3 32Bscore6.883 (D^train only)8.009tab. 5
LLM judge, D^test_NT, Qwen3 8Bscore6.312 (D^train only)7.887tab. 5
Human experts (1 to 5), 200 items, vs Qwen3 32B on D^trainmean score3.223.93§5.3 fig. 3
Human experts, pairwise choiceselection ratebelow 33.4% (implied by "more than twice")66.84%§5.3
Proper-noun accuracy (Sonnet-4.6 judge)accuracy0.8180.878tab. 11

Costs and time: about $3,000 for the pseudo-labelling step §4.2; 15.1 h plus 1.2 h of training on 8 H200 GPUs §4.3; 94.6 h to translate 2.09 million documents on 8 H200 GPUs §4.3. The authors state the pseudo-labels are not a Hanja-only translation, because they use Korean text, glossary entries and retrieved examples §5.1.

Ablations in tab. 3 show each addition helps the pseudo-labeller: glossary, Korean input, retrieval (dynamic shots beat static), FusioN over a single pass, and a 64-shot proposer. The best setting is a 64-shot Proposer with Fusor queries built from Hanja plus English drafts tab. 3 §5.1. Fusion beats best-of-N selection (38.462 against 38.337 BLEU) tab. 8. Diverse proposers gave lower BLEU at about four times the cost tab. 9.

Per-reign expert differences are shown only in fig. 4; the text says HERIT matches the baseline on King Sejong's reign and leads elsewhere §6.1. Scaling with pseudo-label volume fig. 2 is described as improving, with no numbers in the text §5.2.

Paper vs code

Searched and found:

  • The paper links https://github.com/dandamdandam/HERIT §1. At commit 03543cd6074a1b890a9fb3e757b0e02efa00f2a4 it contains only README.md and an image. No LICENSE file, no Python code, no configs, no prompts, no checkpoints and no evaluation scripts.
  • The README's highlights say the repository ships the full pipeline, from data collection through evaluation README.md. The files are not there.
  • The README points to https://github.com/dandamdandam/joseon-analysis-v2 for "Getting Started" README.md. That repository was not found when cloned.
  • The same account owns a repository herit_eval, which is not linked from the paper. It is a Streamlit app for the human evaluation (commit a5ee49460d3c191f30d05e7d17711b7daadc64fe, no LICENSE file). I cite it only as an inference that it is the authors' tool: its paths are relative to that repository, e.g. [code: herit_eval/app.py#L263-L264]. Its code is not licensed, so nothing from it is quoted.

Differences the code and README show:

  • Released artefacts. The paper says the dataset and model are "made publicly available" §4.3; the introduction calls them "publicly available" §1. The README says the checkpoints are "To be released" README.md and the dataset section marks it as still being prepared, with an update due in early October 2026 README.md.
  • Scoring defaults. The evaluation app starts both scores at 3 and offers a 1 to 5 scale app.py app.py. An entry with both scores at 3, no comments and no preference is not stored app.py. The paper does not mention this.
  • Tie rule. Matches the paper: when the two scores are equal, the app requires a preference between translation 1 and 2 app.py. The preference is saved only when the scores tie; otherwise it is the placeholder "N/A" and is not saved app.py app.py.
  • Not checkable from code. Retrieval, pseudo-labelling prompts, training hyperparameters, the decontamination step and the LLM judge prompt have no code in any repository found. Everything in Method and Hyperparameters rests on the paper alone.

Pitfalls

  • Pseudo-labels are not Hanja-only translations. Their quality uses Korean text, glossary entries and retrieved human examples §5.1. Do not read the 38.46 BLEU in tab. 3 as the performance of the final model; that is the 38.43 in tab. 4.
  • Retrieval only from the human data. Shots come from D^train §4.2. Any pseudo-labelling pipeline built this way needs human-translated Korean for the source, which the final model does not need at inference §4.3.
  • Temperature. The text ties the 0.3 setting to the Proposer's five drafts that the Fusor merges §4.2. The Fusor's temperature is not stated. Implement 0.3 for the Proposer and check the Fusor separately.
  • Counts do not reconcile. The paper gives about 315K pseudo-labels after the 800-character filter §3, 317.4K pairs processed §4.2, and about 6.7% of training data filtered § Limitations. A 6.7% cut from 317.4K gives roughly 296K, not 315K. No explanation is given.
  • Decontamination wording. The text says candidates "exceeding" 0.9 are "discarded" §3, but the reported counts (83, 3 and 1 removed) refer to test items. Treat the cut-off as applied to test items. Whether the comparison is strict or not is not stated.
  • Human-eval count. One item was removed from the 200-item subset §3. The paper does not say whether 200 or 199 items were scored.
  • The 66.84% pairwise figure does not state its denominator. The text says pairwise judgments were used "in cases where scores were tied" §5.3, which suggests a tied subset, but this is not confirmed.
  • Costs are not comparable across tables. Gemini-2.5-Flash costs $0.42 in tab. 2 and $0.213 at baseline in tab. 3. The paper says the tab. 3 costs use the Batch API §5.1; tab. 2 does not say how it was priced.
  • Single run, small margin. HERIT leads the human-data-only Qwen3 32B by 1.5 BLEU tab. 4. No seeds, confidence intervals or significance tests are reported.
  • H2KE is not like-for-like. Its BLEU is copied from the original paper tab. 4 caption, so the test set and tokenisation may differ.
  • One judge does two jobs. Sonnet-4.6 scores the outputs (1 to 10) with the Korean text and glossary as references §5.3, and it also scores proper-noun accuracy tab. 11. The judge prompt and whether scoring is per translation or pairwise are not stated.
  • Per-reign results are only in a figure. Use fig. 4 for the reign-level numbers; they are not in the text §6.1.
  • Mixed-data baseline. The paper says mixing D^aug and D^train in one run underperforms the two-stage schedule §5.2 tab. 4. Whether the mixed run used the same learning rate and epochs is not stated.
  • Long inputs. Documents over the limit are chunked at inference § Limitations. The chunk size, overlap and merge rule are not stated.
  • Figures are described, not reproduced. This card gives no image. Figure and table numbers follow the compiled paper's order; the appendix is lettered (A to F) under \appendix.

Lineage

  • Builds on: FusioN, "Making, not Taking, the Best of N" (arXiv:2510.00931, not compiled here), used for draft fusion §2 §4.2. Retrieval-augmented generation for few-shot prompting §2 §4.2.
  • Compared against: H2KE (BLEU from its original paper) and a Pivot-MT baseline built by the authors tab. 4 §5.2; zero-shot Gemini-2.5 Flash and Pro, Sonnet-4.5, GPT-5.1, Kimi-K2, Qwen3 and Hunyuan-MT-7B tab. 2 §4.1.
  • Superseded by: not stated.

Source and license

Adapted from “HERIT: Enabling Global Access to Hanja-to-English Historical Translation by Mitigating Temporal Bias” by Dayeon Jeong, Kyeongpil Kang (arXiv:2611.04949), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/dandamdandam/HERIT at commit 03543cd6074a (none (no LICENSE file in repository)). Any quoted code is shown under that license.