Paper · 2021-06-30 · cs.CL
ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information
ChineseBERT fuses each character's ordinary embedding with a glyph vector (three font bitmaps) and a pinyin vector (romanised reading) before a BERT-style Transformer, pretrained from scratch with whole-word and char masking. Author-reported: new state of the art on MRC, NLI, text classification and sentence matching; OntoNotes 4.0 NER F1 81.65 (base) vs 80.37 for RoBERTa.
Language model pretrainingMasked language modelingNamed entity recognitionchinese-nlppretrainingembeddingsglyphpinyin
arXivPDFCodeMarkdown14 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Chinese characters carry information that a standard BERT model barely sees: what a character looks like, and how it is pronounced. One character can be read in more than one way, with a different meaning for each reading. The shape of a character can also hint at its topic, because several characters share a "water" component. ChineseBERT is a Chinese language model that gives each character two extra signals next to its usual learned vector. One signal comes from the character's shape, drawn in three Chinese font styles. The other comes from its pronunciation, written in the Latin alphabet with tone marks.
The three signals are combined into one vector per character, which then goes into a standard BERT-style Transformer. The model is trained on a large set of Chinese web text with the usual masked-prediction task, in which parts of the text are hidden and must be predicted. It hides whole words as well as single characters.
The authors report that the extra signals help on a range of Chinese benchmarks, and that the model reaches those results with fewer training steps than the baselines. These are the authors' own numbers.
TL;DR
ChineseBERT is a BERT-style Chinese encoder pretrained from scratch. Each token's input embedding fuses its ordinary embedding with a glyph vector built from three 24×24 font bitmaps and a pinyin vector built from its romanised reading. Pretraining uses whole-word and char masking. The authors report new state of the art on Chinese MRC, NLI, text classification and sentence-pair matching, competitive results on NER and word segmentation, and fewer pretraining steps than their baselines. The repository is MIT-licensed and holds inference and fine-tuning code, but not the pretraining code.
Problem
Pretrained Chinese encoders of the time used only the identity of each character (or word) as their lexical signal §1. The paper names two gaps §1:
- Glyph: a character's form carries semantics. The authors give the example of 液, 河 and 湖, which share the water radical. Glyph features had been studied, but not in large-scale pretraining §1 §2.2.
- Pinyin: about 700 of 7000 common characters have more than one pronunciation, and each pronunciation can mean something different §1. A character ID embedding points to the same entry for every reading, and a glyph embedding is the same for every reading, so neither separates them §1.
Two practical limits follow from the design. The model needs a font rendering and a pinyin reading for every character, so the pipeline depends on fonts and a pinyin library §3.2. ChineseBERT also cannot start from an existing BERT checkpoint, because its input layer differs from vanilla BERT, so it is pretrained from scratch and trained for fewer steps than the BERT-wwm and MacBERT baselines, which start from pretrained BERT §5.
Method
ChineseBERT replaces BERT's input layer and keeps the Transformer encoder §3.1. For each character there are three embeddings: char, glyph and pinyin. They are concatenated, mapped by a fully connected layer to the model width D, and added to a learnable absolute position embedding. The result feeds the BERT encoder. The paper omits the segment (token-type) embedding because it does not use next-sentence prediction §3.1. Pretraining uses whole-word masking and char masking §3.1 §4.2.
The paper contains no numbered equations. The steps are in prose and in Figures 1 to 4 §3.1 §3.2 fig. 1 fig. 2 fig. 3 fig. 4.
# ChineseBERT input embedding, one row per token [§3.1] [§3.2]
# Input: input_ids (B, L), pinyin_ids (B, L, 8), positions (B, L)
# Output: embeddings (B, L, D)
def embed(input_ids, pinyin_ids, positions):
# Char embedding: the ordinary learned lookup, as in BERT [§3.2]
char_vectors = char_table[input_ids] # (B, L, D)
# Glyph: the token's three 24x24 font bitmaps, flattened [§3.2] [fig. 2]
font_rows = font_table[input_ids] # (B, L, 1728)
glyph_vectors = glyph_fc(font_rows) # (B, L, D)
# Pinyin: 8 letter and tone slots per token, embedded [§3.2] [fig. 3]
slot_vectors = pinyin_table[pinyin_ids] # (B, L, 8, 128)
# Width-2 convolution over the slots, then max-pool [§3.2] [fig. 3]
pinyin_vectors = pinyin_pool(pinyin_conv(slot_vectors)) # (B, L, D)
# Fusion: concatenate the three vectors, project to D [§3.2] [fig. 4]
fused = fusion_fc(
concat(char_vectors, pinyin_vectors, glyph_vectors)
)
# Add the learned absolute position embedding [§3.2]
return fused + position_table[positions]Glyph embedding. Each character is rendered in three fonts, FangSong, XingKai and LiShu. Each rendering is a 24×24 grid with pixel values from 0 to 255. The three are stacked into 24×24×3, flattened and passed through a fully connected layer §3.2 fig. 2. The code, however, stores the flattened font vectors as a per-token lookup table, initialised from font files, and applies the fully connected layer on top glyph_embedding.py fusion_embedding.py fusion_embedding.py.
Pinyin embedding. The pinyin of each character is produced with tone marks, using the pypinyin package §3.2 fig. 3. The letters and tone tokens are padded to a fixed length of 8 slots, and a convolution of width 2 runs over the slots, followed by max-pooling. Max-pooling makes the output size independent of the syllable length §3.2. The code uses a 128-dimensional slot embedding, a width-2 convolution with D output channels, and max-pooling over the slots pinyin_embedding.py pinyin_embedding.py fusion_embedding.py.
Fusion. The three vectors are concatenated into a 3D-dimensional vector and a fully connected layer maps it to D §3.2 fig. 4. The code concatenates in the order char, pinyin, glyph fusion_embedding.py, which is a different order from the figure caption's listing (char, glyph, pinyin) §3.2 fig. 4.
Pretraining data. Text is taken from CommonCrawl. After filtering out text with too much English and removing HTML markup, about 10% of the data is kept, which the paper puts at 4B Chinese characters §4.1. The LTP toolkit gives word boundaries for whole-word masking §4.1.
Pretraining masks. In nine out of ten cases the masking works on whole words; in the rest it works on single characters. Each word or character is selected with probability 0.15. A selected unit is replaced by the mask token 80% of the time, by a random unit 10% of the time, and left unchanged 10% of the time. Dynamic masking is used to avoid duplicate training instances §4.3.
# Pretraining mask construction for one sequence [§4.2] [§4.3]
# Input: units (one entry per character, grouped into words)
# Output: masked units, labels (None where not scored)
def mask_sequence(units, words):
# Whole-word masking 90% of the time, char masking 10% [§4.3]
if random() < 0.9:
groups = words
else:
groups = [[index] for index in range(len(units))]
labels = [None] * len(units)
for group in groups:
# Each word or character is selected with probability 0.15 [§4.3]
if random() < 0.15:
for index in group:
labels[index] = units[index]
units[index] = replacement(units[index])
return units, labels
def replacement(unit):
# 80% mask token, 10% random unit, 10% unchanged [§4.3]
roll = random()
if roll < 0.8:
return MASK_TOKEN
if roll < 0.9:
return random_unit()
return unitPacked and single inputs. Training alternates between packed inputs, which concatenate several sentences up to 512 tokens, drawn with probability 0.9, and single sentences, drawn with probability 0.1 §4.3.
Inference pinyin ids. The code builds the 8-slot pinyin ids for each token from the sentence. It takes the first pypinyin candidate for each character, uses a precomputed pinyin-to-ids table where one exists, and otherwise spells the syllable letter by letter. Tokens that are not single Chinese characters get all-zero ids bert_dataset.py.
# Pinyin ids per token [code: datasets/bert_dataset.py#L55-L85]
# Input: sentence (str), tokens with character offsets
# Output: pinyin_ids (num_tokens, 8)
def pinyin_ids_for(sentence, tokens):
# First candidate reading per character, tone-numbered
# [code: datasets/bert_dataset.py#L55-L59]
readings = first_candidate_per_char(sentence)
ids_by_offset = {}
for offset, reading in readings:
# Known syllables come from a precomputed table
# [code: datasets/bert_dataset.py#L63-L65]
ids_by_offset[offset] = table_or_spelled(reading)
rows = []
for token in tokens:
# Only single characters are looked up; others get zeros
# [code: datasets/bert_dataset.py#L77-L84]
if is_single_char(token) and token.offset in ids_by_offset:
rows.append(ids_by_offset[token.offset])
else:
rows.append([0] * 8)
return rowsHyperparameters
Pretraining, as stated in the paper:
| Name | Value | Where | Notes |
|---|---|---|---|
| Layers / hidden / heads (base) | 12 / 768 / 12 | §4.3 | |
| Layers / hidden / heads (large) | 24 / 1024 / 16 | §4.3 | |
| Peak learning rate (base) | 1e-4 | §4.3 | Schedule beyond warmup not stated |
| Peak learning rate (large) | 3e-4 | §4.3 | |
| Warmup steps (base) | 20K | §4.3 | |
| Warmup steps (large) | 90K | §4.3 | |
| Batch size (base) | 3.2k | §4.3 | Unit (sequences or tokens) not stated |
| Batch size (large) | 8k | §4.3 | Unit not stated |
| Training steps (base) | 500K | §4.3 | Stated "upon the submission"; tab. 1 lists 1M for ChineseBERT |
| Training steps (large) | 280K | §4.3 | Same caveat |
| Packed input length | 512 tokens, probability 0.9 | §4.3 | Single sentences drawn with probability 0.1 |
| Masking mix | WWM 90%, CM 10% | §4.3 | Per-example or per-word choice not stated |
| Mask probability | 0.15 per word or char | §4.3 | |
| Replacement | 80% mask, 10% random, 10% kept | §4.3 | |
| Masking regime | dynamic | §4.3 | |
| Vocabulary size | 21K | tab. 1 | |
| Pretraining data | CommonCrawl, about 10% kept, 4B characters | §4.1 | |
| Font images | 3 fonts, 24×24 px, values 0 to 255 | §3.2 fig. 2 | |
| Glyph vector length | 2,352 (as written) | §3.2 | Code uses 1728; see Paper vs code |
| Pinyin slots | 8 per character | §3.2 | Padding symbol described as "-" |
| Pinyin convolution | width 2, then max-pool | §3.2 fig. 3 | |
| Pinyin embedding size | not stated | fusion_embedding.py | Code uses 128 |
| Optimizer, weight decay, dropout, hardware | not stated | Pretraining code is not in the repository |
Fine-tuning. The paper gives no fine-tuning hyperparameters §5. The repository's scripts for the two NER tasks are listed below, and the other task READMEs are in tasks/*/README.md README.md.
| Name | OntoNotes 4.0, base | Weibo, base | Source |
|---|---|---|---|
| Learning rate | 3e-5 | 4e-5 | chinesebert_base.sh chinesebert_base.sh |
| Batch size | 26 | 2 | chinesebert_base.sh chinesebert_base.sh |
| Epochs | 5 | 10 | chinesebert_base.sh chinesebert_base.sh |
| Max length | 275 | 150 | chinesebert_base.sh chinesebert_base.sh |
| Warmup proportion | 0.1 | 0.02 | chinesebert_base.sh chinesebert_base.sh |
| Weight decay | 0.001 | 0.01 | chinesebert_base.sh chinesebert_base.sh |
| Dropout | 0.2 | 0.2 | chinesebert_base.sh chinesebert_base.sh |
| Optimizer | torch.adam, which is AdamW | torch.adam, which is AdamW | OntoNotes_trainer.py |
| Classifier head | MLP (Linear, Tanh, Linear) | MLP | chinesebert_base.sh classifier.py |
Code-side optimiser details: AdamW with default betas, epsilon 1e-8, and no weight decay on biases and LayerNorm weights; linear warmup over the given fraction of steps, then linear decay OntoNotes_trainer.py OntoNotes_trainer.py. The OntoNotes large settings use batch 18 with gradient accumulation of 2 README.md, and the large script sets weight decay 0.002 chinesebert_large.sh.
Results
Author-reported. Each task is fine-tuned separately on its own dataset §5. Unless stated otherwise, "BERT" and "RoBERTa" mean BERT-wwm and RoBERTa-wwm §5. A "°" marks a model pretrained on extended data tab. 2 tab. 4. Baselines are ERNIE, BERT-wwm, MacBERT and RoBERTa-wwm §5. Hardware is not stated.
Machine reading comprehension, CMRC 2018, metric EM. Source: tab. 2. CMRC test scores come from the CLUE leaderboard §5.1.
| Model | Size | Dev | Test |
|---|---|---|---|
| ERNIE | base | 66.89 | 74.70 |
| BERT | base | 66.77 | 71.60 |
| BERT° | base | 66.96 | 73.95 |
| RoBERTa° | base | 67.89 | 75.20 |
| MacBERT | base | -- | -- |
| ChineseBERT | base | 67.95 | 75.35 |
| RoBERTa° | large | 70.59 | 77.95 |
| MacBERT | large | -- | -- |
| ChineseBERT | large | 70.70 | 78.05 |
Machine reading comprehension, CJRC, metrics EM and F1. Source: tab. 3. The split differs from the one used by the BERT-wwm authors, so the authors re-ran the released baselines on this split §5.1.
| Model | Size | Dev EM | Dev F1 | Test EM | Test F1 |
|---|---|---|---|---|---|
| BERT | base | 59.8 | 73.0 | 60.2 | 73.0 |
| BERT° | base | 60.8 | 74.0 | 61.4 | 73.9 |
| RoBERTa° | base | 62.9 | 76.6 | 63.8 | 76.6 |
| ChineseBERT | base | 65.2 | 77.8 | 66.2 | 77.9 |
| RoBERTa° | large | 65.6 | 77.5 | 66.4 | 77.6 |
| ChineseBERT | large | 66.5 | 77.9 | 67.0 | 78.3 |
Natural language inference, XNLI, accuracy. Source: tab. 4. Training uses the official machine-translated Chinese data §5.2.
| Model | Size | Dev | Test |
|---|---|---|---|
| ERNIE | base | 79.7 | 78.6 |
| BERT | base | 79.0 | 78.2 |
| BERT° | base | 79.4 | 78.7 |
| RoBERTa° | base | 80.0 | 78.8 |
| MacBERT | base | 80.3 | 79.3 |
| ChineseBERT | base | 80.5 | 79.6 |
| RoBERTa° | large | 82.1 | 81.2 |
| MacBERT | large | 82.4 | 81.3 |
| ChineseBERT | large | 82.7 | 81.6 |
Text classification, accuracy. Source: tab. 5. The TNEWS baseline numbers are copied from the CLUE paper, not re-run tab. 5.
| Model | Size | ChnSentiCorp Dev | ChnSentiCorp Test | THUCNews Dev | THUCNews Test | TNEWS Dev | TNEWS Test |
|---|---|---|---|---|---|---|---|
| ERNIE | base | 95.4 | 95.5 | 97.6 | 97.5 | 58.24 | 58.33 |
| BERT | base | 95.1 | 95.4 | 98.0 | 97.8 | 56.09 | 56.58 |
| BERT° | base | 95.4 | 95.3 | 97.7 | 97.7 | 56.77 | 56.86 |
| RoBERTa° | base | 95.0 | 95.6 | 98.3 | 97.8 | 57.51 | 56.94 |
| MacBERT | base | 95.2 | 95.6 | 98.2 | 97.7 | -- | -- |
| ChineseBERT | base | 95.6 | 95.7 | 98.1 | 97.9 | 58.64 | 58.95 |
| RoBERTa° | large | 95.8 | 95.8 | 98.3 | 97.8 | 58.32 | 58.61 |
| MacBERT | large | 95.7 | 95.9 | 98.1 | 97.9 | -- | -- |
| ChineseBERT | large | 95.8 | 95.9 | 98.3 | 97.9 | 59.06 | 59.47 |
Sentence-pair matching, accuracy. Source: tab. 6.
| Model | Size | LCQMC Dev | LCQMC Test | BQ Dev | BQ Test |
|---|---|---|---|---|---|
| ERNIE | base | 89.8 | 87.2 | 86.3 | 85.0 |
| BERT | base | 89.4 | 87.0 | 86.1 | 85.2 |
| BERT° | base | 89.6 | 87.1 | 86.4 | 85.3 |
| RoBERTa° | base | 89.0 | 86.4 | 86.0 | 85.0 |
| MacBERT | base | 89.5 | 87.0 | 86.0 | 85.2 |
| ChineseBERT | base | 89.8 | 87.4 | 86.4 | 85.2 |
| RoBERTa° | large | 90.4 | 87.0 | 86.3 | 85.8 |
| MacBERT | large | 90.6 | 87.6 | 86.2 | 85.6 |
| ChineseBERT | large | 90.5 | 87.8 | 86.5 | 86.0 |
Named entity recognition, span-level precision, recall and F1, on test sets. Source: tab. 7.
| Model | Size | OntoNotes P | OntoNotes R | OntoNotes F1 | Weibo P | Weibo R | Weibo F1 |
|---|---|---|---|---|---|---|---|
| BERT | base | 79.69 | 82.09 | 80.87 | 67.12 | 66.88 | 67.33 |
| RoBERTa° | base | 80.43 | 80.30 | 80.37 | 68.49 | 67.81 | 68.15 |
| ChineseBERT | base | 80.03 | 83.33 | 81.65 | 68.27 | 69.78 | 69.02 |
| RoBERTa° | large | 80.72 | 82.07 | 81.39 | 66.74 | 70.02 | 68.35 |
| ChineseBERT | large | 80.77 | 83.65 | 82.18 | 68.75 | 72.97 | 70.80 |
Chinese word segmentation, F1 and accuracy, on the PKU and MSRA datasets. Source: tab. 8. Which split is reported is not stated.
| Model | Size | MSRA F1 | MSRA Acc | PKU F1 | PKU Acc |
|---|---|---|---|---|---|
| BERT° | base | 98.42 | 99.04 | 96.82 | 97.70 |
| RoBERTa° | base | 98.46 | 99.10 | 96.88 | 97.72 |
| ChineseBERT | base | 98.60 | 99.14 | 97.02 | 97.81 |
| RoBERTa° | large | 98.49 | 99.13 | 96.95 | 97.80 |
| ChineseBERT | large | 98.67 | 99.26 | 97.16 | 98.01 |
Ablation on OntoNotes 4.0, base models pretrained on the same data and steps, span-level test scores. Source: tab. 9 §6.1.
| Variant | Precision | Recall | F1 | Change in F1 |
|---|---|---|---|---|
| RoBERTa° (baseline row) | 80.43 | 80.30 | 80.37 | -- |
| ChineseBERT (full) | 80.03 | 83.33 | 81.65 | -- |
| Without glyph | 77.67 | 82.75 | 80.13 | -1.52 |
| Without pinyin | 77.54 | 83.65 | 80.48 | -1.17 |
| Without glyph and pinyin | 78.22 | 81.37 | 79.76 | -1.89 |
Training-set size, OntoNotes 4.0 NER test F1, averaged over five random subsamples §6.2. The plotted series come from the figure's data in the source. ChineseBERT is highest at every plotted point, from 77.08 to 81.65, against RoBERTa at 76.59 to 80.37 and BERT at 75.50 to 80.87 fig. 5. The text says the sampled fraction runs from 10% to 90% §6.2, while the plot has ten points on an axis labelled 0 to 1.0 fig. 5. The text reports that the gain is slight below 30% of the data and larger above it §6.2.
Paper vs code
Code at commit 67f45cda50e75053ca3143bb8fadc3748cdfd13e. It contains inference and fine-tuning code, but no pretraining code fill_mask.py.
- Glyph size. The paper says the 24×24×3 stack becomes a 2,352-vector §3.2. 24×24×3 is 1,728, and the code uses 1,728 fusion_embedding.py. The paper's figure is an arithmetic error fig. 2.
- Glyph input. The paper describes the flattened font image passed through a fully connected layer §3.2 fig. 2. The code keeps a lookup table over the whole vocabulary, one row per token, initialised from font files, and applies the fully connected layer on top glyph_embedding.py fusion_embedding.py.
- Token-type embedding. The paper omits the segment embedding §3.1. The code adds a token-type embedding in every forward pass; with no token-type input it uses the zero row fusion_embedding.py fusion_embedding.py fusion_embedding.py.
- Post-fusion normalisation and dropout. The paper does not describe a layer norm or dropout after the sum. The code applies both fusion_embedding.py.
- Concatenation order. The paper's Figure 4 caption lists char, glyph, pinyin §3.2 fig. 4. The code concatenates char, pinyin, glyph fusion_embedding.py.
- Pinyin reading. The paper describes pypinyin as inferring pronunciation from context §3.2. The code takes the first candidate from pypinyin's heteronym list for each character and does no context selection of its own bert_dataset.py.
- Pinyin padding. The paper fills short pinyin sequences with a special "-" letter §3.2. The code fills with zero ids bert_dataset.py bert_dataset.py. The meaning of id 0 lives in
pinyin_map.json, which is not in the repository. - Masked positions at inference. The code replaces the masked token with id 103 and zeroes its pinyin row bert_mask_dataset.py. The paper does not say what happens to the pinyin of masked positions during pretraining.
- Pretraining. The paper gives pretraining hyperparameters §4.3. The repository has no pretraining script, so none of them can be checked fill_mask.py.
- Paper-internal step counts. Table 1 gives 1M training steps and 5B tokens for ChineseBERT tab. 1. The text gives 4B characters §4.1 and 500K and 280K steps §4.3.
- Published NER score. The base OntoNotes script records a test F1 of 81.75 (precision 80.36, recall 83.18) chinesebert_base.sh. The paper reports 81.65 (80.03, 83.33) tab. 7. The script's run is not the paper's reported run.
- README CMRC score. The code README lists the ChineseBERT CMRC test score as 95.7 README.md. The paper reports 75.35 tab. 2.
- Model selection for NER. The trainer saves the top checkpoints by dev F1 and tests the best one OntoNotes_trainer.py OntoNotes_trainer.py. The paper does not state its selection rule.
Pitfalls
- Use 1,728 glyph inputs for three 24×24 renderings (24×24×3). The paper's 2,352 cannot be reproduced from its own description §3.2.
- The glyph table is trainable. The code creates it with a learnable embedding and no freeze flags anywhere in the repository glyph_embedding.py. This is an inference from the code, not a stated design choice. It costs about vocab × 1,728 parameters: at 21K tokens, about 36M, computed from tab. 1 and the code. Freezing it, or replacing it with rendered images, changes the model.
- The font
.npyfiles,pinyin_map.json,id2pinyin.jsonandpinyin2tensor.jsonare not in the repository. They ship with the model download README.md fusion_embedding.py. Their normalisation and the index-0 padding meaning cannot be verified from this code. - Pinyin is the first candidate reading per character, not context-selected by this code bert_dataset.py. Whether that reading is right for a given context depends on pypinyin's behaviour.
- Tokens that are not single characters get all-zero pinyin rows, including multi-character subwords and non-Chinese text bert_dataset.py. A model fed those tokens relies on the glyph and char rows alone.
- Whole-word masking: the paper does not say whether the 90% and 10% strategy choice is made per sequence or per word, nor whether the 80/10/10 replacement is decided once per word or per character §4.3. The pseudocode above makes one decision per sequence and one per word, which is an interpretation.
- The token-type row is not in the paper's design. Ports that follow the paper exactly will not load the code's checkpoints without it §3.1 fusion_embedding.py.
- Pretraining cost is not stated. Compute, hardware and wall-clock time are missing from the compiled text §4.3.
- Unless stated otherwise, "BERT" and "RoBERTa" in the paper mean BERT-wwm and RoBERTa-wwm, not vanilla BERT §5. Tables list a plain BERT row too, so read the baseline's name in each row.
- CJRC results come from a different split than the BERT-wwm authors used, so they are not comparable with that work §5.1.
- TNEWS baseline numbers are copied from the CLUE paper and were not re-run tab. 5.
- The weights' licence was not checked here. The repository's MIT licence LICENSE covers the code only, so do not treat it as covering the checkpoints.
- The ablation's "without glyph and pinyin" row is not RoBERTa. The authors say it was trained on a smaller data set for fewer steps §6.1.
Lineage
- Builds on: BERT, with whole-word masking adapted from the BERT-wwm work §2.1 §4.2. Glyph embeddings follow Glyce, which used a CNN where this paper uses a fully connected layer §2.2 §3.2.
- Compared against: ERNIE, BERT-wwm, MacBERT and RoBERTa-wwm §5 tab. 2 tab. 4.
- Superseded by: not known from wiki content.
Source and license
Adapted from “ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information” by Zijun Sun, Xiaoya Li, Xiaofei Sun, Yuxian Meng, Guoyin Wang, Xiang Ao, Qing He, Fei Wu, Jiwei Li (arXiv:2106.16038), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/ShannonAI/ChineseBert at commit 67f45cda50e7 (MIT). Any quoted code is shown under that license.