Deep Learning for Classical Japanese Literature

In plain words

Old Japanese books are printed and written in a cursive script called Kuzushiji. Most Japanese readers today cannot read it, so a large body of historical text is hard to access. The authors wanted machine learning researchers to work on this material, so they built image datasets of single characters cut from old books and released them with code.

The first dataset is a drop-in replacement for MNIST, the classic handwritten-digit benchmark, but with ten Japanese character classes. The second has 49 classes, and the third has a few thousand Kanji characters, many of them rare. Baselines reach high accuracy on the first two, and stronger models do better still.

The paper also tries a more creative task: turning an old Kanji image into a drawing of the same character in modern stroke form. The method squeezes both styles into small codes with autoencoders, learns how the old codes map onto the modern ones, and then draws strokes from the mapped code. The authors show examples of the output but do not score it.

TL;DR

The paper releases three character-image datasets cut from cursive Japanese books: Kuzushiji-MNIST (10 classes, 28x28), Kuzushiji-49 (49 classes, 28x28) and Kuzushiji-Kanji (3,832 classes, 64x64) §2. It reports test accuracy for classification baselines on the first two tab. 1. It also shows a domain-transfer experiment from Kuzushiji-Kanji to Modern Kanji stroke sequences, but gives no numeric score for it §3.2. The linked repository contains the kNN and small-CNN baseline scripts only; the ResNet baselines and the transfer model are not in it.

Problem

• Kuzushiji was in use for over 1,000 years. After the Meiji-era reforms of the education system it was dropped from the official curriculum, so most Japanese people today cannot read books printed more than 150 years ago §1.
• Digitised copies of many books exist, but few people can read them, so most of their content stays inaccessible §1.
• Benchmarks in machine learning are often chosen for their technical properties rather than for social or cultural value; the authors want a benchmark of the latter kind §1.
• Classes are heavily imbalanced in two of the three datasets. In Kuzushiji-49 one character has only about 400 samples §2. In Kuzushiji-Kanji counts range from 1,766 per class down to a single example §2.
• Within one class, the same character can be written in several unrelated forms (hentaigana). A class can therefore be multi-modal §2 fig. 6.

Method

Dataset construction §2

Characters were cropped from 35 classical books printed in the 18th century. The same 35 books supply the train and test splits, so the class distributions match between the two §2.

Kuzushiji-MNIST takes one character from each of the ten rows of hiragana, which makes it a drop-in replacement for MNIST §2. Kuzushiji-49 covers all 48 hiragana plus one iteration mark §2. Kuzushiji-Kanji covers 3,832 Kanji classes §2. The split and size of each dataset is given below:

• Kuzushiji-49 and Kuzushiji-Kanji stay imbalanced, because the class counts follow how often each character appears in the source books §2.
• Hentaigana make the mapping many-to-one: one hiragana class can contain several unrelated written forms, and the authors say this makes the task harder than MNIST §2 fig. 6.
• The design of Kuzushiji-MNIST was inspired by Fashion-MNIST, which the authors describe as harder than MNIST while keeping the same ease of use §2.
• Figures: sample Kanji from Kuzushiji-Kanji fig. 7; the ten Kuzushiji-MNIST classes next to their modern hiragana fig. 1 fig. 5; the Kuzushiji-49 split fig. 8; images of bookstores and flea markets where the source books were sold fig. 4.

Classification baselines §3.1

Four baselines are reported on MNIST, Kuzushiji-MNIST and Kuzushiji-49 tab. 1:

• a 4-nearest-neighbour classifier on raw pixels;
• a small two-layer convolutional network ("Keras Simple CNN Benchmark" in the table);
• PreActResNet-18, an 18-layer residual network §3.1;
• PreActResNet-18 trained with input mixup and with manifold mixup §3.1.

The paper gives no training details for these. It points to the dataset repository for setup §3.1.

Domain transfer: Kuzushiji-Kanji to Modern Kanji §3.2

The goal is to produce a stroke-based Modern Kanji for a given Kuzushiji-Kanji input. The authors contrast this with prior domain-transfer work, which works on pixel images §3.2.

The training procedure is summarised in alg. 1 and in the overview diagram fig. 11:

1. Train two convolutional VAEs, one on Kuzushiji-Kanji (pixels, 64x64) and one on pixel renders of KanjiVG Modern Kanji (64x64). Each encodes its input into a 64-dimensional latent vector, z_old and z_new. The architecture is identical to the one in the World Models paper [ha2018recurrent] §3.2.
2. For the KL term, do not push the loss below a threshold, following [ha2017neural]. This keeps some information in the latent and still pulls it toward a Gaussian prior §3.2.
3. Train a mixture density network with two hidden layers to model P(z_new | z_old) as a mixture of Gaussians §3.2 alg. 1.
4. Train a Sketch-RNN decoder that generates KanjiVG strokes from a latent. For Modern Kanji that also appear in Kuzushiji-Kanji (around 3,600 of them), the latent is a sample from the MDN given z_old. For all other characters it is z_new encoded from KanjiVG §3.2 fig. 11.
5. At inference, encode a Kuzushiji-Kanji image with the old VAE, sample a modern latent from the MDN, and decode it to strokes with Sketch-RNN fig. 11.

The paper states that the Sketch-RNN is trained on the stroke data after the VAEs, so the decoder can adjust parts of the latent space that the pixel VAE does not capture well §3.2.

# Domain transfer, Kuzushiji-Kanji to Modern Kanji strokes  [alg. 1] [§3.2]
# Input:  kanji_images (N, 64, 64), kanjivg_pixels (M, 64, 64),
#         kanjivg_strokes (M, ...)
# Output: stroke sequence for a given Kuzushiji-Kanji image

# Two convolutional VAEs, one per domain, 64-d latent  [§3.2] [alg. 1]
vae_old = train_conv_vae(kanji_images, latent_dim=64)
vae_new = train_conv_vae(kanjivg_pixels, latent_dim=64)

# Latent codes for each image; z_old and z_new  [§3.2] [fig. 11]
z_old = vae_old.encode(kanji_images)                     # (N, 64)
z_new = vae_new.encode(kanjivg_pixels)                   # (M, 64)

# Mixture density network for P(z_new | z_old), 2 hidden layers  [§3.2]
# The pairing of training samples is not stated; overlapping characters
# are the only ones with both codes available.
mdn = train_mdn(inputs=z_old_overlap, targets=z_new_overlap)

# Condition each KanjiVG character on a latent  [§3.2] [fig. 11]
conditions = []
for each character in kanjivg:
    if character in kanji_set:
        # Overlapping character: sample from the MDN  [§3.2]
        condition = mdn.sample(z_old[character])         # (64,)
    else:
        # Not in Kuzushiji-Kanji: use its own KanjiVG latent  [§3.2]
        condition = z_new[character]                     # (64,)
    conditions.append(condition)

# Sketch-RNN decoder trained on the stroke data  [§3.2] [alg. 1]
sketch_rnn = train_sketch_rnn(kanjivg_strokes, conditions)


# Inference for one Kuzushiji-Kanji image  [§3.2] [fig. 11]
def transfer(kuzushiji_image):
    # Encode the image with the old-domain VAE  [§3.2]
    latent_old = vae_old.encode(kuzushiji_image)         # (64,)
    # Draw a modern latent from the MDN  [§3.2]
    latent_new = mdn.sample(latent_old)                  # (64,)
    # Decode strokes from that latent  [§3.2] [fig. 11]
    return sketch_rnn.decode(latent_new)

Figures describing this experiment: the teaser fig. 3 shows Kuzushiji-Kanji inputs and their Modern Kanji outputs. The problem statement fig. 9 pairs 64x64 Kuzushiji-Kanji samples with stroke-based Modern Kanji. More examples, including the VAE pixel reconstructions for both domains, are in fig. 10. The authors say the pixel reconstructions help human transcribers see which regions are uncertain §3.2.

Hyperparameters

Most values for the classification baselines are in the repository, because the paper defers to it §3.1. The "Where" column says which source gives each value.

Results

Author-reported. Table 1 gives test accuracy on the three benchmarks. The caption says each number is the mean of per-class accuracies, which the authors use to account for class imbalance tab. 1. Bold marks the best value in each column in the paper. Hardware is not stated. Baselines are the paper's own §3.1.

Domain transfer (Kuzushiji-Kanji to Modern Kanji): no quantitative metric is reported. The evidence is qualitative examples of the output on test-set inputs §3.2 fig. 3 fig. 9 fig. 10.

Paper vs code

The repository at code_commit is the dataset repository. It holds the dataset download script and two benchmark scripts. It has no code for the VAEs, the MDN, Sketch-RNN, or the ResNet baselines download_data.py kuzushiji_mnist_cnn.py kuzushiji_mnist_knn.py.

• kNN numbers differ. The paper reports 91.56% on Kuzushiji-MNIST and 86.01% on Kuzushiji-49 tab. 1. The repository's README reports 92.10% and 83.65% README.md, and the kNN script's header comment reports 92.10% kuzushiji_mnist_knn.py. The paper does not say which kNN settings it used. The script uses distance weighting kuzushiji_mnist_knn.py.
• CNN numbers differ. The paper reports 95.12% and 89.25% tab. 1. The README reports 94.63% and 89.36% README.md. The README says the dataset had an image-processing fix in February 2019 and that the newer version gives slightly better results README.md. The paper's arXiv version is from December 2018, so its numbers predate the fix.
• Metric differs for Kuzushiji-49. The paper says all numbers use mean per-class accuracy tab. 1. The CNN script prints plain Keras accuracy kuzushiji_mnist_cnn.py. The README asks for balanced accuracy on Kuzushiji-49 and gives the formula README.md. On the balanced Kuzushiji-MNIST the two coincide; on Kuzushiji-49 they do not.
• Dataset sizes differ. The paper gives 266,407 images for Kuzushiji-49 §2 and 140,426 for Kuzushiji-Kanji §2. The README gives 270,912 images for Kuzushiji-49 (232,365 train, 38,547 test) README.md README.md and 140,424 for Kuzushiji-Kanji README.md.
• Starred ResNet rows. The README marks the PreActResNet-18 rows as obtained with an older dataset version. These values are the ones in the paper's table README.md README.md. The paper does not say this.
• Keras-derived script. The CNN script says it is based on a Keras example under the MIT license kuzushiji_mnist_cnn.py. The repository's LICENSE is CC BY-SA 4.0 LICENSE. No code is quoted in this card, but anyone copying the script should check both.

Pitfalls

• Training details for the classification baselines are not in the paper §3.1. Use the repository scripts for the kNN and CNN settings. The ResNet and mixup numbers cannot be reproduced from the repository.
• The CNN script passes the test set as validation_data on every epoch kuzushiji_mnist_cnn.py. No model selection is coded, so the evaluated model appears to be the one after the last epoch (inferred from the code). Choosing settings on the test set would still leak.
• The domain transfer result is qualitative only §3.2. There is no metric for how well the stroke output matches the modern character, and the examples are the authors' selection.
• Several parts of the transfer method are not stated: the VAE KL threshold, the MDN width and number of components, how the MDN is trained (the paper states it models P(z_new | z_old) but not which pairs it trains on), whether MDN samples are drawn once or during training, whether the encoder mean or a sampled latent is used, and the Sketch-RNN settings §3.2.
• The overlap of about 3,600 characters is approximate and the list is not given §3.2.
• The paper states that training two VAEs separately works better than one end-to-end model, based on the authors' experience. No ablation is shown §3.2.
• "3,999 character types" §2 refers to the full CODH Kuzushiji dataset, not to Kuzushiji-Kanji. Do not confuse the two.
• Attribution for reuse: the repository suggests crediting Kuzushiji-MNIST as a derivative of the original Kuzushiji dataset, which NIJL and others created, and it names the DOI of that original dataset README.md. The dataset's license is CC BY-SA 4.0 README.md.

Lineage

• Builds on: World Models for the VAE architecture, cited in the paper as [ha2018recurrent] §3.2. Sketch-RNN, cited as [ha2017neural] §3.2; the paper gives no arXiv id and no card exists here. Mixture density networks [bishop_mdn] and KanjiVG [kanjivg], both cited by name only §3.2. MNIST [lecun1998mnist] and Fashion-MNIST [xiao2017fashion] as design references §2.
• Compared against: the paper's own baselines tab. 1. For domain transfer it contrasts its approach with pixel-domain transfer work, cited under the keys isola2017image, liu2017unsupervised, wolf2017unsupervised and bousmalis2017unsupervised. The paper names them without giving their arXiv ids, and no cards exist here §3.2.
• Superseded by: not stated in this wiki.