Partial multi-view learning

Overview

A multi-view sample has several views, for example several sensors or several feature sets. In partial multi-view data, some views are absent for some samples. The task is to classify, or to learn representations, from whatever views each sample has 2011.06170 §1, §2.

The usual workarounds are to fill in the missing views, or to split samples by their missing pattern and fit one model per group. The first needs paired samples to learn the fill-in. The second does not scale well as the number of views grows 2011.06170 §1.

The shared idea in the one paper covered here is to keep every sample and every observed view in a single model. Each sample gets one latent code. A network per view decodes that code, and the reconstruction error counts only the views that are observed 2011.06170 §2.1. A class-structure term makes codes of the same class close, so classification can run on the codes 2011.06170 §2.2.

Variants

CPM-Nets (supervised)

The base model. It trains per-view decoders and one code per training sample, with reconstruction over observed views plus a weighted class loss. A new sample is handled by fitting a code to its observed views and then classifying by similarity to the training codes 2011.06170 §2.2, §2.3, alg. 1. Card: Deep Partial Multi-View Learning.

The paper reports classification accuracy above nine baselines across missing rates. On CUB with no missing rate stated for that sentence, it reports 89.48% against 86.27% for the best baseline, LMNN 2011.06170 §4.3. Gaps on the other datasets are given only as percentages, and in a figure [arxiv:2011.06170 §4.3, fig. 4].

CPM-GAN (unsupervised, with imputation)

The same per-view networks also act as generators for the missing entries. One discriminator per view is trained on observed values, and its adversarial loss applies only to the missing entries 2011.06170 §3, eq. 10. The paper uses this variant for unsupervised clustering, and for imputation compared with mean fill, SVD and a low-rank method 2011.06170 §4.4.

The paper's unsupervised figures are averages over several multi-view sets at a missing rate of 0.5. They are 3.36% higher ACC and 5.11% higher NMI than the second-best method 2011.06170 §4.4.2, fig. 8. Imputation error is reported as the lowest at every rate tested 2011.06170 §4.4.1, fig. 7.

When to use which

Caveat: all results are author-reported. The text gives gaps without naming the metric in every case, and some key numbers are only in figures 2011.06170 §4.3, fig. 4.

Implementation sketch

The common core of the paper's method is shown below. It is a simplification: the paper alternates decoder and code updates per iteration, and re-tunes the decoders before testing 2011.06170 alg. 1, §2.3.

# Common core: one code per sample, observed views only  [arxiv:2011.06170 §2.1]
# mask[n, v] = 1 if view v of sample n is observed, else 0

def train(views, mask, labels, lam):
    # One decoder per view, mapping a code back to that view  [§2.1]
    decoders = [make_decoder(v) for v in view_ids]
    # One free code per training sample  [§2.1]
    codes = random_init(num_samples, K)
    for step in range(num_steps):
        # Reconstruct observed views only; missing views add nothing  [eq. 5]
        recon = sum(mask[n, v] * sq_err(decoders[v](codes[n]), views[n, v])
                    for n in samples for v in view_ids)
        # Class term: mean similarity to each class's codes, hinge margin [eq. 7]
        class_term = hinge_class_loss(codes, labels)
        codes -= alpha * grad(recon + lam * class_term, codes)
        # Decoders are updated on the reconstruction term too  [alg. 1]
        decoders -= alpha * grad(recon, decoders)
    return decoders, codes

def predict(views, mask, decoders, train_codes, train_labels):
    # Fit a new code to the observed views only  [§2.3]
    code = random_init(K)
    for step in range(num_test_steps):
        code -= alpha * grad(masked_recon(code, views, mask, decoders), code)
    # Class score: mean similarity to each class's training codes  [§2.2]
    return argmax_c mean(dot(code, train_codes[train_labels == c]))

For CPM-GAN, add one discriminator per view. Its loss covers only missing entries, weighted by 1 minus the mask 2011.06170 eq. 10. The paper's algorithm alternates discriminator, generator and code updates 2011.06170 alg. 2.

Contradictions

No disagreements between papers are known, since only one paper is covered.

Within the paper, the λ grid differs between the hyperparameter section and the training-set section: {0.1, 1, 10} in one place and {0, 0.1, …, 10} in the other 2011.06170 §4.2; §4.3.1. The paper does not give a single value per dataset.

The released code differs from the paper in several settings. These include the learning rates, the latent size, dropout, and the absence of the GAN and re-tuning steps. The card lists each difference with a citation Deep Partial Multi-View Learning. Implementations should follow the paper's text, or state which setting they changed.

Papers

• Deep Partial Multi-View Learning: CPM-Nets for supervised partial multi-view classification, and CPM-GAN for unsupervised imputation and clustering.