Trusted Multi-View Classification

In plain words

Many classifiers that combine several sensors or feature sets (called views here) give each view equal say and output one probability per class. This method instead gives each view its own network, which reports how much evidence it has for each class. Evidence is turned into class probabilities plus an uncertainty amount, so the model can say how sure it is.

The views' answers are merged with a rule from Dempster-Shafer evidence theory. The merged answer lets a view that is confident count for more than one that is unsure. Training pushes the correct class to collect evidence and the wrong classes towards zero. The authors report higher accuracy than the single-view uncertainty baselines they compare with, and better behaviour on unusual inputs. All of these numbers are the authors' own.

TL;DR

• Use this when inputs come from several views or modalities and you need a per-prediction uncertainty, not only a class.
• One forward pass gives the fused class probabilities and an uncertainty; no sampling or ensembles are needed.
• Reported gains are on six multi-view benchmarks, with author-reported numbers, and the released code covers only the Handwritten dataset.
• Several implementation details differ between paper and code (see Paper vs code).

Problem

Multi-view models usually fuse views at the feature or output level, and they assume that each view's reliability is the same for every sample §1. Softmax confidence tends to be high even for wrong predictions, so it is a poor uncertainty signal §3, opening paragraph. Bayesian and ensemble methods give uncertainty but are costly, and the single-view uncertainty methods do not extend to fusing several views §1 §2.

Method

The method has three parts: (i) evidence and a Dirichlet opinion per view §3.1; (ii) fusion of the per-view opinions with Dempster's rule §3.2; (iii) a training loss that makes the evidence learnable §3.3. The optimisation loop is Algorithm 1 alg. 1 (app. A.2).

1. Evidence and opinions per view §3.1

• Each view v has a network whose last layer outputs one non-negative evidence value e_k^v per class k (the softmax is replaced by a non-negative activation) §3.3.
• The Dirichlet parameters are alpha_k^v = e_k^v + 1 §3.1.
• With S^v the Dirichlet strength, the belief mass per class is b_k^v = e_k^v / S^v, and the uncertainty mass is u^v = K / S^v. The masses sum to one eq. 1 eq. 2.
• The expected class probability is alpha_k^v / S^v §3.1.

2. Fusion across views §3.2

• Dempster's rule combines two opinions. The conflict C is the belief mass the two opinions give to different classes. The joint belief is b_k and the joint uncertainty is u eq. 3 eq. 4.
• Views are combined in turn, M = M^1 ⊕ M^2 ⊕ … ⊕ M^V eq. 5.
• The joint opinion is turned back into Dirichlet parameters: S = K / u, e_k = b_k S, alpha_k = e_k + 1 eq. 6.

3. Loss §3.3

• The conventional cross-entropy eq. 7 is replaced by its expectation under the Dirichlet, written with digamma functions eq. 8.
• A KL term to the uniform Dirichlet, Dir(1), penalises evidence for wrong classes. The parameters in the KL are alpha_tilde = y + (1 - y) * alpha, so the true-class evidence is not penalised eq. 9.
• The per-sample loss is the expected cross-entropy plus lambda_t times the KL term. lambda_t > 0 is a balance factor that is increased gradually from 0 towards 1 eq. 10 §3.3 app. A.6.
• The total loss sums that per-sample loss over all samples and over the fused opinion and each single view eq. 11.

4. Optimisation alg. 1 (app. A.2)

• Loop until convergence: compute each view's alpha^v, form each view's opinion, fuse them into the joint alpha, compute the loss in eq. 11, and take a gradient step on all networks alg. 1. The paper gives no convergence test.
• At test time, the joint belief masses and the uncertainty mass give the prediction and its uncertainty alg. 1 §3.2.

# One training step of trusted multi-view classification  [alg. 1]
# Input:  views: list of V tensors (B, D_v); labels: (B,) class ids
# Output: scalar loss to minimise

def training_step(views, labels, epoch):
    # Non-negative evidence per view and class  [§3.1] [§3.3]
    evidences = [net(x) for net, x in zip(view_nets, views)]
    # Dirichlet parameters per view  [§3.1]
    alphas = [evidence + 1 for evidence in evidences]
    # Fuse the view opinions into one joint Dirichlet  [§3.2] [eq. 5] [eq. 6]
    joint_alpha = fuse_opinions(alphas)
    # Sum the loss over each view and the joint opinion  [eq. 11]
    loss = 0
    for alpha in alphas + [joint_alpha]:
        loss = loss + sample_loss(labels, alpha, epoch)
    return loss

# Per-sample loss: cross-entropy plus KL term  [eq. 8] [eq. 9] [eq. 10]
# Input:  labels: (B,); alpha: (B, K); epoch: int
# Output: (B,) loss per sample

def sample_loss(labels, alpha, epoch):
    # One-hot targets  [§3.3]
    one_hot = one_hot_encode(labels, num_classes)
    # Dirichlet strength per sample  [§3.1]
    strength = sum over classes of alpha
    # Expected cross-entropy under Dir(alpha)  [eq. 8]
    cross_entropy = sum over classes of (
        one_hot * (digamma(strength) - digamma(alpha)))
    # Keep the true-class evidence, shrink the other classes  [§3.3]
    alpha_tilde = one_hot + (1 - one_hot) * alpha
    # Balance factor; it rises from 0 towards 1  [§3.3] [app. A.6]
    weight = min(1.0, epoch / annealing_epochs)
    return cross_entropy + weight * kl_to_uniform(alpha_tilde)

# Dempster's rule for two Dirichlet opinions  [eq. 2] [eq. 3] [eq. 4]
# Input:  alpha_1, alpha_2: (B, K); K is the number of classes
# Output: (B, K) Dirichlet parameters of the combined opinion

def fuse_two(alpha_1, alpha_2):
    # Belief masses and uncertainty of each opinion  [eq. 2]
    belief_1 = (alpha_1 - 1) / strength(alpha_1)
    belief_2 = (alpha_2 - 1) / strength(alpha_2)
    unc_1 = K / strength(alpha_1)
    unc_2 = K / strength(alpha_2)
    # Conflict: belief given to different classes  [eq. 4]
    conflict = (sum(belief_1) * sum(belief_2)
                - sum(belief_1 * belief_2))
    # Combined belief and uncertainty, renormalised  [eq. 4]
    belief = (belief_1 * belief_2 + belief_1 * unc_2
              + belief_2 * unc_1) / (1 - conflict)
    unc = unc_1 * unc_2 / (1 - conflict)
    # Back to Dirichlet parameters  [eq. 6]
    return belief * (K / unc) + 1

Key equations:

u^v + \sum_{k=1}^{K} b_k^v = 1 eq. 1

b_k^v = \frac{e_k^v}{S^v} = \frac{\alpha_k^v - 1}{S^v}, \qquad u^v = \frac{K}{S^v} eq. 2

b_k = \frac{1}{1-C}\left(b_k^1 b_k^2 + b_k^1 u^2 + b_k^2 u^1\right), \qquad u = \frac{1}{1-C} u^1 u^2, \qquad C = \sum_{i \neq j} b_i^1 b_j^2 eq. 4

\mathcal{L}_{ace}(\boldsymbol{\alpha}_i) = \sum_{j=1}^{K} y_{ij}\left(\psi(S_i) - \psi(\alpha_{ij})\right) eq. 8

\mathcal{L}(\boldsymbol{\alpha}_i) = \mathcal{L}_{ace}(\boldsymbol{\alpha}_i) + \lambda_t \, KL\!\left[D(\mathbf{p}_i \mid \tilde{\boldsymbol{\alpha}}_i) \,\|\, D(\mathbf{p}_i \mid \mathbf{1})\right] eq. 10

\mathcal{L}_{overall} = \sum_{i=1}^{N}\left[\mathcal{L}(\boldsymbol{\alpha}_i) + \sum_{v=1}^{V} \mathcal{L}(\boldsymbol{\alpha}_i^v)\right] eq. 11

Figures are described in words only. fig. 1 is the overview with the four steps (evidence, Dirichlet, opinion, fusion) and the combination of two belief sets. fig. 2 shows Dirichlet densities for a confident case, an out-of-distribution case, a high-uncertainty case, and the opinion on the simplex §3.1. fig. 3 plots accuracy against an uncertainty threshold §4.2. fig. 4 plots accuracy under increasing noise on half the views §4.2. fig. 5 shows uncertainty densities for in-distribution and noisy samples §4.2. fig. 6 is the end-to-end feature-fusion setup app. A.7.

Hyperparameters

Paper values are cited to the paper; the code column reports the released defaults, and the differences are listed in Paper vs code.

Results

Author-reported. Setup: six multi-view datasets app. A.3; each uncertainty baseline is reported with its best single view for both accuracy and AUROC §4.2. Values are mean ± standard deviation over 30 runs §4.1. Bold marks the best value per row in the paper.

Main comparison with single-view uncertainty methods tab. 1 §4.2:

The text reports gains of about 7.6% (PIE) and 14.8% (Scene15) in accuracy over the second-best method, which is EDL or MCDO §4.2.

Comparison with CCA-based methods, linear SVM on the learned representation, accuracy, 30 runs tab. 2 (app. A.4). Not bolded in the paper:

Removing one view, accuracy, 30 runs tab. 3 (app. A.5). "None" is all views:

End-to-end on multi-modal data, accuracy, 10 runs tab. 4 (app. A.7). "Best view" uses one backbone; "feature fusion" concatenates the two backbone outputs:

Uncertainty behaviour is described in words: higher-accuracy datasets have lower in-distribution uncertainty, and noisy out-of-distribution samples get much higher uncertainty on all datasets §4.2 fig. 5. Accuracy on the retained predictions rises as their uncertainty decreases fig. 3 §4.2. The noise curves fig. 4 are only in the figure and were not transcribed.

Paper vs code

Code: official repository Han-Zongbo/TMC at a3272b8746861c76a3461943b5eee51df5b5a8fe, folder TMC ICLR/. The repository's README links this paper README.md. The repository has no LICENSE file, so no code is quoted; citations are path and line only. The repository's ETMC TPAMI/ folder is the code of a different paper (arXiv 2204.11423) and is not used here.

Search: no code link in the paper text or in the arXiv abstract page. GitHub search for "trusted multi-view classification" and "TMC zongbo" returned Han-Zongbo/TMC, whose README cites this paper and gives the first author's address; the owner is treated as the authors' repository.

Discrepancies found:

• Evidence activation. The paper uses ReLU §3.3. The code applies Softplus after the last linear layer model.py. Softplus never gives exactly zero evidence, so the KL term drives evidence towards zero but not to it.
• Network. The paper says "fully connected networks" without depth or width §4.1. For Handwritten the code uses one linear layer per view, from the view's dimension to the class count, followed by Softplus, with no hidden layer model.py train.py.
• Balance factor. The paper says lambda_t "slowly increases" from 0 to 1 §3.3 app. A.6. The code uses min(1, epoch / annealing) with annealing = 50 model.py train.py, and the epoch starts at 1 train.py.
• Regularisation. The paper says an L2 trade-off of 0.0001 §4.1. The code uses Adam with weight_decay=1e-5 train.py. These are not the same quantity.
• Learning rate. The paper selects the learning rate by 5-fold CV from a grid §4.1. The code fixes 3e-4 as the default and has no CV code train.py.
• Loss reduction. The paper sums the loss over samples eq. 11. The code averages over the batch model.py. The scale changes the balance against the weight-decay term.
• Normalisation. The paper does not describe feature scaling. The code fits a min-max scaler separately on the training split and on the test split data.py data.py, so test features are scaled with their own minimum and maximum.
• Training length. The paper's loop runs "while not converged" alg. 1. The code runs a fixed 500 epochs with batch size 200 train.py train.py.
• Repeats. The paper reports 30 runs §4.1. The code sets no random seed, so repeated runs are not reproducible from the repository.
• Coverage. The paper reports six datasets and an end-to-end setting §4.1 app. A.6. The repository contains only handwritten_6views.mat and the Handwritten training script data.py. The other datasets and the end-to-end experiment are not in the repository.
• Unused loss. mse_loss is defined but not used in the training path model.py model.py. The paper does not mention it.
• Matches. The digamma cross-entropy eq. 8 matches model.py. The KL term eq. 9 matches model.py. Dempster's rule eq. 4 and the conversion back to alpha eq. 6 match model.py. The code folds the views pairwise, which is the same as the chained rule in eq. 5 model.py.

Pitfalls

• Conflict and numerics. The normalising factor 1/(1 - C) diverges when the conflict C approaches 1 eq. 4. The paper does not say how to handle this, and the code adds no epsilon or clamp model.py. Small combined uncertainty gives a large joint strength S = K/u eq. 6, which can overflow in float32.
• Notation typo. The text says alpha_i^v = e^i_i + 1 §3.3. The index should be e^v_i.
• Overclaim. The text says the method "outperforms other methods on all datasets" §4.2. Table 1 shows MCDO best on CUB AUROC and DE best on Caltech101 AUROC tab. 1. The "all datasets" claim holds for accuracy only.
• Inconsistent tables. The "Ours" values differ between the main table and the CCA table: CUB 91.00±0.42 vs 90.83±3.23, PIE 91.99±1.01 vs 91.91±0.11, HMDB 65.26±0.76 vs 65.26±2.52, Handwritten 98.51±0.15 vs 98.51±0.13 tab. 1 tab. 2. The means differ for CUB and PIE. Use the main table and flag this.
• Baseline protocol. Single-view baselines are scored at their best view, which is not the same protocol as the fused comparison that concatenates features §4.2 app. A.7. Compare like with like.
• Hyperparameters not stated. Batch size, epochs, network depth and width, the lambda_t schedule, and per-dataset learning rates are not in the paper. The code values above are defaults in a script, not the paper's settings.
• Noise levels. The noise levels for the multi-view comparison are not in the text, only in fig. 4. Use the code's noise procedure only after checking the script; no such script is in the repository.
• Numbering. Equation, table, figure and appendix numbers were counted from the LaTeX source in source order, not read from a rendered PDF. Float placement could change table and figure numbers in the compiled paper. Where unsure, cite the section.
• Environment. The code hard-codes CUDA: os.environ["CUDA_VISIBLE_DEVICES"] = "1" train.py and .cuda() calls model.py.
• Requirements. The README lists PyTorch 1.3.0 and Python 3 README.md. The code was not run for this card.

Lineage

• Builds on: evidential deep learning and subjective logic, which the paper uses to define the per-view opinions and the loss §1 §3.1 §3.3.
• Compared against: MC dropout (MCDO), deep ensembles (DE), uncertainty-aware attention (UA), evidential deep learning (EDL) §4.1; CCA, DCCA, DCCAE and BCCA app. A.4.
• Extended by: a TPAMI 2022 paper, "Trusted Multi-View Classification with Dynamic Evidential Fusion" (arXiv 2204.11423), named in the repository README README.md. It has no card in this wiki.