Trusted Multi-View Classification with Dynamic Evidential Fusion

In plain words

Many systems look at the same thing through several inputs, such as a depth camera and a colour camera, or several feature sets for one sample. The usual approach mixes the inputs together, so one broken input can quietly spoil the answer. This paper gives every input its own confidence score and lets the model say "I have little evidence here" when an input is unclear.

Each input network counts up evidence for each class. Those counts become a distribution over class probabilities, and that distribution also reports how much evidence was missing. The inputs are then merged with a rule from Dempster-Shafer evidence theory, so an input that is unsure has less effect on the final decision. The result is a prediction together with an uncertainty value for each sample. The view networks are trained jointly.

TL;DR

Use it when several inputs describe the same sample and some of them may be noisy, corrupted or out of distribution. Each view produces a Dirichlet distribution, and the views are combined with a reduced Dempster-Shafer rule, so the fused prediction carries an uncertainty value. The paper reports gains over single-view uncertainty methods, CCA baselines and concatenation-based fusion on several datasets. All numbers are author-reported.

Problem

Multi-view models usually assume that every view is equally reliable, or that a fixed weight per view is enough §1. In practice a view's reliability changes from sample to sample, for example when one sensor is abnormal or one text is uninformative §1. Softmax confidence from a single view tends to be overconfident, and methods that estimate uncertainty usually do so for one view at a time §1§3. The paper asks for a fusion step that uses each view's uncertainty and also yields uncertainty for the joint decision §1§3.

Method

The model has four parts. Each part is described in the paper's own sections.

1. Per-view evidence and Dirichlet parameters. Each view m has its own network. Its output is passed through a non-negative activation, giving evidence e^m, and the Dirichlet parameters are alpha^m = e^m + 1. The paper allows Softplus or ReLU; the code uses Softplus §3.1 TMC.py.

2. Subjective opinion per view. With Dirichlet strength S = sum(alpha), the belief mass of class k is (alpha_k - 1) / S and the uncertainty is K / S for K classes. The masses sum to one §3.2 eq. 10 eq. 11.

3. Fusion with the reduced Dempster rule. Two opinions are combined into one with the conflict term C and the scale 1 / (1 - C). The views are then combined pairwise into a joint opinion over all M views. The joint evidence is recovered from the joint uncertainty §3.3 eq. 13 eq. 14 eq. 15. The code fuses the depth and RGB opinions first, then the pseudo-view TMC.py.

4. Pseudo-view (optional, for ETMC). An extra view is built from the concatenated features of the original views. It gets its own Dirichlet and is fused in the same way §3.5. The paper says this adds interaction between views at the representation level §3.5.

Training. The loss has one variational term per view and one for the fused opinion, plus one for the pseudo-view when it is used. Each term is the expected log-likelihood under the Dirichlet, minus a KL term to the uniform Dirichlet. The KL weight is ramped up over training §3.1 §3.6 alg. 1. The loss is minimised by gradient descent alg. 1.

Inference. The prediction is the class with the largest fused parameter, and the fused uncertainty is K / S [code: ETMC TPAMI/train_ETMC_sunrgbd.py#L72-L74] train.py.

Key equations

The subjective opinion, the reduced combination rule and the mapping back to Dirichlet parameters §3.2 §3.3:

b_k = \frac{b^1_k b^2_k + b^1_k u^2 + b^2_k u^1}{1 - C}, \qquad u = \frac{u^1 u^2}{1 - C}, \qquad C = \sum_{i \neq j} b^1_i b^2_j eq. 13

b_k = \frac{e_k}{S} = \frac{\alpha_k - 1}{S}, \qquad u = \frac{K}{S} eq. 11

S = \frac{K}{u}, \qquad e_k = b_k S, \qquad \alpha_k = e_k + 1 eq. 15

The per-view objective, with the KL term on alpha_tilde = y + (1 - y) * alpha (the true-class parameter is reset to 1) §3.1 eq. 9:

\mathcal{L}(\mathbf{x}^m, \mathbf{y}) = \mathbb{E}_{q}\left[\log p(\mathbf{y} \mid \boldsymbol{\mu}^m)\right] - \lambda_t \, \mathrm{KL}\left[\mathrm{Dir}(\boldsymbol{\mu}^m \mid \tilde{\boldsymbol{\alpha}}^m) \,\|\, \mathrm{Dir}(\boldsymbol{\mu}^m \mid \mathbf{1})\right] eq. 9

The KL term has a closed form §3.1 eq. 8:

\mathrm{KL} = \log\frac{\Gamma(\sum_k \tilde{\alpha}_k)}{\Gamma(K)\prod_k \Gamma(\tilde{\alpha}_k)} + \sum_k (\tilde{\alpha}_k - 1)\left[\psi(\tilde{\alpha}_k) - \psi\left(\sum_j \tilde{\alpha}_j\right)\right] eq. 8

Pseudocode

The pseudocode follows the paper's Algorithm 1 for one training step with the fused opinion and the per-view terms alg. 1. It is written from the paper's equations and checked against the code. The pseudo-view and the dropout placement are left out.

# Trusted multi-view classification: one training step  [alg. 1] [§3.6]
# Inputs: views (list of M tensors, (B, d_m)); labels (B,); lambda_t
# Output: scalar loss to minimise (the negative of the objective in [eq. 9])

def training_step(views, labels, lambda_t):
    # Per-view Dirichlet parameters: softplus output plus one  [§3.1]
    alphas = [softplus(head(view)) + 1 for head, view in zip(heads, views)]
    # Per-view opinions from the Dirichlet parameters  [§3.2] [eq. 11]
    opinions = [to_opinion(alpha) for alpha in alphas]
    # Fuse the views one by one with the reduced Dempster rule  [§3.3]
    fused = opinions[0]
    for opinion in opinions[1:]:
        fused = combine(fused, opinion)
    # Dirichlet parameters of the joint opinion  [§3.3] [eq. 15]
    fused_alpha = to_alpha(fused)
    # Sum the per-view and fused terms, all with the same KL weight  [§3.6]
    loss = 0
    for alpha in alphas + [fused_alpha]:
        loss += fit_term(labels, alpha) + lambda_t * kl_term(labels, alpha)
    # The code averages over the batch instead of summing  [code]
    return mean(loss)

# Opinion from Dirichlet parameters  [§3.2] [eq. 11]
def to_opinion(alpha):
    strength = sum(alpha)  # (B, 1)
    belief = (alpha - 1) / strength  # (B, K)
    uncertainty = K / strength  # (B, 1)
    return belief, uncertainty

# Reduced Dempster combination of two opinions  [§3.3] [eq. 13]
def combine(opinion_a, opinion_b):
    belief_a, unc_a = opinion_a
    belief_b, unc_b = opinion_b
    # Mass that the two views assign to different classes  [eq. 13]
    conflict = sum(belief_a) * sum(belief_b) - sum(belief_a * belief_b)
    scale = 1 - conflict
    # Agreed belief, plus each view's belief weighted by the other's uncertainty
    belief = (belief_a * belief_b + belief_a * unc_b + belief_b * unc_a) / scale
    # Joint uncertainty is the product of the two uncertainties  [eq. 13]
    unc = unc_a * unc_b / scale
    return belief, unc

# Dirichlet parameters from a joint opinion  [§3.3] [eq. 15]
def to_alpha(opinion):
    belief, unc = opinion
    strength = K / unc
    return belief * strength + 1

# Negative expected log-likelihood under Dir(alpha)  [§3.1] [eq. 6]
def fit_term(labels, alpha):
    strength = sum(alpha)
    # One-hot labels select the digamma difference of the true class
    return sum(onehot(labels) * (digamma(strength) - digamma(alpha)))

# KL to the uniform Dirichlet, with the true-class parameter set to 1  [eq. 8]
def kl_term(labels, alpha):
    # Reset the true-class parameter to 1, keep the others  [§3.1] [eq. 7]
    alpha_tilde = onehot(labels) + (1 - onehot(labels)) * alpha
    total = sum(alpha_tilde)
    log_norm = lgamma(total) - lgamma(K) - sum(lgamma(alpha_tilde))
    spread = sum((alpha_tilde - 1) * (digamma(alpha_tilde) - digamma(total)))
    return log_norm + spread

The KL weight lambda_t is ramped up linearly. The paper states only that it is increased gradually, and the released code uses min(1, epoch / T) with T set by a command-line argument §3.1 TMC.py.

Hyperparameters

Paper values come from the text. Code values come from the released scripts. Entries marked "not stated" appear in neither the paper nor the code.

Results

Author-reported. All numbers are mean ± standard deviation over the runs stated in the text. The vector-type experiments use 30 runs and a 20% test split §4.1. RGB-D uses the fixed SUN RGB-D and NYUD2 splits §4.2. Image-text uses 5 runs §4.3.

Vector-type data, single-view baselines and fused models tab. 1. Baselines (MCDO, DE, UA, EDL) use their best single view. TMC and ETMC fuse all views. Scores are accuracy (ACC) and AUROC.

The text gives ETMC's accuracy gains over the second-best single-view model as about 9.4 and 18.7 points on PIE and Scene15 §4.1. The text calls these "%", but the arithmetic in tab. 1 shows percentage points.

Vector-type data, CCA-based baselines tab. 2. Each CCA method feeds a linear SVM.

RGB-D on SUN RGB-D (classification accuracy, %) tab. 3. ETMC has the best fusion score. TMC has the best RGB score and ETMC the best depth score.

RGB-D on NYUD Depth V2 (classification accuracy, %) tab. 4. The "Aug" row uses a backbone fine-tuned on SUN RGB-D.

The text reports that ETMC is 0.6 points above TMC on SUN RGB-D fusion §4.2. On NYUD2, TMC is best on depth alone §4.2 tab. 4.

Compute tab. 5. Both fused models are close to late fusion in cost.

Image-text, UMPC-FOOD101 (accuracy, %) tab. 6. "Best view" uses the better single modality. "Feature fusion" concatenates the two backbones' outputs, as the paper does for all baselines.

Uncertainty. On the vector datasets, the paper adds Gaussian noise to selected views and compares how the methods degrade §4.1 fig. 4. The fused models degrade less than the baselines. Out-of-distribution test samples (noise sigma = 1, 10, 100) get higher uncertainty than in-distribution ones, and higher noise usually gives higher uncertainty §4.1 fig. 5 fig. 6. Accuracy rises as the uncertainty threshold drops §4.1 fig. 7.

Decision by uncertainty (SUN RGB-D). Dropping samples whose uncertainty exceeds the largest belief reduces the misclassification rate from 0.387 to 0.187 §4.2 fig. 8 fig. 9. These are author-reported and use the paper's own threshold rule.

Sanity values from the text. For three classes, alpha = [41, 2, 2] gives a sharp distribution and alpha = [1.0001, 1.0001, 1.0001] gives uncertainty close to 1 §3.2 fig. 2. The paper's alpha = [6, 6, 6] example is described as "relatively low uncertainty" §3.2. From the definition in eq. 11, u = 3/45 ≈ 0.067, u ≈ 1 and u = 3/18 ≈ 0.167. These are derived from the paper's own equations, not stated there.

Paper vs code

The code is the official repository linked in the paper's introduction (footnote) §1. Its README names the first author's email, which supports ownership by the authors README.md. The repository has no LICENSE file, so no code is quoted here. It contains the image experiments (ETMC TPAMI/) and a Handwritten demo for the vector case (TMC ICLR/). The card follows the paper version fetched by wikillm fetch (arXiv v3 per the raw metadata).

• KL weight schedule. The paper says lambda_t is increased gradually §3.1. The code uses min(1, epoch / annealing), with annealing_epoch = 10 for RGB-D and 50 for the vector demo TMC.py train_ETMC_sunrgbd.py train.py. For RGB-D, epoch starts at 0, so the KL weight is zero in the first epoch train_ETMC_sunrgbd.py.
• Weight decay. The paper says the l2 trade-off is 0.0001 §4.1. Both code paths use Adam with weight_decay=1e-5 train_ETMC_sunrgbd.py train.py.
• Pseudo-view for vector data. The paper concatenates the original feature vectors to form the pseudo-view §4.1. The vector demo has no pseudo-view at all model.py. For RGB-D the concatenation of backbone outputs matches the paper §4.2 TMC.py.
• Loss form. The paper writes sums over samples §3.6. The code averages over the batch TMC.py model.py. RGB-D sums four terms with equal weight: depth, RGB, pseudo-view and fused train_ETMC_sunrgbd.py. This matches the paper's overall loss §3.6.
• Sign. Algorithm 1 says "Maximize Eq. (loss)" alg. 1. The code minimises the negative form, A + B with A the cross-entropy part and B the KL term TMC.py. The two are consistent only if the paper's objective is read as negated.
• Activation. The paper allows Softplus or ReLU §3.1. The code uses Softplus TMC.py.
• Model selection on the test set (RGB-D). The paper does not mention a validation set §4.2. The released RGB-D scripts use the test loader for the LR scheduler and for choosing the best checkpoint, and then report the test metrics on that same loader train_ETMC_sunrgbd.py train_ETMC_sunrgbd.py train_ETMC_sunrgbd.py.
• Vector evaluation protocol. The paper uses 5-fold CV, a 20% test split and 30 runs §4.1. The demo trains for a fixed number of epochs and reports the final-epoch test accuracy without a validation split train.py.
• Hidden layers. The paper says "fully connected networks" §4.1. The code default for RGB-D is no hidden layer, one linear map per view train_ETMC_sunrgbd.py TMC.py. The vector demo's [240] dims also give a single linear map train.py model.py.
• Image encoder comment. The comment in models/image.py says 2048-dimensional features image.py. The code uses ResNet-18, which gives 512 features, and img_hidden_sz = 512 train_ETMC_sunrgbd.py.
• Not in the code. Food101, the five other vector datasets, the CCA baselines and the uncertainty experiments are not in the repository.
• Matches. Class counts (19 SUN RGB-D, 10 NYUD2), ResNet-18 pretraining, resize 256 and crop 224, Adam, and the NYUD2 10-class reorganisation match the paper §4.2 train_ETMC_nyud2.py.

Pitfalls

• Test-set model selection. The RGB-D numbers may be optimistic: the best checkpoint and the LR schedule are both chosen on the test set in the released code (see Paper vs code). Re-run with a separate validation split before comparing with other work.
• Undocumented KL schedule. The paper gives no ramp length. The code's ramp uses the epoch index, so the KL weight starts at 0 and reaches 1 after annealing_epoch epochs. A different ramp changes early training.
• Gradient accumulation across epochs. The code clears gradients at the start of each epoch train_ETMC_sunrgbd.py, but it decides when to step from a global batch counter train_ETMC_sunrgbd.py. If the number of batches per epoch is not a multiple of the accumulation count, the last few batches of an epoch are dropped and the first step of the next epoch uses fewer batches than the others.
• Batch-level scale. The paper sums over samples §3.6. The code averages over the batch and also divides by the accumulation count train_ETMC_sunrgbd.py. This changes the scale of the loss relative to weight decay. The paper does not state how these interact.
• Division by 1 − C. Both code paths divide by 1 − C with no epsilon TMC.py TMC.py. Full conflict (C = 1) gives a division by zero. The paper does not discuss this case.
• Hardcoded GPU. The KL function creates beta with .cuda() [code: ETMC TPAMI/models/TMC.py#L9] model.py. Running on CPU needs a change here.
• Unstated regularisation. The code uses dropout 0.1 in the view heads TMC.py. The paper does not mention dropout.
• Table inconsistencies. The same TMC results differ between tables: CUB ACC is 91.00±0.42 in tab. 1 and 90.83±3.23 in tab. 2; PIE ACC is 91.99±1.01 in tab. 1 and 91.91±0.11 in tab. 2; HMDB ACC std is 0.76 in tab. 1 and 2.52 in tab. 2; Handwritten std is 0.15 and 0.13. Use tab. 1 with caution.
• Zero standard deviations. ETMC on Handwritten reports 98.75±0.00 tab. 1, and CCA on Caltech101 reports 90.50±0.00 tab. 2. Identical accuracy in all 30 runs is possible but unusual. Check the logs before relying on these.
• Section cross-references. The introduction points to Sec. 3.3 for the propositions and Sec. 3.4 for the pseudo-view §1. In the compiled numbering the propositions are in §3.4 and the pseudo-view is in §3.5.
• Propositions. They analyse one combination step for fixed opinions. They do not analyse training, and Prop. 4 states only a qualitative correlation. The main-text and appendix proofs differ in their intermediate steps §3.4 app.: Proof Details.
• Uncertainty-threshold claims. The subjective confusion-matrix result depends on the paper's rule "uncertain when u exceeds the largest belief" §4.2. It is not a standard calibration metric.
• Image-text results. The Food101 result has no released code and uses ResNet-152 and BERT §4.3. Reproduction depends on details the paper does not state.
• Transfer to other domains. Results are on six vector datasets, two RGB-D datasets and one image-text dataset. The paper's claims about other domains are not tested here.

Lineage

• Builds on: evidential deep learning and subjective logic §2 §3.2. These are named by the paper and not compiled here.
• Conference version: the ICLR 2021 "Trusted Multi-View Classification" §1. The code README links arXiv:2102.02051 for it README.md.
• Compared against: MCDO §4.1, deep ensembles §4.1, uncertainty-aware attention §4.1, EDL §4.1, CCA family §4.1, late fusion and TRecg §4.2.
• Superseded by: not stated.