Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features

In plain words

Brain recordings taken while a person looks at pictures exist only for a limited set of object types, and collecting them is expensive. Pictures and written descriptions of objects are easy to find for almost any type. This paper asks whether a computer can still guess which object a person is looking at when that object type never appeared in the brain-recording training data.

The approach is to learn one shared internal representation from three kinds of data at once: the brain signal, a picture of the object, and an encyclopedia text describing it. Once the three are linked for the object types that have brain data, the model can use pictures and text of new object types to build the same kind of representation. At test time only the brain signal is fed in. It is mapped into the shared space, and a simple classifier picks the most likely object type from a known list of candidates. The authors also report that adding the text helps even though the people only saw pictures, which they read as a sign that language shapes how we see.

TL;DR

• Decodes which of a known list of candidate categories a viewer saw, from fMRI or EEG, including categories that have no brain data in training §I §III.A.
• Trains one multimodal VAE on brain, image and text features. Its joint posterior is a mixture of products of experts, plus a mutual-information term within each modality and a contrastive term across modalities §III.D §III.E.
• At test time only the brain encoder runs. An RBF SVM, trained on latents of the candidates' image and text features, classifies the brain latent §III.F fig. 4.
• Author-reported fMRI averages over 8 subjects: 22.43% top-1 and 60.33% top-5 with image and text (chance 2% and 10%), ahead of four VAE baselines tab. IV.
• The official code is MIT-licensed and matches most hyperparameters. It differs from the paper's equations in several places, listed under Paper vs code.

Problem

• Brain activity is available only for a few visual categories, while images and text exist for almost all categories. The setting is zero-shot neural decoding: seen classes have brain, image and text data; novel classes have image and text only, and their brain data is used only at test time §III.A.
• The novel classes are a known set of candidates, not arbitrary categories §III.A.
• Earlier identification methods rely on large numbers of paired stimulus-response recordings, which are hard to collect §II.
• Word vectors of class names are weakly aligned with visual content; Wikipedia articles carry more visual description and gave better results in earlier work §I.
• A multimodal VAE trained with the plain ELBO can suffer posterior collapse, where the joint latent ignores its inputs and the modalities are weakly coherent §III.D.

Method

Overview: (A) joint representation learning on seen data plus unlabeled novel image and text data; (B) an SVM trained on latents of novel image and text; (C) decoding brain latents with that SVM fig. 4.

Preprocessing §III.B

• fMRI. On DIR-Wiki, voxels are selected per region of interest by stability: the mean pairwise Pearson correlation across the five repetitions of each stimulus, keeping the top 15% §IV.B fig. 5 data_prepare_with_aug_DIR_Wiki.py. GOD-Wiki has one trial per stimulus, so no selection is applied §IV.B. Each voxel is normalised with mean and standard deviation from seen-class training data. PCA keeping 99% of the variance is fitted on seen-class training data only, and its weights are applied to the test samples §III.B data_prepare_with_aug_GOD_Wiki.py.
• Images. Intermediate feature maps of RepVGG-b3g4 (ImageNet top-1 80.21%, as stated in the paper) are flattened, normalised, and reduced by PCA to 99% variance §III.B.
• Text. ALBERT or GPT-Neo gives a sentence embedding as the mean of token embeddings. Long articles are split into windows of 256 tokens with 50 tokens of overlap; window embeddings are averaged, and so are articles of the same class §III.B.
• EEG (ThingsEEG-Text). 17 occipital and parietal channels, 70–400 ms after onset, down-sampled to 100 Hz fig. 6 §IV.A. Visual features are the top 100 principal components of CORnet-S features. Text is a BLIP caption embedded with CLIP §IV.A.

Encoders and decoders tab. III

Each modality has its own encoder and decoder. An encoder is an MLP with two hidden layers and ReLU, and outputs the mean and log-variance of a 32-dimensional Gaussian. The decoder mirrors the encoder and outputs the mean of the reconstruction.

# Modality encoder and decoder  [tab. III] [§IV.B]
# Input features: brain (B, d_b), image (B, 4996), text (B, 2560)

def encode(modality, features):
    # Two hidden layers of ReLU MLP units  [tab. III]
    hidden = mlp_relu(features, units=hidden_units[modality])
    # Mean and log-variance of a 32-dimensional Gaussian  [§IV.B]
    return linear_mu(hidden), linear_logvar(hidden)    # (B, 32) each

def decode(modality, z):
    # Reconstruction mean; the Gaussian scale is fixed at 0.75  [§III.D]
    hidden = mlp_relu(z, units=hidden_units[modality])  # [tab. III]
    return linear_out(hidden)                           # (B, d_m)

Joint posterior §III.D tab. I

For every non-empty subset S of the available modalities, the experts of S are combined as a product of Gaussians (a precision-weighted product). The joint posterior is then a mixture over the subsets, with equal weights. This is the mixture-of-products-of-experts (MoPoE) form, which the authors chose over the product-of-experts (PoE) and mixture-of-experts (MoE) forms tab. VII.

\bar q(z \mid \mathbb{X}) = \frac{1}{|\mathcal{P}(\mathbb{X})|} \sum_{\mathbb{X}_s \in \mathcal{P}(\mathbb{X})} \prod_{x_m \in \mathbb{X}_s} q_{\phi_m}(z \mid x_m) §III.D

# Joint posterior over the available modalities  [§III.D] [tab. I]
# inputs: dict modality -> features (B, d_m); output: mean, log-var (B, 32)

def joint_posterior(inputs):
    # One Gaussian expert per available modality  [§III.D]
    experts = {m: encode(m, x) for m, x in inputs.items()}
    # Product of experts inside every non-empty subset  [§III.D]
    subset_posteriors = []
    for subset in nonempty_subsets(sorted(inputs)):
        subset_posteriors.append(product_of_experts(experts, subset))
    # Equal-weight mixture over the subsets; the released code assigns
    # contiguous blocks of rows to subsets, see Pitfalls
    # [code: BraVL_fMRI/utils/utils.py#L45-L67]
    return block_mixture(subset_posteriors)

Objective terms §III.D §III.E

The ELBO with the joint posterior is the reconstruction term minus the KL of the mixture to the standard normal prior §III.D.

\mathcal{L}_M = \mathbb{E}_{q}\Big[\sum_{x_m\in\mathbb{X}} \log p_{\theta_m}(x_m\mid z)\Big] - D_{KL}\big[\bar q(z\mid\mathbb{X}) \,\|\,p(z)\big] §III.D

The intra-modality term lower-bounds the mutual information between the joint latent and each observed modality, using an auxiliary network Q_m on top of the decoder. The entropy term H(z) is constant.

I(z;x_m) \ge \mathbb{E}\big[\log Q_{\psi_m}(z\mid x_m)\big] + H(z) \mathcal{L}_{intra} = \sum_m \mathbb{E}\big[\log Q_{\psi_m}(z\mid x_m)\big] + \text{const} §III.E

The inter-modality term treats the observed tuple as a positive and tuples in which one or more modalities come from other samples as negatives. It rewrites the three pairwise mutual informations as a sum of 3 positive log-likelihoods minus log-sums over negatives §III.E. For the trimodal seen data there are six negative types; for novel data (image and text) there are two. The positive ELBO-type term is estimated with the ELBO and the negatives with the CUBO (chi-upper-bound) estimator with K = 30 samples §III.E §IV.B.

\mathrm{CUBO}_K = \mathbb{E}\Big[\log\sqrt{\tfrac{1}{K}\textstyle\sum_{k=1}^{K} \big(p(z_k,\mathbb{X})/q(z_k\mid\mathbb{X})\big)^2}\Big] §III.E

The full objective is maximised over the model parameters, with the auxiliary networks Q maximising the intra-modality bound first §III.F.

\mathcal{L} = \mathcal{L}_M + \lambda_1\mathcal{L}_{intra} + \lambda_2\mathcal{L}_{inter}, \qquad \lambda_1 = \lambda_2 = 0.001 §III.F §IV.B

Training alg. 1 §III.F §IV.B

Algorithm 1 is a two-stage update for each minibatch. Stage 1 tightens the intra-modality bound by updating only the Q networks. Stage 2 updates encoders and decoders with the full objective. Seen-class minibatches contain brain, image and text; novel minibatches contain image and text only. Extra image-text pairs and unimodal data are allowed for semi-supervised training §III.F §IV.A.

# One minibatch of seen-class data  [alg. 1] [§III.F]
# elbo_loss is the negative ELBO; mi_bound the intra-modality bound
# [code: BraVL_fMRI/run_epochs_trimodal.py#L249-L350]

def minibatch_loss(brain, image, text, epoch):
    # Stage 1 (Q networks only): maximise mi_bound  [§III.F]
    # Stage 2: minimise the loss below w.r.t. encoders and decoders
    # KL weight grows by 0.01 per epoch, capped at 1  [§IV.B]
    beta = min(1.0, 0.01 * epoch)
    elbo_loss = negative_elbo(brain, image, text, beta)
    loss = elbo_loss - lambda1 * mi_bound
    # Inter-modality term: positive tuple against shuffled negatives  [§III.E]
    # Six negative cases; their CUBO values are averaged  [§III.E]
    neg_cubo = mean([cubo(batch_k) for k in range(6)])   # K = 30  [§IV.B]
    loss = loss + lambda2 * (elbo_loss + neg_cubo)
    return loss

Decoding an unseen category §III.F fig. 4

Latents of the novel candidates' image and text are the training set for an RBF-kernel SVM, with the candidates' labels. A brain recording is then encoded with the brain encoder only, and the SVM ranks the candidates.

# Zero-shot decoding of novel categories  [§III.F] [fig. 4]
# novel_images, novel_texts: features of candidate categories (no brain)

def fit_decoder(novel_images, novel_texts, novel_labels):
    # Five stochastic latent samples per novel item
    # [code: BraVL_fMRI/run_epochs_trimodal.py#L813-L818]
    latents = [joint_sample(image=novel_images, text=novel_texts)
               for _ in range(5)]
    # RBF SVM, gamma 1e-5, C 1.0  [§IV.B] [code: run_epochs_trimodal.py#L851]
    return SVC(kernel="rbf", gamma=1e-5, C=1.0, probability=True).fit(
        concat(latents), concat_labels(novel_labels, 5))

def decode(svm, brain):
    # Brain-only latent: one sample from the brain encoder  [§III.F]
    z = joint_sample(brain=brain)
    # Class ranking; top-5 accuracy uses predict_proba  [§IV.C]
    return svm.predict_proba(z)

Hyperparameters

Results

Author-reported. Setup: fMRI on GOD-Wiki (5 subjects) and DIR-Wiki (3 subjects); novel-class test; top-1 and top-5 accuracy averaged over the 8 subjects; chance is 10% top-5 as stated in the text §IV.C, and 2% top-1 fig. 13. Significance from paired t-tests (p < 0.05) is marked in the paper's table.

Ablation of the MI regularisers, averaged across subjects tab. VI:

Joint posterior form tab. VII: PoE 31.42 / 66.87 (DIR top-1 / top-5) and 10.30 / 39.56 (GOD); MoE 35.71 / 72.33 and 10.97 / 48.17; MoPoE 36.50 / 72.39 and 13.39 / 53.54.

Text representation tab. V: class names versus Wikipedia articles. With ALBERT: class names 31.80 / 70.58 (DIR) and 11.49 / 45.47 (GOD); articles 36.11 / 74.69 and 13.09 / 57.04. With GPT-Neo: class names 31.41 / 69.67 and 10.77 / 36.69; articles 36.50 / 72.39 and 13.39 / 53.54.

Other findings the authors report:

• Adding text to vision raises average top-1 by about 6% for most test classes fig. 7 §IV.C. The authors read this as evidence that viewers call up linguistic representations, which then shape visual processing §IV.C.
• Extra unimodal or bimodal data improves decoding, and more pairs help fig. 10 fig. 11.
• Higher-level visual areas (LOC, FFA, PPA, IT) decode better than low-level areas (V1–V3); the whole visual cortex is best fig. 13 §IV.C.
• Synthetic brain latents from image and text resemble real ones in t-SNE, with exceptions fig. 12.
• Fine-tuning CORnet-S on ThingsEEG training images hurts (intra-subject V&T 50-way top-5 34.93 with fine-tuned visual features against 40.28 with the frozen ones) tab. VIII.

EEG (ThingsEEG-Text) tab. VIII, averaged over 10 subjects, with the frozen visual features:

The text gives a chance level of 10% for 50-way top-5 and 2.5% for 200-way top-5.

Paper vs code

The official repository is ChangdeDu/BraVL, linked from the paper. Its owner is the first author, and the repository is MIT-licensed (Copyright 2022 Changde Du) LICENSE. The commit is 233b09e5c0fbb84fb75371313433c99419398acf. The fMRI and EEG code is in separate folders. Most hyperparameters match: learning rate 1e-4, batch 512 (fMRI), latent 32, λ1 = λ2 = 0.001, K = 30, 15% stability selection on DIR-Wiki, 99% PCA, 256/512/2048 hidden units, and the EEG values (batch 1024, 256 units, 500 epochs, gamma 1e-3). Where the code and paper differ:

1. KL term. The paper takes the KL of the mixture to the prior §III.D. The code averages the per-subset KLs, sum over k of w_k KL(q_k ‖ p), which is an upper bound by convexity mm_div.py BaseMMVae.py BaseMMVae.py.
2. Mixture. The paper's mixture samples from the components. The code assigns contiguous row blocks of each batch to the subsets: floor(B × w_k) rows each, the last block takes the remainder utils.py BaseMMVae.py.
3. Prior expert. The code adds a standard-normal expert to the product only for the full set of three modalities BaseMMVae.py. The paper does not mention a prior expert.
4. Q networks. The paper describes Q_m as a network giving a distribution §III.E. The code uses only the mean head; the scale is fixed at 0.75, and the log-variance output is commented out QNET.py run_epochs_trimodal.py.
5. Likelihood. The paper does not name the likelihood family §III.D. The modalities are built with 'normal', so each decoder gives a Normal with mean x̂ and fixed standard deviation 0.75 experiment.py MLP_Brain.py MLP_Image.py MLP_Text.py. The likelihood_m* flags default to laplace but are never read flags.py.
6. Inter-modality term. The paper sums three positive terms and six negative terms with log-sums §III.E. The code adds one positive ELBO term and the mean of the six negative CUBO values, scaled by λ2 run_epochs_trimodal.py. The log/exp scaling in the negative term cancels, because m_cubo already averages over the batch and returns a scalar, so the negative term is the mean CUBO plus 1e6 × log(B/N), where B and N are the batch sizes. They are equal here, so the offset is zero run_epochs_trimodal.py run_epochs_trimodal.py.
7. Schedule. Algorithm 1 draws a minibatch from seen or novel data alg. 1. Each epoch in the code runs three passes: extra image-text data, then the novel-class image and text from the test split, then seen-class training data run_epochs_trimodal.py run_epochs_trimodal.py run_epochs_trimodal.py.
8. KL weight. The paper anneals the weight to 1 §IV.B. The code uses min(1, 0.01 × epoch), so with 100 epochs it reaches 0.99 only at the last epoch run_epochs_trimodal.py.
9. Learning rate. The scheduler has gamma 1.0, so the learning rate is constant experiment.py. The paper says nothing about a schedule.
10. SVM. The paper says gamma comes from 5-fold grid search on training data §IV.B. The code fixes gamma at 1e-5 (fMRI) and 1e-3 (EEG) and contains no search code run_epochs_trimodal.py run_epochs_trimodal.py.
11. Test latents. The code draws five stochastic latent samples per novel item for the SVM and one sample per brain recording. The paper does not mention sampling run_epochs_trimodal.py.
12. Missing code. The release contains neither the text-embedding scripts (ALBERT, GPT-Neo, CLIP, BLIP) nor the CORnet-S fine-tuning script. The loaders read precomputed .mat files from ./data/, which are not in the repository. The paper's footnotes link the datasets on figshare, OpenNeuro and OSF §IV.A.

Pitfalls

• Block mixtures depend on row order. At decoding time the whole novel test set goes through the model as one batch. The joint posterior then gives the first third of rows to the image-only expert, the next third to the text-only expert and the rest to the image-text expert run_epochs_trimodal.py utils.py. Which candidate's latent comes from which expert depends on the order of the test tensors, which the repository does not show. If rows are sorted by class, whole classes would share one expert. Before reusing the code, shuffle the rows or sample each subset's latents separately.
• Training minibatches get the same block assignment, but DataLoader shuffling makes it a random mixture in expectation. Each batch still has fixed counts per subset (for example 73 rows per subset for batch 512, with the last subset taking 74 rows).
• Novel images and texts are used in training. The code reads them from the test split with no labels and no brain data run_epochs_trimodal.py. This is transductive use of the candidate categories, which the paper's setup allows, but a reader should not assume that held-out categories are untouched.
• Decoding is a single stochastic sample. The brain encoder alone is run, and z is a random draw. Results vary with the seed, and the SVM is trained on five random draws per candidate. The paper does not report run-to-run variance, and the code sets only the global seed (2021) run_epochs_trimodal.py.
• Two stability-selection details matter. On DIR-Wiki the selection uses training trials only; the test voxels are then picked with the same indices data_prepare_with_aug_DIR_Wiki.py. Normalisation uses seen-class training statistics.
• Reported cost. Each seen minibatch runs about eight forward passes: one for stage 1, one for stage 2, and six for the negative cases with K = 30 CUBO samples each run_epochs_trimodal.py run_epochs_trimodal.py.
• The div_weight_* flags do nothing for the joint path. The joint divergence receives its own uniform weights, so the alpha values have no effect on joint_elbo runs main_trimodal.py BaseMMVae.py.
• Constant terms are dropped. The paper's +3H(z) and the Q entropy term do not affect gradients, so the code omits them. The fixed 0.75 scale is used in both the decoders and the Q networks, so changing it changes both the reconstruction term and the intra-modality bound.
• Large λ values hurt. The paper reports that large λ values degrade performance and that 0.001–0.01 is safe. Deeper MLPs overfit §IV.C. The authors tuned λ on one subject and applied it to the others §IV.B.
• Text and visual results are not reproducible from this repository alone. The text features and the fMRI data are not in the repository, so the results cannot be re-derived from the code.

Lineage

• Builds on: MoPoE (mixture of products of experts) §III.D; MVAE (product of experts) and MMVAE (mixture of experts) §III.D; the InfoGAN lemma for the intra-modality bound §III.E; the CUBO estimator §III.E. Mutual-information maximisation for representation learning §II.
• Compared against: CADA-VAE, MVAE, MMVAE, MoPoE-VAE tab. IV.
• Related work on Wikipedia-based class descriptions and word vectors for neural decoding: cited in §I and §II.
• Superseded by: not known from wiki content.