Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses

In plain words

Language models, translation systems and simple tagging tools all turn words into internal numbers. People usually describe those numbers with loose labels such as "syntactic", "semantic" or "low-level". This paper asks whether the set of such internal representations has a measurable shape, and it answers yes.

The method compares representations by how well one can be used to predict another. Each representation is squeezed through a small linear bottleneck, and then every pair is tested as a predictor of the other. Scores from those tests place all 100 representations on a shared map. On that map the representations form a trail that runs from plain word embeddings to the embedding of the next word, passing through the middle layers of language models.

The same map helps explain brain recordings. When people listen to stories while their brains are scanned, some representations predict the scans better than others. The paper finds that the map can match each representation to its pattern of brain fit. This suggests that the map picks up something real about how language is represented, though the authors state that this is a suggestion, not a proof.

TL;DR

• Builds a "language representation embedding" for 100 feature spaces by measuring how well each one's bottleneck latent predicts each other representation §3 app. F.
• The first principal dimension runs from word embeddings and early layers to the next-word embedding, and the second shows a middle-layer peak §4.1 fig. 2.
• The embedding predicts which feature spaces fit fMRI encoding models, with author-reported matching above 90% for every representation §4.2.4 fig. 4.
• Code is partial: the official repo has only the analysis notebook (GPL-3.0). The encoder and decoder training and the encoding-model code are not released.

Problem

Qualitative labels for representations are common, but the paper argues they assume an unstated low-dimensional structure §1. The authors want a quantitative map of how representations relate, so that the labels can be checked against it §1 §2.

The method needs a set of feature spaces that all come from the same stimuli, so that each can be paired with each other §3. It also needs one encoder and one decoder per pair, which for n representations means n² decoders §3.2. The paper uses n = 100, so 10,000 decoders §3.2 §4.1.

Method

Prose overview. Each representation t is a function from a stimulus to one feature vector per word §3. The method runs in three parts §3 fig. 1:

1. Encoders §3.1. Map a shared universal input U(S) to a 20-dimensional linear latent space, one encoder per representation. The paper uses GloVe word embeddings for U(S). The decoder half of the training network is then dropped, and only the encoder half is kept. Latent size 20 is chosen as small, and the authors report that size changes do not matter much §3.1.
2. Decoders §3.2. For each ordered pair of representations (source, target), train a linear decoder from the source's frozen latent space to the target. All n² pairs are trained, including source equal to target app. E.
3. Tournament and ranking §3.3 app. F. For each target t, compare the decoders that point at t on a held-out test story. For each ordered pair (i, j), count the test samples where decoder i has lower error than decoder j, and divide by the count where it does not. Use this ratio matrix W_t. Its principal eigenvector, normalised to sum to 1, scores each source's transfer to t. Store that score vector as one row of the embedding matrix R, and set the entry at the target's own position to 0.1 §3.3 fig. 2.

The embedding matrix R (n × n) has one row per target and one column per source fig. 2. The main diagonal is fixed to 0.1 after normalisation, so row sums are only approximately 1 §3.3 Representation_Embeddings_Tutorial.ipynb.

Mapping step §4.1.1. Rows of R are weighted so that each full model counts equally, then multidimensional scaling is applied §4.1.1 fig. 2. The paper does not name the distance; see Paper vs code. The paper reports the first two MDS dimensions and an eigenvalue scree plot fig. 2 §4.1.1.

Brain step §4.2. Ridge encoding models predict each voxel's fMRI response from each representation, with time delays §4.2.2. The vector of encoding-model correlations for each voxel is z-scored across the 100 representations §4.2.3. Each voxel is projected on the first MDS dimension §4.2.3 fig. 3.

Matching test §4.2.4. For each pair of representations (i, j), a linear regression is trained on the embedding rows of the other 98 representations, as a map from embedding row to correlation vector. Each held-out row is then used to predict the held-out correlation vectors. A discriminability score is positive when the correct pairs match better than the swapped pairs §4.2.4 fig. 4.

Equations. The paper states its equations inline or as unnumbered displays, so this card cites them by section: the transfer mapping and the encoder objective §3, the tournament ratio §3.3, the encoding model and its correlation criterion §4.2.2, and the discriminability score §4.2.4. The paper has no numbered equations to cite.

# Language representation embedding  [§3] [fig. 1]
# Input:  representations (n), universal inputs U(S), stimuli S
# Output: embedding matrix R (n x n)

def build_embedding(reps, U):
    # One bottlenecked linear encoder per representation  [§3.1]
    latents = {}
    for t in reps:
        encoder = train_linear_bottleneck(U, target=t, latent_dim=20)
        latents[t] = encoder(U)

    # One linear decoder per ordered pair, including same-task pairs  [§3.2] [app. E]
    errors = {}
    for source in reps:
        for target in reps:
            decoder = train_linear_decoder(latents[source], target)
            errors[source, target] = squared_error(decoder, held_out)

    # Tournament per target, then principal eigenvector  [§3.3] [app. F]
    R = zeros(n, n)
    for target in reps:
        wins = tournament_ratio(errors, target)      # (n, n) with zero diagonal
        scores = principal_eigenvector(wins)         # normalised to sum 1
        R[row_of(target)] = scores
        R[row_of(target), col_of(target)] = 0.1      # main diagonal  [§3.3]
    return R

Hyperparameters

Values stated in the paper are given with their location. Values that appear only in the code or are missing are marked.

Representation families app. A:

Results

Author-reported. Setup: 100 feature spaces, five subjects who listened to English podcast stories while being scanned with fMRI §4.2.1. Baseline: none reported, because the comparisons are between representations; the matching test is a two-way choice §4.2.4.

Paper vs code

The official repository is HuthLab/rep_structure, at commit 5502d90897d3534bbf2b7af8e27ecf8c9785f8d0 §4.1 [code]. Alexander Huth is an author of the paper. The repository is licensed GPL-3.0. Code is cited by path and line only; no code is quoted.

The repository contains one notebook, Representation_Embeddings_Tutorial.ipynb, plus a README and an MDS plotting helper. It loads precomputed per-sample error data from a file that is not in the repository (Figshare), and it covers only the analysis from the tournament onward to the plot Representation_Embeddings_Tutorial.ipynb Representation_Embeddings_Tutorial.ipynb. The encoder and decoder training, the encoding-model fits and the matching test are not in the repository, so their hyperparameters can only be checked against the paper.

Discrepancies:

• Scree plot. The paper says the scree plot comes from exploratory factor analysis of the embedding matrix §4.1.1. The notebook plots eigenvalues of a classical MDS on a correlation-distance matrix Representation_Embeddings_Tutorial.ipynb Representation_Embeddings_Tutorial.ipynb Representation_Embeddings_Tutorial.ipynb. These are different analyses, so the 22% and 10% figures may not match a re-run of the notebook.
• Tournament diagonal. The paper says the diagonal of the tournament matrix is set to 0 app. F. The notebook sets the whole column whose encoder is the target to 0.0001, not only the diagonal entry Representation_Embeddings_Tutorial.ipynb.
• Ratio orientation. The notebook counts the opposite comparison (decoder i worse than j) and then transposes Representation_Embeddings_Tutorial.ipynb Representation_Embeddings_Tutorial.ipynb. After the transpose, the matrix matches the paper's definition, so this is a trap for re-implementers rather than a bug.
• Ties. Equal errors fall in the "worse" count Representation_Embeddings_Tutorial.ipynb. The paper's ratio does not say how ties are handled.
• Eigenvector. The paper describes the principal eigenvector of W_t §3.3. The notebook normalises the matrix by column L1 norms and keeps the eigenvector whose eigenvalue is 1 to within 1e-8, discarding imaginary parts Representation_Embeddings_Tutorial.ipynb. It has no fallback when no such eigenvalue exists, so the index lookup would fail.
• MDS distance. The paper does not name the distance used in MDS §4.1.1. The notebook uses 1 minus the Pearson correlation between rows Representation_Embeddings_Tutorial.ipynb. It also flips the sign of the top three dimensions to match row 13 Representation_Embeddings_Tutorial.ipynb; this is a sign convention only.
• Model weighting. Consistent with the paper: the notebook repeats each row by a per-model weight so that every model counts equally Representation_Embeddings_Tutorial.ipynb fig. 2.
• Affinity normalisation and diagonal. Consistent with the paper: the eigenvector is renormalised to sum 1 and the diagonal is reset to 0.1 Representation_Embeddings_Tutorial.ipynb Representation_Embeddings_Tutorial.ipynb.
• Encoding-model code. Not in the repository. The notebook says Part 2, covering encoding models, is "being prepared" Representation_Embeddings_Tutorial.ipynb.

Pitfalls

• Ridge grid and loss are unstated. The α grid for the ridge regressions and the exact correlation criterion used during fitting are not stated §4.2.2.
• Early stopping is ambiguous. The paper caps training at 1000 batches and also stops at the first batch with no loss improvement app. E. Reading the second rule as patience of one is one option; the paper does not define a patience value.
• Learning-rate search set. The stated search includes 1e-4 twice in the set of five values app. E. The main text uses 1e-4 for encoders and 2e-5 for decoders §3.2. Implementers should use the main-text values.
• Two loss functions. The encoder-decoder pretraining uses a negative correlation coefficient loss, and the retrained decoders use MSE app. E. The main text (§3) does not mention the first loss, so an implementation following only the main text will differ.
• Decoder retraining. All decoders are retrained after the encoder stage, including same-task decoders, so that comparisons are consistent app. E. Skipping that retraining changes the tournament.
• Numerically sensitive ratio. The tournament entries are ratios of counts with a small epsilon in the denominator Representation_Embeddings_Tutorial.ipynb. When one side has zero count, the ratio is set by the epsilon and can be very large, and this feeds the eigenvector.
• Eigenvector shortcut. The eigenvector step depends on the normalisation choice and not only on the paper's description Representation_Embeddings_Tutorial.ipynb.
• Orientation of R. The paper's text says the vectors are stacked into R, while the figure and appendix caption say rows correspond to the target. The notebook puts each target's vector in its own row Representation_Embeddings_Tutorial.ipynb.
• Inconsistent first-dimension variance. The text gives 22% for the first dimension §4.1.1, while the brain figure says about 20% fig. 3.
• Choice of representations. The authors state that the embedding is only as good as the sampled set of 100 representations, and that unsampled parts of the space could change the picture §5.
• Universal input is an approximation. GloVe stands in for a universal input, and the authors say it is not truly universal §5.
• Linear limit. All encoders and decoders are linear, which the authors say may underestimate relatedness §5.
• Cost. About one week on 53 GPUs and 64 CPU servers, with n² = 10,000 decoders §4.1 §3.2.
• Temporal resolution. The authors attribute negative scores for GloVe next-word embedding to the coarse timing of BOLD responses, which makes it nearly identical to plain GloVe for brain data §4.2.4. This is the authors' explanation, not a tested result.
• Small sample. Five subjects, one test story, and the brain analysis depends on the specific stimuli §4.2.1 app. B.

Lineage

• Builds on: the encoder-decoder transfer method for vision tasks, from Zamir et al. (CVPR 2018), with the analytic hierarchy process for ranking §2 §3. No card in this wiki, so not linked.
• Compared against: none. The paper compares feature spaces to each other and to brain data, not to other embedding methods §2.
• Related encoding-model work the paper cites: Huth et al. 2016, Schrimpf et al. 2020, Jain & Huth 2018, and Jain et al. 2020 §2. No cards in this wiki, so not linked.