Intrinsic dimension

Overview

Intrinsic dimension is a measure of how many free parameters a fine-tuning task really needs. Full fine-tuning updates every weight of a pretrained model. The measure asks instead how small a trainable subspace can get while still doing most of the job. The one card so far applies this measure to language-model fine-tuning 2012.13255 §1 2012.13255 §3.

The method keeps all D pretrained weights frozen. It trains a vector \theta^{d} of d numbers and maps it into weight space through a fixed random projection P 2012.13255 eq. 1. The measured value is d_{90}: the smallest d that reaches 90% of the score of standard full fine-tuning 2012.13255 §3 2012.13255 §4.1. The authors report that for RoBERTa-Large on MRPC, about 200 to 300 numbers reach that target 2012.13255 §4.2 2012.13255 tab. 1. The two variants give 207 (SAID) and 322 (DID) for that cell.

The paper treats d_{90} as an upper bound on the true intrinsic dimension, because random projections make the estimate crude 2012.13255 §4.2.

Variants

Direct intrinsic dimension (DID)

The projection acts on all weights at once, with no knowledge of the layer structure 2012.13255 §3. The projection is the Fastfood construction M = HG\Pi HB, which can be applied in \mathcal{O}(D \log d) 2012.13255 eq. 2 2012.13255 §3. On MRPC, DID reaches 90% of full fine-tuning with 322 trainable numbers for RoBERTa-Large and 1861 for BERT-Base 2012.13255 tab. 1. Card: Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning.

Structure-aware intrinsic dimension (SAID)

The paper assumes that the layers of an attention model specialise. SAID spends m of the d trainable values, where m is the number of layers, on one learned scale \lambda_i per layer. The other d - m values feed the same projection as before 2012.13255 §3 2012.13255 eq. 3. The paper's own table gives 207 numbers for RoBERTa-Large on MRPC, against 322 for DID 2012.13255 tab. 1. SAID is the paper's new contribution; DID is the variant it builds on 2012.13255 §3.

The paper says SAID improves consistently over DID 2012.13255 §4.2. Table 1 does not fully support that wording, since the two tie on RoBERTa-Large for QQP at 774 numbers 2012.13255 tab. 1. See the Contradictions section.

When to use which

The paper's experiments cover sentence-pair classification on GLUE-style tasks (MRPC and QQP) with BERT and RoBERTa 2012.13255 §4.1. Choose by the constraint that binds you.

For the pre-training question, the paper reports that d_{90} falls as pre-training continues 2012.13255 §5.1 2012.13255 fig. 2, and that it falls as the parameter count grows across public models 2012.13255 §5.2 2012.13255 fig. 3. Both results are qualitative in the card, because the figures are described in words only.

For generalization, the paper reports that lower d_{90} goes with higher evaluation accuracy and a smaller relative gap 2012.13255 §5.3 2012.13255 fig. 4 2012.13255 fig. 5. This is a correlation, not a causal claim. Its bound, \mathcal{O}(\sqrt{d/m}), is asymptotic, has no stated constants, and applies only to models trained inside the subspace 2012.13255 §5.3.1 2012.13255 eq. 5.

Implementation sketch

The protocol is a search over d, with one full training run per candidate. The subspace step keeps the pretrained weights fixed and trains only \theta^{d} 2012.13255 §3 2012.13255 eq. 2.

# Subspace fine-tuning run for one value of d  [arxiv:2012.13255 §3] [eq. 2]
# Input:  theta_0 (D,) pretrained weights; d; seed; train_data
# Output: trained theta_d (d,)

def subspace_finetune(theta_0, d, seed, train_data):
    # Draw the fixed random factors once per seed; never train them  [§3]
    G, B, Pi = draw_fastfood_factors(D, seed)
    # Start at zero, which is the pretrained model  [§4.1]
    theta_d = zeros(d)
    for batch in train_data:
        # Map the d trainable values into weight space  [eq. 2]
        theta_D = theta_0 + fastfood(theta_d, G, Pi, B)
        loss = model_loss(theta_D, batch)
        theta_d = sgd_step(theta_d, grad(loss, theta_d))
    return theta_d

# d90 search: smallest d reaching 90% of full fine-tuning  [§3] [§4.1]
def find_d90(full_score, train_and_eval, low, high):
    target = 0.9 * full_score
    while low < high:
        mid = (low + high) // 2
        if train_and_eval(mid) >= target:
            high = mid
        else:
            low = mid + 1
    return low

Where the variants differ: SAID splits \theta^{d} into d - m projected values and m per-layer scales, and multiplies each layer's projected part by its scale 2012.13255 eq. 3. The paper's search is over a log-spaced sweep of d in §4.1 and a binary search in §5.1 2012.13255 §4.1 2012.13255 §5.1. The sketch uses a binary search, which assumes the score rises with d. The card does not confirm that monotonicity.

Contradictions

• Text and table differ slightly. The text says "roughly 200" numbers for RoBERTa-Large on MRPC and "800" for QQP 2012.13255 §4.2. Table 1 gives 207 and 774 2012.13255 tab. 1. Both come from the same paper.
• SAID versus DID. The text says SAID improves consistently 2012.13255 §4.2. Table 1 has SAID and DID tied at 774 for RoBERTa-Large on QQP 2012.13255 tab. 1.

Only one card covers this concept, so no cross-paper disagreement is known.

Papers

• Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning: introduces the d_{90} measure for language models, the Fastfood and SAID projections, and the pre-training and generalization experiments.