--- kind: concept slug: intrinsic-dimension title: Intrinsic dimension summary: >- Measures how many free parameters a fine-tuning task needs by training only d numbers pushed through a fixed random projection onto frozen pretrained weights. The smallest d reaching 90% of full fine-tuning is the task's intrinsic dimension; for RoBERTa-Large on MRPC it is 207 to 322 numbers. Values are upper bounds and depend on the metric and the projection. tags: [fine-tuning, intrinsic-dimension, pretraining, generalization] papers: ["2012.13255"] related: [parameter-efficient-fine-tuning, low-rank-adaptation, language-model-pretraining, implicit-bias-of-gradient-descent] status: draft updated_at: 2026-10-19 --- ## Overview Intrinsic dimension is a measure of how many free parameters a fine-tuning task really needs. Full fine-tuning updates every weight of a pretrained model. The measure asks instead how small a trainable subspace can get while still doing most of the job. The one card so far applies this measure to language-model fine-tuning [arxiv:2012.13255 §1] [arxiv:2012.13255 §3]. The method keeps all $D$ pretrained weights frozen. It trains a vector $\theta^{d}$ of $d$ numbers and maps it into weight space through a fixed random projection $P$ [arxiv:2012.13255 eq. 1]. The measured value is $d_{90}$: the smallest $d$ that reaches 90% of the score of standard full fine-tuning [arxiv:2012.13255 §3] [arxiv:2012.13255 §4.1]. The authors report that for RoBERTa-Large on MRPC, about 200 to 300 numbers reach that target [arxiv:2012.13255 §4.2] [arxiv:2012.13255 tab. 1]. The two variants give 207 (SAID) and 322 (DID) for that cell. The paper treats $d_{90}$ as an upper bound on the true intrinsic dimension, because random projections make the estimate crude [arxiv:2012.13255 §4.2]. ## Variants ### Direct intrinsic dimension (DID) The projection acts on all weights at once, with no knowledge of the layer structure [arxiv:2012.13255 §3]. The projection is the Fastfood construction $M = HG\Pi HB$, which can be applied in $\mathcal{O}(D \log d)$ [arxiv:2012.13255 eq. 2] [arxiv:2012.13255 §3]. On MRPC, DID reaches 90% of full fine-tuning with 322 trainable numbers for RoBERTa-Large and 1861 for BERT-Base [arxiv:2012.13255 tab. 1]. Card: [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](/papers/2012.13255). ### Structure-aware intrinsic dimension (SAID) The paper assumes that the layers of an attention model specialise. SAID spends $m$ of the $d$ trainable values, where $m$ is the number of layers, on one learned scale $\lambda_i$ per layer. The other $d - m$ values feed the same projection as before [arxiv:2012.13255 §3] [arxiv:2012.13255 eq. 3]. The paper's own table gives 207 numbers for RoBERTa-Large on MRPC, against 322 for DID [arxiv:2012.13255 tab. 1]. SAID is the paper's new contribution; DID is the variant it builds on [arxiv:2012.13255 §3]. The paper says SAID improves consistently over DID [arxiv:2012.13255 §4.2]. Table 1 does not fully support that wording, since the two tie on RoBERTa-Large for QQP at 774 numbers [arxiv:2012.13255 tab. 1]. See the Contradictions section. ## When to use which The paper's experiments cover sentence-pair classification on GLUE-style tasks (MRPC and QQP) with BERT and RoBERTa [arxiv:2012.13255 §4.1]. Choose by the constraint that binds you. | Situation | Use | Why | |---|---|---| | You want to know how much trainable capacity a task needs on a fixed pretrained model | The $d_{90}$ protocol with SAID | This is the paper's question. It needs about 100 subspace runs per dataset and model [arxiv:2012.13255 §4.1]. | | Your model is small and the task is a sentence-pair dataset like MRPC | SAID, then DID as a check | Table 1 gives the most detail for this setting [arxiv:2012.13255 tab. 1]. | | You must store a projection matrix explicitly | Not feasible at scale; use Fastfood | The authors estimate 1.42 TB for a dense projection of RoBERTa-Large at $d=1000$ [arxiv:2012.13255 §3]. | | You need a deployable adapter that is trained and served | This paper does not evaluate one | It compares only DID against full fine-tuning and reports no other method [arxiv:2012.13255 §4]. See [parameter-efficient-fine-tuning](/concepts/parameter-efficient-fine-tuning) and [low-rank-adaptation](/concepts/low-rank-adaptation) for other cards. | | You need training speed or memory numbers | Not supported by this card | The card reports $d_{90}$ and accuracy only. Training cost is not stated, so do not cite savings from this paper. | | You are comparing $d$ across models | Only with the same metric and the same search | $d_{90}$ depends on the metric, the learning-rate search and the full fine-tuning baseline [arxiv:2012.13255 §4.1] [arxiv:2012.13255 §5.1]. | For the pre-training question, the paper reports that $d_{90}$ falls as pre-training continues [arxiv:2012.13255 §5.1] [arxiv:2012.13255 fig. 2], and that it falls as the parameter count grows across public models [arxiv:2012.13255 §5.2] [arxiv:2012.13255 fig. 3]. Both results are qualitative in the card, because the figures are described in words only. For generalization, the paper reports that lower $d_{90}$ goes with higher evaluation accuracy and a smaller relative gap [arxiv:2012.13255 §5.3] [arxiv:2012.13255 fig. 4] [arxiv:2012.13255 fig. 5]. This is a correlation, not a causal claim. Its bound, $\mathcal{O}(\sqrt{d/m})$, is asymptotic, has no stated constants, and applies only to models trained inside the subspace [arxiv:2012.13255 §5.3.1] [arxiv:2012.13255 eq. 5]. ## Implementation sketch The protocol is a search over $d$, with one full training run per candidate. The subspace step keeps the pretrained weights fixed and trains only $\theta^{d}$ [arxiv:2012.13255 §3] [arxiv:2012.13255 eq. 2]. ```pseudocode # Subspace fine-tuning run for one value of d [arxiv:2012.13255 §3] [eq. 2] # Input: theta_0 (D,) pretrained weights; d; seed; train_data # Output: trained theta_d (d,) def subspace_finetune(theta_0, d, seed, train_data): # Draw the fixed random factors once per seed; never train them [§3] G, B, Pi = draw_fastfood_factors(D, seed) # Start at zero, which is the pretrained model [§4.1] theta_d = zeros(d) for batch in train_data: # Map the d trainable values into weight space [eq. 2] theta_D = theta_0 + fastfood(theta_d, G, Pi, B) loss = model_loss(theta_D, batch) theta_d = sgd_step(theta_d, grad(loss, theta_d)) return theta_d # d90 search: smallest d reaching 90% of full fine-tuning [§3] [§4.1] def find_d90(full_score, train_and_eval, low, high): target = 0.9 * full_score while low < high: mid = (low + high) // 2 if train_and_eval(mid) >= target: high = mid else: low = mid + 1 return low ``` Where the variants differ: SAID splits $\theta^{d}$ into $d - m$ projected values and $m$ per-layer scales, and multiplies each layer's projected part by its scale [arxiv:2012.13255 eq. 3]. The paper's search is over a log-spaced sweep of $d$ in §4.1 and a binary search in §5.1 [arxiv:2012.13255 §4.1] [arxiv:2012.13255 §5.1]. The sketch uses a binary search, which assumes the score rises with $d$. The card does not confirm that monotonicity. ## Contradictions - **Text and table differ slightly.** The text says "roughly 200" numbers for RoBERTa-Large on MRPC and "800" for QQP [arxiv:2012.13255 §4.2]. Table 1 gives 207 and 774 [arxiv:2012.13255 tab. 1]. Both come from the same paper. - **SAID versus DID.** The text says SAID improves consistently [arxiv:2012.13255 §4.2]. Table 1 has SAID and DID tied at 774 for RoBERTa-Large on QQP [arxiv:2012.13255 tab. 1]. Only one card covers this concept, so no cross-paper disagreement is known. ## Papers - [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](/papers/2012.13255): introduces the $d_{90}$ measure for language models, the Fastfood and SAID projections, and the pre-training and generalization experiments.