Foundation model

Overview

A foundation model is a model trained on broad data, usually with self-supervision at scale, that can be adapted to many downstream tasks. BERT, GPT-3 and CLIP are the report's examples 2108.07258 §1. Its building blocks, deep networks and self-supervised learning, are not new. Scale, the Transformer architecture and more data are what changed 2108.07258 §1. The pattern is transfer learning plus scale: pretrain on a surrogate task, then adapt to the task you care about 2108.07258 §1.

The report's two central themes are emergence and homogenization. The same base model gives leverage across many applications, and it also passes its defects on to every model adapted from it 2108.07258 §1.1; fig. 1.

Formal view of adaptation. Adaptation is a constrained optimization: the adapted parameters minimize the adaptation loss, given the pretrained parameters, under an optional complexity constraint. The report says current learning theory does not fully explain the gap between pretraining and adaptation 2108.07258 §4.10.

Three of the cited papers are surveys: the report that defines the term 2108.07258, a survey of hallucination in these models 2309.05922, and a survey of foundation models in robotic manipulation 2402.02385. The other eight each take one model family and report how it is used on one kind of input: promptable segmentation of medical images, image-text pretraining for aerial images, reprogramming a frozen language model for time-series forecasting, per-pixel embeddings of satellite time series, monocular depth and surface normals, pretrained vision features for off-road vehicle dynamics and terrain traversability, and pretrained video models for summarising robot footage. Their numbers are author-reported and come from those inputs only.

Variants

The variants below are ways of using a pretrained foundation model. The first two come from the report. The robotics survey supplies the planner-and-controller variant. The rest come from the model papers. None of the model papers is a general method, so each variant is described as that paper used it. The hallucination survey adds no variant; it is covered under Risks.

Low-storage adaptation

Freeze most pretrained parameters and learn a small set of task-specific ones. The report lists tuning only the last layer, tuning bias vectors, low-rank residuals, masks, soft prompts and inserted modules 2108.07258 §4.3. It notes that some of these methods trade compute for memory 2108.07258 §4.3. Card: On the Opportunities and Risks of Foundation Models.

Prompt-based adaptation

The report cites a study in which a well-tuned prompt can be worth about 100 training examples 2108.07258 §4.3. Soft prompts are one of the low-storage options above.

Zero-shot prompting of a promptable model

Use the released model as it is and steer it with a per-query input, such as a click or a box. Segment Anything (SAM) was evaluated this way on 19 medical imaging datasets, 28 tasks, with no training 2304.10517 §2.1. With one box per part of the target, the mean IoU was 0.6542 2304.10517 §3.1. Single-prompt IoU ranged from 0.1136 to 0.9118 across tasks [arxiv:2304.10517 §3.1]. With one point, SAM averaged 0.4595 against 0.2240 for the best of three interactive baselines 2304.10517 §3.2. Card: Segment Anything Model for Medical Image Analysis.

Continued pretraining on in-domain image-text data

Keep training a contrastive image-text model on data from a new domain. The RemoteCLIP authors started from CLIP and trained on 165,745 aerial images with five captions each 2306.11029 §4.1.2. The captions were made from detection boxes and segmentation masks, so the set was about 12 times larger than the existing caption-only sets 2306.11029 §3.2. On 12 classification datasets, zero-shot average accuracy rose over CLIP by 6.39 points for ViT-B-32 and by 5.63 for ViT-L-14 2306.11029 tab. 3. Card: RemoteCLIP.

Frozen language model with trained input and output layers

Keep a language model's weights frozen and train only the layers around it. Time-LLM uses a frozen Llama-7B for forecasting. A small patch-to-text cross-attention layer maps series patches onto learned text prototypes, a text prompt is prepended, and one linear layer maps the outputs to the forecast horizon 2310.01728 §3.1. The paper reports fewer than 6.6 million trainable parameters 2310.01728 §4.5. On the long-term benchmarks, author-reported MSE is best on seven of eight datasets. On ETTh1, averaged over four horizons, MSE is 0.408 against 0.465 for GPT4TS [arxiv:2310.01728 §4.1]. With the first 10% of training steps, ETTh1 MSE is 0.556 against 0.590 for GPT4TS 2310.01728 §4.3. The 32-layer backbone measured 32,136 MiB of GPU memory at ETTh1 with a 96-step horizon 2310.01728 §4.5. Card: Time-LLM.

Frozen embeddings with a small head

Pretrain once, release the frozen embeddings, and train only a small head per task. TESSERA produces a 128-value embedding per 10 m pixel per year from Sentinel-1 and Sentinel-2 time series 2506.20380 §1. Its evaluation uses a two-layer MLP for pixel-wise tasks and a U-Net for patch-level tasks 2506.20380 §4.1. Card: TESSERA.

Foundation model as robot planner and controller

Use a language or vision-language model in the robot's control stack, either to plan the steps of a task or to supply features for a low-level policy. The survey splits high-level planning by plan form: policy code that calls control primitives, free-text plans, and PDDL that an external planner solves 2402.02385 §2.1. It adds visual grounding, extra knowledge and environment feedback as assistants to planning 2402.02385 §2.2. For low-level control it groups work into policy learning, environment modelling and representation learning 2402.02385 §3, and rates the approaches in its table on reward and data needs 2402.02385 tab. 1. The ratings are qualitative. The survey reports no experiments, so no numbers from it are transferable to a specific robot. Card: A Survey on Robotics with Foundation Models.

Foundation-model features inside a robot's own model

Feed features from a pretrained vision model into a model the robot runs itself, and learn only the part that uses them. In the vehicle dynamics paper, DINOv2 ViT-S/14 features are reduced by PCA to 40 values per location. A small encoder compresses each wheel's values, and an LSTM corrects a physics model with the result. The encoder is trained end to end with the dynamics model 2412.00581 §3.2.1; §5; fig. 1. In SALON, each pixel's feature becomes a set of distances to k-means cluster centres. Those features are projected into a bird's-eye-view map, paired with a roughness value from the vehicle's vibration, and fed to a Gaussian process that is updated online 2412.07826 §3.1; §3.2; §3.4. The vision model is DINOv2 ViT-B on the ATV and RADIOv2.5 ViT-B on the wheelchair 2412.07826 §4.1.1; §4.1.2.

With hindsight features, which need sensor data the robot does not have at runtime, the dynamics model's mean summed loss is about 10% below the no-feature baseline 2412.00581 §6.1. With features from realistic projection distances, the model beats the baseline only within 0 to 30 m 2412.00581 §6.2. On a figure-8 course of five laps with one ATV, SALON at R_max 0.2 had 3 interventions, against 4 for Velociraptor. Its average speed was 3.60 m/s against 4.82 m/s 2412.07826 tab. 4. The authors also state that its navigation is comparable to methods trained on 100 to 1000 times more data. The card does not check that claim 2412.07826 §1. Cards: Dynamics Modeling using Visual Terrain Features; SALON.

Query-driven use of pretrained video models

Use pretrained video, language and video-language models without fine-tuning, and let an operator's question choose what to show. The generic path scores frames with STVT, trained on TVSum, then keeps a storyboard or a set of segments chosen under a time budget 2411.10016 §3.1; tab. 1. The query path embeds the question and fixed-length segments with LanguageBind, and keeps the six segments with the highest dot product 2411.10016 §3.2; tab. 1. A text version is written by VideoChat2 2411.10016 §3.2.

In a 30-person study on four 40-minute robot videos, query-driven summaries raised answer accuracy over raw video, with coefficient 0.5494 at p = 0.005. They also added task time, 156.3 s at p = 0.002 2411.10016 tab. 2. Storyboards were more accurate than video, with coefficient 0.9054 at p = 0.004. Their accuracy was significantly lower on event questions, with interaction coefficient −2.4466 at p < 0.001 2411.10016 tab. 2. These are the authors' results from one study. Card: What did the Robot do in my Absence?.

Zero-shot metric depth and normals from one image

Give the model one RGB image and its focal length, and read off metric depth in metres and a unit surface normal per pixel. No fine-tuning on the target set is needed for the zero-shot rows. Metric3D v2 reports zero-shot AbsRel of 0.067 on NYUv2 and 0.051 on KITTI for its ViT-g model 2404.15506 tab. 1. On NYUv2 the same AbsRel equals that of the best zero-shot baseline in its table, Polymax at 0.067, and δ1 is higher, 0.980 against 0.969 [arxiv:2404.15506 tab. 1]. Training images are mapped to one canonical focal length, 1000 px, and the prediction is mapped back to the real camera 2404.15506 §3.2. The focal length must be known for the metric result 2404.15506 §3.1. Card: Metric3D v2.

When to use which

The report names three factors to weigh when choosing an adaptation procedure: compute and memory, task data, and gradient access 2108.07258 §4.3. The model papers add a fourth, whether you can give the model prompts at use time, a fifth, whether you can run pretraining at all, a sixth, whether the camera's focal length is known, and a seventh, whether you can afford a multi-billion-parameter backbone at inference. The table gives the option that each source supports for each constraint. It does not give a fixed rule.

Implementation sketch

The survey proposes no training or inference algorithm. The first block records the three factors it names, in our own words. It is not a rule from the report.

# Collect the three factors the report names for choosing adaptation  [§4.3]
def adaptation_factors(model, task):
    # memory and storage left for adaptation, in bytes  [§4.3]
    budget = memory_and_storage_available()
    # labelled examples available for the target task  [§4.3]
    examples = count_task_examples(task)
    # True only if the weights, not just outputs, can be used for updates  [§4.3]
    gradients = model.gradient_access()
    return budget, examples, gradients

The second block shows the head used on frozen TESSERA embeddings. The optimizer and schedule for the head are not restated here.

# Small head on a frozen per-pixel embedding  [arxiv:2506.20380 §4.1] [app. TESSERA]
# Input:  z (128,) frozen embedding of one pixel
# Output: class logits (num_classes,)

def pixel_head(z, num_classes):
    # Two hidden layers of 256 and 128 units with ReLU  [app. TESSERA]
    h = relu(linear(z, 256))
    h = relu(linear(h, 128))
    # Softmax is applied in the loss, not here  [app. TESSERA]
    return linear(h, num_classes)

The third block shows the canonical focal-length transform for training labels, the label variant in the Metric3D v2 card. The image is not changed.

# Canonical focal-length transform on depth labels  [arxiv:2404.15506 §3.2]
# Input:  depth_gt (H, W) in metres, fx focal length in px
# Output: depth_c (H, W) in canonical units

def to_canonical_depth(depth_gt, fx, canonical_focal=1000.0):
    # Scale factor omega_d = f_c / f  [arxiv:2404.15506 §3.2]
    ratio = canonical_focal / fx
    # Train in canonical space; invert the ratio at inference  [arxiv:2404.15506 §3.2]
    return ratio * depth_gt

Risks

• Emergence. Abilities can appear in a large, broadly trained model without being programmed in, so what a model can do is hard to predict 2108.07258 §1.1.
• Homogenization. Many systems share one base model, so a flaw in the base model appears in every system built on it 2108.07258 §1.1; fig. 1.
• Deployment. The report says deployment is a separate decision from construction, and that deployed systems need more rigorous testing than research models 2108.07258 §1.2.
• Bias and misuse. It separates intrinsic bias, which belongs to the foundation model, from extrinsic harm, which arises in a specific application 2108.07258 §5.1. It defines misuse as using the model as designed to cause social harm 2108.07258 §5.2.
• Robot safety and data. The robotics survey lists hallucination, safety, on-device compute and the cost of collecting robot data as open challenges 2402.02385 §5.2; §5.3; §5.5; §5.6.
• Hallucination. The hallucination survey defines it as content that strays from factual reality or includes fabricated information 2309.05922 §1.2. Its table marks each of 28 works as detecting or mitigating, using the authors' own judgement 2309.05922 tab. 1. See hallucination-detection and hallucination-mitigation.

Contradictions

The survey says adaptation improves robustness across many distribution shifts, and attributes that claim to existing work 2108.07258 §4.8. RemoteCLIP reports zero-shot accuracy below CLIP on several datasets for every backbone it tested, with the largest drop on EuroSAT: −24.83 for ResNet-50 and −11.15 for ViT-B-32 2306.11029 tab. 3. The authors attribute these drops to input resolution and image distribution 2306.11029 §4.2.3. The two sides measure different things, so this page does not resolve them.

Within RemoteCLIP, the ResNet-50 zero-shot average is 53.46 in the main table 2306.11029 tab. 3 and 55.06 in the pretraining-method ablation 2306.11029 tab. 5. The card for that paper marks this as unexplained.

Caveats

• The survey is dated. The arXiv version used here is v3, from July 2022. Its model examples and cost figures reflect that time 2108.07258 §1.
• The survey's numbers are second-hand. The report cites them from other work, so check the cited source before reusing one 2108.07258 §1.1.
• The SAM evaluation builds prompts from the ground truth. It simulates user prompts, and it does not measure real ones 2304.10517 §2.2.1. Oracle numbers pick the best of three masks against the answer key, so they are an upper bound 2304.10517 §2.2.
• The RemoteCLIP training code is not in its public repository. Caption templates, the centre-region size and the de-duplication threshold are not fully specified 2306.11029 §3.2.1. Its counting and few-shot results are plots only 2306.11029 fig. 8; fig. 9.
• TESSERA's public repository does not contain its pretraining code. Several of its configs differ from the paper's stated values, including learning rate, batch size and mixup setting 2506.20380 §3.3. Report the conflict rather than picking one silently.
• The Metric3D v2 paper gives one learning rate for all layers 2404.15506 §4. Its released configs use a lower rate for the encoder, and use different loss weights. The card lists each difference (Metric3D v2).
• Time-LLM's released code differs from its paper in several settings, such as learning rate, batch size and the output projection. The paper's table is therefore not a recipe that reproduces its scripts [arxiv:2310.01728 app. B.4]. Its card lists each difference, and the ETTh1 numbers above are author-reported (Time-LLM).
• The robotics survey's section numbers follow the order of the LaTeX sectioning commands, because no compiled PDF was in the fetched files. Check them against the PDF before citing one precisely. Its dataset and simulator facts are as of early 2024 (Robotics survey).
• The hallucination survey runs no experiments, and its figures are quoted from the works it cites 2309.05922 §1. Check the original before reusing one.
• The dynamics paper has no official code. Its equations 4 and 6 use symbols that the text does not define, including η_x, η_y and ŷ_f 2412.00581 eq. 4; eq. 6. Per-model distance errors appear only as box plots, and it gives no runtime or memory figures 2412.00581 fig. 4; fig. 5. Its training and test trajectories come from the same logs, and the paper does not say how the split was made 2412.00581 §6. The card lists these and other gaps (Dynamics Modeling).
• SALON's code is partial. The top-level repository holds a workspace and a launch-file list, and the algorithm sits in a submodule pinned to an older commit. That submodule had no LICENSE file at that commit, so do not copy code from it until its licence is checked. The code also refits the Gaussian process online and uses a subset of feature columns for the speed model, which the paper does not describe 2412.07826 §3.4. See the SALON card for the full list.
• The video-summary study is one study of 30 participants with four videos from one mission type 2411.10016 §3.4.1. Its query frame rate is stated as 1 fps in one place and as the original or lower rate in another 2411.10016 tab. 1; §3.2. The authors also call preference scores subjective and recommend the accuracy and time results for design decisions 2411.10016 §4.4. The card has the full list (What did the Robot do in my Absence?).

Papers

• On the Opportunities and Risks of Foundation Models: defines the term, surveys capabilities, applications, adaptation and societal risks. It has no experiments of its own.
• A Survey of Hallucination in "Large" Foundation Models: a literature survey that tabulates 28 works on hallucination across text, image, video and audio. It has no experiments of its own.
• A Survey on Robotics with Foundation Models: reviews LLMs and VLMs for robotic manipulation, covering high-level planning and low-level control. It has no experiments of its own.
• Segment Anything Model for Medical Image Analysis: evaluates SAM zero-shot on medical images under five prompt modes, with box prompts ahead of point prompts.
• RemoteCLIP: continues CLIP pretraining on aerial image-text pairs, with captions generated from detection and segmentation labels.
• Time-LLM: reprograms a frozen Llama-7B for time-series forecasting, training only input and output layers and a text prompt.
• TESSERA: a pixel-wise self-supervised model over satellite time series, released as frozen embeddings for small heads.
• Metric3D v2: predicts metric depth and surface normals from one RGB image, given the focal length, with a canonical-camera training transform.
• Dynamics Modeling using Visual Terrain Features for High-Speed Autonomous Off-Road Driving: a hybrid vehicle model with a learned correction, fed by DINOv2 terrain features compressed by an encoder.
• SALON: Self-supervised Adaptive Learning for Off-road Navigation: online traversability learning on vision-foundation-model features, with a Gaussian process and a one-click label for lethal objects.
• What did the Robot do in my Absence?: generic and query-driven summaries of robot video from pretrained video models, shown as storyboards, short videos or text.