--- kind: concept slug: foundation-model title: Foundation model summary: >- A model trained once on broad data, usually with self-supervision at scale, then reused for many tasks by prompting, adapting or building on its embeddings. The main choice is how to use it, given labels, compute and model access. The report's two central risks are emergence and homogenization. tags: [pretraining, adaptation, transfer-learning, zero-shot, societal-impact, hallucination, depth-estimation, time-series, robotics, traversability, video-understanding] papers: ["2108.07258", "2304.10517", "2306.11029", "2309.05922", "2310.01728", "2402.02385", "2404.15506", "2411.10016", "2412.00581", "2412.07826", "2506.20380"] related: [self-supervised-learning, transfer-learning, emergent-abilities, language-model-pretraining, in-context-learning, promptable-segmentation, vision-language-pretraining, linear-probing, domain-adaptation, monocular-depth-estimation, hallucination-detection, hallucination-mitigation, time-series-forecasting, prompt-engineering, llm-planning, vision-language-action-model, robot-simulation, gaussian-process, bird-eye-view-perception, anomaly-detection, model-predictive-control, long-short-term-memory, human-in-the-loop] status: draft updated_at: 2026-10-25 --- ## Overview A foundation model is a model trained on broad data, usually with self-supervision at scale, that can be adapted to many downstream tasks. BERT, GPT-3 and CLIP are the report's examples [arxiv:2108.07258 §1]. Its building blocks, deep networks and self-supervised learning, are not new. Scale, the Transformer architecture and more data are what changed [arxiv:2108.07258 §1]. The pattern is transfer learning plus scale: pretrain on a surrogate task, then adapt to the task you care about [arxiv:2108.07258 §1]. The report's two central themes are emergence and homogenization. The same base model gives leverage across many applications, and it also passes its defects on to every model adapted from it [arxiv:2108.07258 §1.1; fig. 1]. **Formal view of adaptation.** Adaptation is a constrained optimization: the adapted parameters minimize the adaptation loss, given the pretrained parameters, under an optional complexity constraint. The report says current learning theory does not fully explain the gap between pretraining and adaptation [arxiv:2108.07258 §4.10]. Three of the cited papers are surveys: the report that defines the term [arxiv:2108.07258], a survey of hallucination in these models [arxiv:2309.05922], and a survey of foundation models in robotic manipulation [arxiv:2402.02385]. The other eight each take one model family and report how it is used on one kind of input: promptable segmentation of medical images, image-text pretraining for aerial images, reprogramming a frozen language model for time-series forecasting, per-pixel embeddings of satellite time series, monocular depth and surface normals, pretrained vision features for off-road vehicle dynamics and terrain traversability, and pretrained video models for summarising robot footage. Their numbers are author-reported and come from those inputs only. ## Variants The variants below are ways of using a pretrained foundation model. The first two come from the report. The robotics survey supplies the planner-and-controller variant. The rest come from the model papers. None of the model papers is a general method, so each variant is described as that paper used it. The hallucination survey adds no variant; it is covered under Risks. ### Low-storage adaptation Freeze most pretrained parameters and learn a small set of task-specific ones. The report lists tuning only the last layer, tuning bias vectors, low-rank residuals, masks, soft prompts and inserted modules [arxiv:2108.07258 §4.3]. It notes that some of these methods trade compute for memory [arxiv:2108.07258 §4.3]. Card: [On the Opportunities and Risks of Foundation Models](/papers/2108.07258). ### Prompt-based adaptation The report cites a study in which a well-tuned prompt can be worth about 100 training examples [arxiv:2108.07258 §4.3]. Soft prompts are one of the low-storage options above. ### Zero-shot prompting of a promptable model Use the released model as it is and steer it with a per-query input, such as a click or a box. Segment Anything (SAM) was evaluated this way on 19 medical imaging datasets, 28 tasks, with no training [arxiv:2304.10517 §2.1]. With one box per part of the target, the mean IoU was 0.6542 [arxiv:2304.10517 §3.1]. Single-prompt IoU ranged from 0.1136 to 0.9118 across tasks [arxiv:2304.10517 §3.1]. With one point, SAM averaged 0.4595 against 0.2240 for the best of three interactive baselines [arxiv:2304.10517 §3.2]. Card: [Segment Anything Model for Medical Image Analysis](/papers/2304.10517). ### Continued pretraining on in-domain image-text data Keep training a contrastive image-text model on data from a new domain. The RemoteCLIP authors started from CLIP and trained on 165,745 aerial images with five captions each [arxiv:2306.11029 §4.1.2]. The captions were made from detection boxes and segmentation masks, so the set was about 12 times larger than the existing caption-only sets [arxiv:2306.11029 §3.2]. On 12 classification datasets, zero-shot average accuracy rose over CLIP by 6.39 points for ViT-B-32 and by 5.63 for ViT-L-14 [arxiv:2306.11029 tab. 3]. Card: [RemoteCLIP](/papers/2306.11029). ### Frozen language model with trained input and output layers Keep a language model's weights frozen and train only the layers around it. Time-LLM uses a frozen Llama-7B for forecasting. A small patch-to-text cross-attention layer maps series patches onto learned text prototypes, a text prompt is prepended, and one linear layer maps the outputs to the forecast horizon [arxiv:2310.01728 §3.1]. The paper reports fewer than 6.6 million trainable parameters [arxiv:2310.01728 §4.5]. On the long-term benchmarks, author-reported MSE is best on seven of eight datasets. On ETTh1, averaged over four horizons, MSE is 0.408 against 0.465 for GPT4TS [arxiv:2310.01728 §4.1]. With the first 10% of training steps, ETTh1 MSE is 0.556 against 0.590 for GPT4TS [arxiv:2310.01728 §4.3]. The 32-layer backbone measured 32,136 MiB of GPU memory at ETTh1 with a 96-step horizon [arxiv:2310.01728 §4.5]. Card: [Time-LLM](/papers/2310.01728). ### Frozen embeddings with a small head Pretrain once, release the frozen embeddings, and train only a small head per task. TESSERA produces a 128-value embedding per 10 m pixel per year from Sentinel-1 and Sentinel-2 time series [arxiv:2506.20380 §1]. Its evaluation uses a two-layer MLP for pixel-wise tasks and a U-Net for patch-level tasks [arxiv:2506.20380 §4.1]. Card: [TESSERA](/papers/2506.20380). ### Foundation model as robot planner and controller Use a language or vision-language model in the robot's control stack, either to plan the steps of a task or to supply features for a low-level policy. The survey splits high-level planning by plan form: policy code that calls control primitives, free-text plans, and PDDL that an external planner solves [arxiv:2402.02385 §2.1]. It adds visual grounding, extra knowledge and environment feedback as assistants to planning [arxiv:2402.02385 §2.2]. For low-level control it groups work into policy learning, environment modelling and representation learning [arxiv:2402.02385 §3], and rates the approaches in its table on reward and data needs [arxiv:2402.02385 tab. 1]. The ratings are qualitative. The survey reports no experiments, so no numbers from it are transferable to a specific robot. Card: [A Survey on Robotics with Foundation Models](/papers/2402.02385). ### Foundation-model features inside a robot's own model Feed features from a pretrained vision model into a model the robot runs itself, and learn only the part that uses them. In the vehicle dynamics paper, DINOv2 ViT-S/14 features are reduced by PCA to 40 values per location. A small encoder compresses each wheel's values, and an LSTM corrects a physics model with the result. The encoder is trained end to end with the dynamics model [arxiv:2412.00581 §3.2.1; §5; fig. 1]. In SALON, each pixel's feature becomes a set of distances to k-means cluster centres. Those features are projected into a bird's-eye-view map, paired with a roughness value from the vehicle's vibration, and fed to a Gaussian process that is updated online [arxiv:2412.07826 §3.1; §3.2; §3.4]. The vision model is DINOv2 ViT-B on the ATV and RADIOv2.5 ViT-B on the wheelchair [arxiv:2412.07826 §4.1.1; §4.1.2]. With hindsight features, which need sensor data the robot does not have at runtime, the dynamics model's mean summed loss is about 10% below the no-feature baseline [arxiv:2412.00581 §6.1]. With features from realistic projection distances, the model beats the baseline only within 0 to 30 m [arxiv:2412.00581 §6.2]. On a figure-8 course of five laps with one ATV, SALON at R_max 0.2 had 3 interventions, against 4 for Velociraptor. Its average speed was 3.60 m/s against 4.82 m/s [arxiv:2412.07826 tab. 4]. The authors also state that its navigation is comparable to methods trained on 100 to 1000 times more data. The card does not check that claim [arxiv:2412.07826 §1]. Cards: [Dynamics Modeling using Visual Terrain Features](/papers/2412.00581); [SALON](/papers/2412.07826). ### Query-driven use of pretrained video models Use pretrained video, language and video-language models without fine-tuning, and let an operator's question choose what to show. The generic path scores frames with STVT, trained on TVSum, then keeps a storyboard or a set of segments chosen under a time budget [arxiv:2411.10016 §3.1; tab. 1]. The query path embeds the question and fixed-length segments with LanguageBind, and keeps the six segments with the highest dot product [arxiv:2411.10016 §3.2; tab. 1]. A text version is written by VideoChat2 [arxiv:2411.10016 §3.2]. In a 30-person study on four 40-minute robot videos, query-driven summaries raised answer accuracy over raw video, with coefficient 0.5494 at p = 0.005. They also added task time, 156.3 s at p = 0.002 [arxiv:2411.10016 tab. 2]. Storyboards were more accurate than video, with coefficient 0.9054 at p = 0.004. Their accuracy was significantly lower on event questions, with interaction coefficient −2.4466 at p < 0.001 [arxiv:2411.10016 tab. 2]. These are the authors' results from one study. Card: [What did the Robot do in my Absence?](/papers/2411.10016). ### Zero-shot metric depth and normals from one image Give the model one RGB image and its focal length, and read off metric depth in metres and a unit surface normal per pixel. No fine-tuning on the target set is needed for the zero-shot rows. Metric3D v2 reports zero-shot AbsRel of 0.067 on NYUv2 and 0.051 on KITTI for its ViT-g model [arxiv:2404.15506 tab. 1]. On NYUv2 the same AbsRel equals that of the best zero-shot baseline in its table, Polymax at 0.067, and δ1 is higher, 0.980 against 0.969 [arxiv:2404.15506 tab. 1]. Training images are mapped to one canonical focal length, 1000 px, and the prediction is mapped back to the real camera [arxiv:2404.15506 §3.2]. The focal length must be known for the metric result [arxiv:2404.15506 §3.1]. Card: [Metric3D v2](/papers/2404.15506). ## When to use which The report names three factors to weigh when choosing an adaptation procedure: compute and memory, task data, and gradient access [arxiv:2108.07258 §4.3]. The model papers add a fourth, whether you can give the model prompts at use time, a fifth, whether you can run pretraining at all, a sixth, whether the camera's focal length is known, and a seventh, whether you can afford a multi-billion-parameter backbone at inference. The table gives the option that each source supports for each constraint. It does not give a fixed rule. | Constraint | Option the sources support | What the sources report | Source | |---|---|---|---| | Limited memory or storage | Low-storage methods | Some trade compute for memory | [arxiv:2108.07258 §4.3] | | Little task-specific data | Prompt-based, or frozen embeddings with a small head | A well-tuned prompt may be worth about 100 examples (cited study). TESSERA-based heads beat the compared embeddings at 1% labels on two of the reported tasks | [arxiv:2108.07258 §4.3]; [arxiv:2506.20380 tab. 1] | | Point or box prompts can be given at use time | Zero-shot prompting | Box prompts beat point prompts in the reported setting. Results vary widely by task, so test on your own target | [arxiv:2304.10517 §3.1] | | Iterative clicks are available | Zero-shot prompting with clicks | More points add little for SAM, and two interactive baselines pass it at 5 or more points | [arxiv:2304.10517 §3.3] | | Domain far from web data, paired text or convertible labels, and pretraining compute | Continued pretraining | Gains over CLIP on average, with drops on some datasets. The largest model took 233.4 h on four 3090 Ti GPUs | [arxiv:2306.11029 tab. 3]; [arxiv:2306.11029 §4.1.3] | | Only model outputs, no weights or gradients | Prompting, or use of released embeddings | The survey expects large models to be offered through restricted API access | [arxiv:2108.07258 §4.3]; [arxiv:2506.20380 §1] | | One image per query, focal length known, metric scale needed | Zero-shot use of a geometric foundation model | Zero-shot AbsRel 0.067 on NYUv2 and 0.051 on KITTI for ViT-g. Results are for the paper's own benchmarks and intrinsics | [arxiv:2404.15506 tab. 1] | | Time series with few labelled examples, a multi-billion-parameter backbone affordable at inference, and no weight updates to the backbone | Frozen language model with trained input and output layers | Few-shot ETTh1 MSE 0.556 against 0.590 for GPT4TS, with first 10% of steps. Its backbone ablations show a drop with 8 layers instead of 32. Results are author-reported and the code differs from the paper in several settings | [arxiv:2310.01728 §4.3]; [arxiv:2310.01728 §4.5]; [arxiv:2310.01728 app. B.4] | | Robot must act on a language instruction, with a long task that needs several skills in order | Foundation model as robot planner, with low-level policy learned separately | The survey names long-horizon sequencing and poor generalisation of single-task policies as the problems. It names the lack of large robot data as the central bottleneck. It reports no experiments | [arxiv:2402.02385 §2 intro]; [arxiv:2402.02385 §5.3] | | Outputs must be factually reliable, e.g. journalism, healthcare, law | Hallucination detection and mitigation, see [hallucination-detection](/concepts/hallucination-detection) and [hallucination-mitigation](/concepts/hallucination-mitigation) | The survey names these fields as ones where factual errors are costly. It tabulates 28 detection or mitigation works | [arxiv:2309.05922 §1.2]; [arxiv:2309.05922 tab. 1] | | Robot adapts to new terrain from its own driving, with no offline dataset and one human label at most | Foundation-model features with an online Gaussian-process head | On one ATV course, SALON at R_max 0.2 had 3 interventions against 4 for Velociraptor, with lower average speed. One course, author-reported, and the code is partial | [arxiv:2412.07826 tab. 4]; [arxiv:2412.07826 §4.2.3] | | Vehicle dynamics must be predicted from camera features, and the features come from projection distances that vary at runtime | Foundation-model features with a learned encoder in a hybrid model | The gain over no-feature baseline holds within 0 to 30 m only. Results with hindsight features, which runtime lacks, are larger | [arxiv:2412.00581 §6.2]; [arxiv:2412.00581 §6.1] | | Operator must review long recorded robot video and can afford some extra task time | Query-driven summaries from pretrained video models | Accuracy rose with query-driven summaries, with about 156 s more task time. Storyboards were the most accurate modality, but weakest on event questions | [arxiv:2411.10016 tab. 2] | | Operator needs an accurate overview without a query, and questions are mostly about objects | Generic storyboard summary | Storyboards beat video on accuracy and cut task time, but this is weaker for event questions. The sample is one study of 30 participants | [arxiv:2411.10016 tab. 2]; [arxiv:2411.10016 §4.4] | ## Implementation sketch The survey proposes no training or inference algorithm. The first block records the three factors it names, in our own words. It is not a rule from the report. ```pseudocode # Collect the three factors the report names for choosing adaptation [§4.3] def adaptation_factors(model, task): # memory and storage left for adaptation, in bytes [§4.3] budget = memory_and_storage_available() # labelled examples available for the target task [§4.3] examples = count_task_examples(task) # True only if the weights, not just outputs, can be used for updates [§4.3] gradients = model.gradient_access() return budget, examples, gradients ``` The second block shows the head used on frozen TESSERA embeddings. The optimizer and schedule for the head are not restated here. ```pseudocode # Small head on a frozen per-pixel embedding [arxiv:2506.20380 §4.1] [app. TESSERA] # Input: z (128,) frozen embedding of one pixel # Output: class logits (num_classes,) def pixel_head(z, num_classes): # Two hidden layers of 256 and 128 units with ReLU [app. TESSERA] h = relu(linear(z, 256)) h = relu(linear(h, 128)) # Softmax is applied in the loss, not here [app. TESSERA] return linear(h, num_classes) ``` The third block shows the canonical focal-length transform for training labels, the label variant in the Metric3D v2 card. The image is not changed. ```pseudocode # Canonical focal-length transform on depth labels [arxiv:2404.15506 §3.2] # Input: depth_gt (H, W) in metres, fx focal length in px # Output: depth_c (H, W) in canonical units def to_canonical_depth(depth_gt, fx, canonical_focal=1000.0): # Scale factor omega_d = f_c / f [arxiv:2404.15506 §3.2] ratio = canonical_focal / fx # Train in canonical space; invert the ratio at inference [arxiv:2404.15506 §3.2] return ratio * depth_gt ``` ## Risks - **Emergence.** Abilities can appear in a large, broadly trained model without being programmed in, so what a model can do is hard to predict [arxiv:2108.07258 §1.1]. - **Homogenization.** Many systems share one base model, so a flaw in the base model appears in every system built on it [arxiv:2108.07258 §1.1; fig. 1]. - **Deployment.** The report says deployment is a separate decision from construction, and that deployed systems need more rigorous testing than research models [arxiv:2108.07258 §1.2]. - **Bias and misuse.** It separates intrinsic bias, which belongs to the foundation model, from extrinsic harm, which arises in a specific application [arxiv:2108.07258 §5.1]. It defines misuse as using the model as designed to cause social harm [arxiv:2108.07258 §5.2]. - **Robot safety and data.** The robotics survey lists hallucination, safety, on-device compute and the cost of collecting robot data as open challenges [arxiv:2402.02385 §5.2; §5.3; §5.5; §5.6]. - **Hallucination.** The hallucination survey defines it as content that strays from factual reality or includes fabricated information [arxiv:2309.05922 §1.2]. Its table marks each of 28 works as detecting or mitigating, using the authors' own judgement [arxiv:2309.05922 tab. 1]. See [hallucination-detection](/concepts/hallucination-detection) and [hallucination-mitigation](/concepts/hallucination-mitigation). ## Contradictions The survey says adaptation improves robustness across many distribution shifts, and attributes that claim to existing work [arxiv:2108.07258 §4.8]. RemoteCLIP reports zero-shot accuracy below CLIP on several datasets for every backbone it tested, with the largest drop on EuroSAT: −24.83 for ResNet-50 and −11.15 for ViT-B-32 [arxiv:2306.11029 tab. 3]. The authors attribute these drops to input resolution and image distribution [arxiv:2306.11029 §4.2.3]. The two sides measure different things, so this page does not resolve them. Within RemoteCLIP, the ResNet-50 zero-shot average is 53.46 in the main table [arxiv:2306.11029 tab. 3] and 55.06 in the pretraining-method ablation [arxiv:2306.11029 tab. 5]. The card for that paper marks this as unexplained. ## Caveats - The survey is dated. The arXiv version used here is v3, from July 2022. Its model examples and cost figures reflect that time [arxiv:2108.07258 §1]. - The survey's numbers are second-hand. The report cites them from other work, so check the cited source before reusing one [arxiv:2108.07258 §1.1]. - The SAM evaluation builds prompts from the ground truth. It simulates user prompts, and it does not measure real ones [arxiv:2304.10517 §2.2.1]. Oracle numbers pick the best of three masks against the answer key, so they are an upper bound [arxiv:2304.10517 §2.2]. - The RemoteCLIP training code is not in its public repository. Caption templates, the centre-region size and the de-duplication threshold are not fully specified [arxiv:2306.11029 §3.2.1]. Its counting and few-shot results are plots only [arxiv:2306.11029 fig. 8; fig. 9]. - TESSERA's public repository does not contain its pretraining code. Several of its configs differ from the paper's stated values, including learning rate, batch size and mixup setting [arxiv:2506.20380 §3.3]. Report the conflict rather than picking one silently. - The Metric3D v2 paper gives one learning rate for all layers [arxiv:2404.15506 §4]. Its released configs use a lower rate for the encoder, and use different loss weights. The card lists each difference ([Metric3D v2](/papers/2404.15506)). - Time-LLM's released code differs from its paper in several settings, such as learning rate, batch size and the output projection. The paper's table is therefore not a recipe that reproduces its scripts [arxiv:2310.01728 app. B.4]. Its card lists each difference, and the ETTh1 numbers above are author-reported ([Time-LLM](/papers/2310.01728)). - The robotics survey's section numbers follow the order of the LaTeX sectioning commands, because no compiled PDF was in the fetched files. Check them against the PDF before citing one precisely. Its dataset and simulator facts are as of early 2024 ([Robotics survey](/papers/2402.02385)). - The hallucination survey runs no experiments, and its figures are quoted from the works it cites [arxiv:2309.05922 §1]. Check the original before reusing one. - The dynamics paper has no official code. Its equations 4 and 6 use symbols that the text does not define, including η_x, η_y and ŷ_f [arxiv:2412.00581 eq. 4; eq. 6]. Per-model distance errors appear only as box plots, and it gives no runtime or memory figures [arxiv:2412.00581 fig. 4; fig. 5]. Its training and test trajectories come from the same logs, and the paper does not say how the split was made [arxiv:2412.00581 §6]. The card lists these and other gaps ([Dynamics Modeling](/papers/2412.00581)). - SALON's code is partial. The top-level repository holds a workspace and a launch-file list, and the algorithm sits in a submodule pinned to an older commit. That submodule had no LICENSE file at that commit, so do not copy code from it until its licence is checked. The code also refits the Gaussian process online and uses a subset of feature columns for the speed model, which the paper does not describe [arxiv:2412.07826 §3.4]. See the [SALON card](/papers/2412.07826) for the full list. - The video-summary study is one study of 30 participants with four videos from one mission type [arxiv:2411.10016 §3.4.1]. Its query frame rate is stated as 1 fps in one place and as the original or lower rate in another [arxiv:2411.10016 tab. 1; §3.2]. The authors also call preference scores subjective and recommend the accuracy and time results for design decisions [arxiv:2411.10016 §4.4]. The card has the full list ([What did the Robot do in my Absence?](/papers/2411.10016)). ## Papers - [On the Opportunities and Risks of Foundation Models](/papers/2108.07258): defines the term, surveys capabilities, applications, adaptation and societal risks. It has no experiments of its own. - [A Survey of Hallucination in "Large" Foundation Models](/papers/2309.05922): a literature survey that tabulates 28 works on hallucination across text, image, video and audio. It has no experiments of its own. - [A Survey on Robotics with Foundation Models](/papers/2402.02385): reviews LLMs and VLMs for robotic manipulation, covering high-level planning and low-level control. It has no experiments of its own. - [Segment Anything Model for Medical Image Analysis](/papers/2304.10517): evaluates SAM zero-shot on medical images under five prompt modes, with box prompts ahead of point prompts. - [RemoteCLIP](/papers/2306.11029): continues CLIP pretraining on aerial image-text pairs, with captions generated from detection and segmentation labels. - [Time-LLM](/papers/2310.01728): reprograms a frozen Llama-7B for time-series forecasting, training only input and output layers and a text prompt. - [TESSERA](/papers/2506.20380): a pixel-wise self-supervised model over satellite time series, released as frozen embeddings for small heads. - [Metric3D v2](/papers/2404.15506): predicts metric depth and surface normals from one RGB image, given the focal length, with a canonical-camera training transform. - [Dynamics Modeling using Visual Terrain Features for High-Speed Autonomous Off-Road Driving](/papers/2412.00581): a hybrid vehicle model with a learned correction, fed by DINOv2 terrain features compressed by an encoder. - [SALON: Self-supervised Adaptive Learning for Off-road Navigation](/papers/2412.07826): online traversability learning on vision-foundation-model features, with a Gaussian process and a one-click label for lethal objects. - [What did the Robot do in my Absence?](/papers/2411.10016): generic and query-driven summaries of robot video from pretrained video models, shown as storyboards, short videos or text.