On the Power of Foundation Models

In plain words

Big models are first trained on a generic task, such as predicting missing words or matching pictures to captions, and then reused for other jobs. This paper asks a blunt question: if the model were perfect and had unlimited data and compute, could it be used for everything? It answers with category theory, a branch of mathematics that describes things only by the arrows between them, not by what the things are made of.

The paper builds a map of what the model's internal states can express. Using only a frozen model and a text or image prompt, you can reach exactly those tasks that have a matching "representative" in that map. Retraining a small head on top of the model is much more powerful: in the ideal case it can reach any task. A third result says that if the way the model sees one domain (text) lines up with another (images), the second domain can contain objects that were never in the training data. That result is about existence, not about generating an image; a separate generator is still needed for that. The work is purely theoretical and runs no experiments.

TL;DR

• The paper treats a trained model's representation as defining a category: objects are inputs with equal representations, and arrows come from a readout built on a monoid action §5.2.
• Prompt tuning on a frozen ideal model solves a task iff the task is representable; the best prompt is its representative [Thm. 3] §5.5.
• Fine tuning on top of an ideal model can, given enough resources, solve any task in this framework [Thm. 5] §5.5.
• No experiments, hyperparameters or official code. Treat it as a framework, not a method to implement.

Problem

The paper asks which downstream tasks a foundation model can solve once it is trained on a pretext task and frozen, under idealised assumptions: infinite data, infinite compute, a perfect training algorithm and zero generalisation error on the pretext task §1. Existing theory of representation, optimisation and generalisation does not address this, because those issues are assumed away [Abstract] §1.

The paper argues that the question is about the structure of the tasks rather than about the model's representational power, optimisation or generalisation §1. It is model-oblivious: no assumptions on data distribution or network architecture are made §1.

The framework only models relationships that are consistent. Noisy labels are treated outside the framework, as implementation side effects §7.2.

Method

The paper is a set of definitions and theorems. The working pieces are:

1. Representation and objects. A trained model is a map f: D -> H from inputs to a representation space. Two inputs with equal representations are identified. Objects of the induced category are the classes D/~f, and an object is identified with its reachable representation f(X) [Def. 2] §5.2.

2. Readout and composition. A data-oblivious readout k_f(f(X), f(Y)) gives the morphisms from X to Y, a composition law, and a unique identity per object. Associativity and unit laws must hold [Def. 2] §5.2.

3. Monoid-action construction. Given a monoid M_f with unit e and a right action alpha_f on the reachable representations, define k_f(f(X), f(Y)) as the set of monoid elements m with alpha_f(f(X), m) = f(Y). Composition is n o m = mn, and the identity is e §5.2.

4. Contrastive methods (Thm. 1). Take M_f = (H, +) acting by translation. Then the only morphism X -> Y is f(Y) - f(X), composition is addition, and the identity is zero [Thm. 1] §5.2.

5. Language modelling (Thm. 2). Take M_f as the free monoid V* of token strings, acting by appending tokens, so alpha_f(f(X), u) = f(Xu). A morphism X -> Y is the triple (X, Y, P_Y), present iff Y is reachable from X, where P_Y is the future-continuation law from state f(Y). Composition takes the continuation law from the target [Thm. 2] §5.2.

6. Yoneda view. Define h_C(X) = hom_C(X, .). A canonical realisation Phi_f maps f(X) to h_C(X), and the paper then identifies f(X) with h_C(X) by convention. Yoneda's lemma gives hom_{C^vee}(A, h_C(X)) ~ A(X) [Lemma 2] §5.4.

7. Prompt tuning (Thm. 3). Tasks are functors T in C^vee. Prompt tuning sends a prompt P and input X to the frozen model, and T(X) ~ hom_{C^vee}(T, h_C(X)) := k_f(T, f(X)) eq. 1 §5.5. The model solves T with prompt P iff T is isomorphic to h_C(P), i.e. T is representable [Def. 6] [Thm. 3] §5.5.

8. Prompt tuning on LLMs (Thm. 4). For a task T on inputs X, define a task T_P on prompt-plus-input strings, with T_P(P + X) = T(X) and empty output for strings not starting with P. The LLM solves T with prompts iff some prompt P has T_P isomorphic to f(P) [Thm. 4] §5.5.

9. Fine tuning (Lemma 3, Thm. 5). For a task T, the extension functor h^dagger T(A) = hom_{C^vee}(T, A) satisfies h^dagger T o h_C ~ T. A fine-tuning head that learns this extension perfectly, given enough data and compute, lets the concatenated model solve T [Lemma 3] [Thm. 5] §5.5.

10. Full embeddings across categories (Lemma 1, Def. 7, Thm. 6). A full embedding F: C -> B exists iff F factors through an isomorphic full subcategory of B [Lemma 1] §3. A feature-aligned functor \hat F: C^vee -> B^vee satisfies \hat F(h_C(X)) ~ h_B(F(X)) [Def. 7] §6.1. Under these assumptions, Hom structure is preserved, and each \hat F(h_C(X)) is represented by some Y_X in B, which may be taken as F(X) [Thm. 6] §6.1.

11. Application and chaining. Cor. 2 applies Thm. 6 to CLIP, with the assumption that CLIP learns a feature-aligned functor [Cor. 2] §6.2. Thm. 7 chains full embeddings F_1 ... F_{n-1} across n categories [Thm. 7] §6.3.

Pseudocode for the monoid-action readout §5.2 and the prompt-tuning criterion [Thm. 3] §5.5:

# Readout from a monoid action  [§5.2]
# Input:  representations f(X), f(Y); monoid M; right action alpha
# Output: set of morphisms X -> Y

def readout(f_x, f_y, monoid, alpha):
    # Morphisms are monoid elements that move state f_x to f_y  [§5.2]
    return {m for m in monoid if alpha(f_x, m) == f_y}


def compose(n, m):
    # Composition of morphisms X->Y and Y->Z is the monoid product  [§5.2]
    return m * n


# Can prompt tuning solve task T with a frozen ideal model?  [Thm. 3] [§5.5]
# Input:  task T (functor on C), candidate objects P of C
# Output: a prompt P whose representable functor matches T, or None

def solve_by_prompt(T, objects):
    # The solving prompt is a representative of T  [Def. 6] [Thm. 3]
    for P in objects:
        if isomorphic(T, representable(P)):
            return P
    return None

Hyperparameters

The paper reports no training or evaluation hyperparameters, and no experiments §1. The table lists the parameters that the theorems leave open.

Results

Author-reported. The paper has no experiments. Its results are theorems under stated assumptions. Numbering is one sequence per environment, as in the source (no per-section numbering).

Setups and baselines: none. The paper compares no models against each other.

Paper vs code

No official code.

Search performed on 2026-11-06:

• arXiv abstract page for 2211.16327: no code link, project page or "code available" statement. Its comments field reads "ICML'23. This version polished paper with the help of LLM, fixed a few notational issues".
• GitHub repository search for "power of foundation models category" (author-independent): no results.
• GitHub repository search for "yoneda foundation model": no results.

No repository belonging to the authors was found.

Pitfalls

• Ideal assumptions carry the result. Every theorem assumes a model that solves its pretext task perfectly, and in some cases infinite or "enough" data and compute §1 [Thm. 5]. The paper says these are not literal requirements and that the framework also applies to small datasets, with smaller categories §7.1. The card cannot verify that claim; it is the paper's own argument.

• Category equals reachable states, not inputs. Two inputs with the same representation are the same object [Def. 2] §5.2. Anything the representation collapses is invisible to the framework.

• f(X) is not literally a functor. The paper identifies f(X) with h_C(X) by a notational convention §5.4. An implementer should not treat a vector from the model as a Yoneda representable functor; the realisation Phi_f is needed for that reading.

• Language-model morphisms do not record generation probability. A morphism X -> Y records the continuation law P_Y after reaching Y, not the probability of producing Y from X §5.2. Composition keeps the continuation law of the target §5.2.

• Thm. 4 proof step is thin. From agreement on objects, the proof concludes T_P ~ h_C(P), citing that functor structure is "determined by the prefix structure" §5.5. The text does not argue this in detail. The acknowledgements say that a reviewer identified "a bug in the application of Theorem 3 to large language models" [Acknowledgements]. This card does not know what changed, and the source does not say.

• Thm. 3 needs a representative, and the paper does not say how to find one. The proof of solvability assumes a representative P of T exists [Thm. 3] §5.5. The pseudocode loops over objects only to make that existence explicit; it is not a procedure the paper gives.

• Fine tuning is not constructive. Thm. 5 needs a head that learns h^dagger T perfectly [Thm. 5] §5.5. The paper itself calls the bound a "raw upper bound with strong assumptions on the resources" and says a small network is what people use in practice §5.5.

• Rotation corollary is a pretext-only argument. Cor. 1 says the pretext objective guarantees only the four-fold rotational structure, and assumes no help from architecture, optimiser or implicit bias [Cor. 1] §5.5. This means "not guaranteed", not "impossible". Representations may hold more information than the objective requires.

• Generalisation theorem is about existence. Thm. 6 and Cor. 2 show that an image object exists in the representable image of h_B §6.1 §6.2. Producing an image needs an image-side decoder or generator, and the paper says so §6.2.

• Feature alignment is assumed, not learned. Cor. 2 assumes CLIP learns a feature-aligned functor for a full embedding §6.2. The paper says the full-embedding assumption is crucial. It does not show that CLIP learns such a functor.

• Noise is excluded by design. The framework assumes self-consistent objects and relationships with no label noise §7.2. Applying it to noisy data needs the separate Bayesian-optimal-classifier argument, which the paper leaves outside the framework §7.2.

• RKHS is a special case. A kernel gives a real-valued similarity per pair, while the paper's framework allows richer relationships; the paper argues that RKHS is insufficient for NLP-style relations §7.3.

Lineage

• Builds on: category theory, Yoneda lemma, Kashiwara and Schapira's extension result, and Adamek et al.'s full-embedding lemma §5.4 [Lemma 1] [Lemma 3].
• Relates to: SimCLR and MoCo, whose objectives the paper reformulates as an induced category §3.1 [Thm. 1]; spectral-clustering views of contrastive learning, which the paper cites as motivation for approximating pretext tasks §5.3.
• Applied to: CLIP and DALL-E 2 as examples of the generalisation theorem §6.2.
• Compares against: no baselines.
• Superseded by: none stated in this paper.