---
kind: concept
slug: inductive-bias
title: Inductive bias
summary: >-
  The built-in assumptions a learner uses to prefer one explanation over another, which every generalization claim depends on. A position paper argues that deep learning needs extra biases from high-level, verbalisable causal structure for out-of-distribution and systematic generalization. It proposes nine biases, gives no algorithm and no experiments (author-reported design arguments).
tags: [inductive-bias, generalization, causality, position-paper]
papers: ["2011.15091"]
related: [systematic-generalization, causal-inference, meta-learning, disentangled-representation, neural-symbolic-integration, world-model, domain-generalization, attention-mechanism]
status: draft
updated_at: 2026-10-24
---

## Overview

An inductive bias is any preference a learner has over functions beyond the
training data. The paper's starting point is that some such preference is
needed to generalize at all, and that test data drawn from the training
distribution cannot reveal a bias toward that distribution
[arxiv:2011.15091 §2]. Without assumptions about how training and test
distributions relate, nothing can be said about a changed distribution
[arxiv:2011.15091 §1.1].

The paper argues that data, compute and architecture alone are not enough, and
that deep learning needs extra biases to generalize out of distribution and
systematically [arxiv:2011.15091 §1] [arxiv:2011.15091 tab. 2]. The biases it
proposes are taken from high-level, conscious cognition: variables that can be
put into words, that change sparsely, and that combine through a sparse
structure. It presents these as hypotheses and as an open research direction,
not as a tested method [arxiv:2011.15091 §1].

## Variants

Here "variant" means one of the paper's proposed biases. They are design
arguments that can be combined, and the paper does not test any of them alone
or together.

### Verbalisable causal variables

Representations should be built from high-level variables, which the paper
says are generally verbalizable and often causal [arxiv:2011.15091 §3.4]
[arxiv:2011.15091 §3.5]. Attention is read as a soft form of variable binding,
but the paper says discrete typed arguments and hard choices are missing
[arxiv:2011.15091 §3.2]. See [attention mechanism](/concepts/attention-mechanism).

### Sparse, local changes

Most distribution shifts come from one or a few variables or mechanisms, and
interventions start at one or a few nodes of the causal graph
[arxiv:2011.15091 §3.6]. A model that assumes this should adapt by changing
only the parts that moved.

### Stable and fast knowledge

Stable knowledge is learned slowly and shared, and volatile knowledge is
learned fast and is the only part that needs adapting. The paper says this
needs at least two learning speeds, like the outer and inner loops of
[meta-learning](/concepts/meta-learning) [arxiv:2011.15091 §3.7]
[arxiv:2011.15091 §2].

### Sparse factor graph

The joint distribution over high-level variables should be a sparse factor
graph, not a set of independent marginals. The paper ties this sparsity to the
small capacity of a global-workspace bottleneck [arxiv:2011.15091 §3.8]
[arxiv:2011.15091 §3.3].

### Reusable schemas

Knowledge is written as schemas whose argument variables are bound to
instances, so one rule covers many objects and is not stored per instance
[arxiv:2011.15091 §3.9].

### Short causal chains

Learning and inference chains should be short in depth, even when the events
they link are far apart in time [arxiv:2011.15091 §3.10]. The authors say earlier
work they cite found this the strongest remedy for vanishing gradients over
long dependencies [arxiv:2011.15091 §3.10].

### Top-down and bottom-up combination

Top-down context should combine with bottom-up sensory signals dynamically, at
every level of the hierarchy [arxiv:2011.15091 §3.11].

### Independent mechanisms

Intervening on one mechanism should not change the others, and knowing one
mechanism should say nothing about another [arxiv:2011.15091 §4.1]. The paper
states the structural causal model behind this in [arxiv:2011.15091 §4 eq. 1]
and [arxiv:2011.15091 §4 eq. 2].

### Use non-stationarity

Data should not be shuffled into an i.i.d. sample, because changes between
sources carry causal information [arxiv:2011.15091 §4.2].

## When to use which

The paper reports no experiments, so none of these rows is backed by a measured
comparison. The rows follow the paper's own conditions for each bias.

| Situation | Use | Why |
|---|---|---|
| Distribution changes are expected to affect only a few factors | Sparse, local changes; sparse factor graph | The paper assumes changes are local and names the few variables [arxiv:2011.15091 §3.6] [arxiv:2011.15091 §3.8] |
| Data arrives in sources that differ over time | Use non-stationarity | Source changes are treated as a causal signal, not noise [arxiv:2011.15091 §4.2] |
| Some knowledge holds across many instances | Reusable schemas | Knowledge is shared across objects instead of stored per instance [arxiv:2011.15091 §3.9] |
| Long dependencies must be learned from few steps | Short causal chains | The paper says this prior helps against vanishing gradients, citing earlier work [arxiv:2011.15091 §3.10] |
| Only a small set of causal variables is available to name | Verbalisable causal variables | The paper's argument rests on those variables being nameable [arxiv:2011.15091 §3.4] |
| You need a working recipe today | None in this paper | The paper gives no algorithm, and says combining the biases in one system is open [arxiv:2011.15091 §1] |

## Implementation sketch

The paper gives no training algorithm. The sketch below only shows how one of
its formal objects, the structural causal model, can be sampled. Its order is a
standard topological order, not the paper's procedure
[arxiv:2011.15091 §4 eq. 2].

```pseudocode
# Sample one joint draw from a structural causal model
# Input:  parents[i] for each of M variables, from the DAG
#         functions[i], noise_samplers[i]
# Output: values of X_1..X_M for one draw

def sample_scm(parents, functions, noise_samplers):
    # One independent noise per variable  [arxiv:2011.15091 §4 eq. 2]
    noises = [draw(noise_samplers[i]) for i in range(M)]
    values = [None] * M
    # Visit variables so each parent is set before its children
    for i in topological_order(parents):
        # X_i := f_i(parents of i, N_i)  [arxiv:2011.15091 §4 eq. 2]
        values[i] = functions[i](values[parents[i]], noises[i])
    return values
```

## Contradictions

Only one paper card covers this concept, so there is no conflict between cards.
The paper does argue against an assumption that is common in disentangling
work: it rejects treating high-level variables as marginally independent and
asks for a sparse joint structure instead [arxiv:2011.15091 §3.8]. See
[disentangled representation](/concepts/disentangled-representation). The paper
also says its compositional biases may be worth "even possibly exponentially
more data", without a derivation [arxiv:2011.15091 §2]. The paper does not
resolve these points.

## Papers

- [Inductive Biases for Deep Learning of Higher-Level Cognition](/papers/2011.15091): position paper that proposes nine inductive biases from conscious, causal processing for out-of-distribution and systematic generalization. No experiments.