---
kind: concept
slug: information-bottleneck
title: Information bottleneck
summary: >-
  Information bottleneck (IB) appears in two compiled papers. One reads deep-network training as fitting, diffusion and equilibrium phases, with layer compression that the paper finds mostly saturation. The other uses a graph IB theorem to motivate a learned edge-drop augmentation for self-supervised graph contrastive learning.
tags: [training-dynamics, information-theory, interpretability, contrastive-learning]
papers: ["2403.18494", "2106.05819"]
related: [physics-informed-neural-network, adaptive-loss-weighting, gradient-descent-stability, deep-learning, graph-neural-network, contrastive-learning, data-augmentation]
status: draft
updated_at: 2026-10-24
---

## Overview

The information bottleneck method looks for a representation T of an input X
that keeps as much information as possible about an output Y and discards the
rest [arxiv:2403.18494 §1.3]. Applied to deep networks, IB theory says training
has two phases, fitting and diffusion, separated by a phase transition that
shows up in the signal-to-noise ratio (SNR) of the gradients. The theory holds
that most learning happens in the slow diffusion phase [arxiv:2403.18494 §1.3].
It also explains the transition by compression of layer activations, a claim
the paper reports as debated [arxiv:2403.18494 §1.3] [arxiv:2403.18494 §3.3].

Two compiled cards use the concept, for different purposes. The first uses IB
as an analysis lens on full-batch Adam training of physics-informed networks
[arxiv:2403.18494 §3.1]. The second uses an IB theorem for graphs as motivation
for a learned augmentation in self-supervised training [arxiv:2106.05819 §3.1].
Neither card uses IB as a training loss with labels. Read the variants below as
three different uses, not as a comparison of methods.

## Variants

### Phase view from gradient SNR

Track the batch-wise SNR of the gradient over training. Fitting has a high SNR
and a fast decline in error. Diffusion has a falling SNR while the error barely
moves. A third phase, "diffusion equilibrium", appears after the transition, in
which the SNR is flat and the error drops steeply [arxiv:2403.18494 §2.1]
[arxiv:2403.18494 §3.1]. Card: [Learning in PINNs: Phase transition, diffusion equilibrium, and generalization](/papers/2403.18494).

### Compression view from binned mutual information

Estimate I(X;T) for each layer by discretising its activations and taking the
entropy of the resulting codes. The paper uses 30 equidistant bins per neuron
[arxiv:2403.18494 §3.3]. Its repository sets the bin width from the mean
neuron range, which the paper does not state [code: utils/mi.py#L52-L56]. The
paper reports lossy compression after the transition, and most information
still available in every layer from early iterations, when binning is relative.
Deeper layers become close to binary [arxiv:2403.18494 §3.3] [fig. 8]. Card:
[Learning in PINNs: Phase transition, diffusion equilibrium, and generalization](/papers/2403.18494).

### Learned augmentation motivated by a graph IB theorem

Self-supervised graph contrastive learning needs an augmentation, and the usual
choices are hand-made. AD-GCL learns one. An encoder maximises agreement between
each graph and an edge-dropped copy, while an augmenter tries to make the copies
differ and pays a penalty on the expected drop ratio [arxiv:2106.05819 §3.2]
[eq. 6]. The paper's theorem links this min-max setup to a graph IB objective
[arxiv:2106.05819 §3.1]. It bounds what the optimal encoder keeps about the
graph that is irrelevant to the label, and lower-bounds what it keeps about the
label [arxiv:2106.05819 §3.1] [app. B]. The theorem needs a countable graph
space and a 1-WL-powerful encoder, and it does not give guarantees for a
trained GNN [arxiv:2106.05819 §3.1]. Card: [Adversarial Graph Augmentation to Improve Graph Contrastive Learning](/papers/2106.05819).

## When to use which

| Situation | Use | Why |
|---|---|---|
| You want to find where full-batch Adam changes regime on a PINN | Phase view from gradient SNR | The paper's phases are measured this way on PINN test problems [arxiv:2403.18494 §3.1]. The paper gives no detector for the onset, so it is set by eye [arxiv:2403.18494 §4]. |
| You want to measure how much input information each layer keeps | Compression view, with a stated binning | The estimate depends on the discretisation. The paper reports that absolute and relative binning gave different pictures, and it prefers relative binning [arxiv:2403.18494 §3.3]. Report the bin scheme with any number. |
| You want compression to explain generalisation | Not supported by this paper | Its own run shows the more-compressing vanilla model generalises worse [arxiv:2403.18494 §3.3]. |
| You want a self-supervised graph augmentation learned from data, without labels | Learned augmentation motivated by graph IB | The IB principle needs downstream labels, which self-supervised training lacks [arxiv:2106.05819 §1] [arxiv:2106.05819 §3.1]. The paper's method is a contrastive min-max, not an IB loss. |
| You want to cite the graph IB theorem as a guarantee | Not supported by this paper | The paper uses the theorem as motivation, under assumptions that a trained GNN does not meet [arxiv:2106.05819 §3.1] [app. B]. |
| You want a learned augmentation beyond edge dropping | Not established by this paper | The paper chose edge dropping as a proof of concept and leaves other search spaces to future work [arxiv:2106.05819 §3.2] [arxiv:2106.05819 §5.1]. |

## Implementation sketch

The compression view reduces to binning and counting. The paper's entropy over
discretised codes is [eq. 29] [arxiv:2403.18494 §3.3]. Reading that entropy as
I(X;T) assumes T is a deterministic function of X. That step is our derivation,
not the paper's wording.

```pseudocode
# Binned information estimate for one layer  [arxiv:2403.18494 §3.3] [eq. 29]
# Input:  activations A (N inputs, n neurons), bins = 30
# Output: entropy in bits, an estimate of I(X;T) for a deterministic layer

def binned_information(A, bins=30):
    # Bin width from the mean per-neuron range; a repository choice  [code]
    width = mean(max(A, axis=0) - min(A, axis=0)) / bins
    # Each neuron becomes a bin index; each input becomes a code of n digits
    codes = floor((A - min(A)) / width)
    # Count how many inputs share each code
    _, counts = unique(codes, axis=0, return_counts=True)
    p = counts / N
    # Entropy in bits; at most log2(N) when all codes differ
    return -sum(p * log2(p))
```

The repository's bin width differs from the paper's stated rule, so the value
depends on which one you follow. The card's Paper vs code section lists the
details. The AD-GCL card has its own pseudocode for the learned augmentation.

## Contradictions

- **Compression as the cause of the phase transition.** IB theory attributes
  the phase transition to saturation-driven compression. The paper notes that
  the transition has been replicated with ReLU, which is not bounded
  [arxiv:2403.18494 §3.3].
- **Compression and generalisation.** The paper lists ongoing debate about the
  compression phenomenon and its contribution to generalisation, citing several
  works [arxiv:2403.18494 §1.3]. Its own run shows the more-compressing vanilla
  model generalises worse [arxiv:2403.18494 §3.3].
- **What the measured compression means.** IB theory reads compression as
  information lost about the input. The paper finds some lossy compression
  after the transition, but reads it mainly as saturation of the activations.
  It reports that most information is still present in all layers
  [arxiv:2403.18494 §3.3].
- **Binning scheme.** The paper reports that absolute binning, used in some
  earlier IB studies, gave mixed results, and it prefers relative binning
  [arxiv:2403.18494 §3.3]. Those earlier studies are not on cards.
- **Two uses of IB.** One card uses IB as a phase and compression analysis of
  training [arxiv:2403.18494 §3.3]. The other uses it as a theorem that
  motivates augmentation [arxiv:2106.05819 §3.1]. Neither card tests the other's
  claim.

The debates above involve other works that have no card here. They are named
in the papers' bibliographies and are not summarised on this page.

## Papers

- [Learning in PINNs: Phase transition, diffusion equilibrium, and generalization](/papers/2403.18494): applies IB phases and layer compression to full-batch Adam on PINNs, and proposes residual-based attention.
- [Adversarial Graph Augmentation to Improve Graph Contrastive Learning](/papers/2106.05819): uses a graph IB theorem to motivate AD-GCL, a learned edge-drop augmentation for graph contrastive learning.