Information bottleneck

Overview

The information bottleneck method looks for a representation T of an input X that keeps as much information as possible about an output Y and discards the rest 2403.18494 §1.3. Applied to deep networks, IB theory says training has two phases, fitting and diffusion, separated by a phase transition that shows up in the signal-to-noise ratio (SNR) of the gradients. The theory holds that most learning happens in the slow diffusion phase 2403.18494 §1.3. It also explains the transition by compression of layer activations, a claim the paper reports as debated 2403.18494 §1.3 2403.18494 §3.3.

Two compiled cards use the concept, for different purposes. The first uses IB as an analysis lens on full-batch Adam training of physics-informed networks 2403.18494 §3.1. The second uses an IB theorem for graphs as motivation for a learned augmentation in self-supervised training 2106.05819 §3.1. Neither card uses IB as a training loss with labels. Read the variants below as three different uses, not as a comparison of methods.

Variants

Phase view from gradient SNR

Track the batch-wise SNR of the gradient over training. Fitting has a high SNR and a fast decline in error. Diffusion has a falling SNR while the error barely moves. A third phase, "diffusion equilibrium", appears after the transition, in which the SNR is flat and the error drops steeply 2403.18494 §2.1 2403.18494 §3.1. Card: Learning in PINNs: Phase transition, diffusion equilibrium, and generalization.

Compression view from binned mutual information

Estimate I(X;T) for each layer by discretising its activations and taking the entropy of the resulting codes. The paper uses 30 equidistant bins per neuron 2403.18494 §3.3. Its repository sets the bin width from the mean neuron range, which the paper does not state mi.py. The paper reports lossy compression after the transition, and most information still available in every layer from early iterations, when binning is relative. Deeper layers become close to binary 2403.18494 §3.3 fig. 8. Card: Learning in PINNs: Phase transition, diffusion equilibrium, and generalization.

Learned augmentation motivated by a graph IB theorem

Self-supervised graph contrastive learning needs an augmentation, and the usual choices are hand-made. AD-GCL learns one. An encoder maximises agreement between each graph and an edge-dropped copy, while an augmenter tries to make the copies differ and pays a penalty on the expected drop ratio 2106.05819 §3.2 eq. 6. The paper's theorem links this min-max setup to a graph IB objective 2106.05819 §3.1. It bounds what the optimal encoder keeps about the graph that is irrelevant to the label, and lower-bounds what it keeps about the label 2106.05819 §3.1 app. B. The theorem needs a countable graph space and a 1-WL-powerful encoder, and it does not give guarantees for a trained GNN 2106.05819 §3.1. Card: Adversarial Graph Augmentation to Improve Graph Contrastive Learning.

When to use which

Implementation sketch

The compression view reduces to binning and counting. The paper's entropy over discretised codes is eq. 29 2403.18494 §3.3. Reading that entropy as I(X;T) assumes T is a deterministic function of X. That step is our derivation, not the paper's wording.

# Binned information estimate for one layer  [arxiv:2403.18494 §3.3] [eq. 29]
# Input:  activations A (N inputs, n neurons), bins = 30
# Output: entropy in bits, an estimate of I(X;T) for a deterministic layer

def binned_information(A, bins=30):
    # Bin width from the mean per-neuron range; a repository choice  [code]
    width = mean(max(A, axis=0) - min(A, axis=0)) / bins
    # Each neuron becomes a bin index; each input becomes a code of n digits
    codes = floor((A - min(A)) / width)
    # Count how many inputs share each code
    _, counts = unique(codes, axis=0, return_counts=True)
    p = counts / N
    # Entropy in bits; at most log2(N) when all codes differ
    return -sum(p * log2(p))

The repository's bin width differs from the paper's stated rule, so the value depends on which one you follow. The card's Paper vs code section lists the details. The AD-GCL card has its own pseudocode for the learned augmentation.

Contradictions

• Compression as the cause of the phase transition. IB theory attributes the phase transition to saturation-driven compression. The paper notes that the transition has been replicated with ReLU, which is not bounded 2403.18494 §3.3.
• Compression and generalisation. The paper lists ongoing debate about the compression phenomenon and its contribution to generalisation, citing several works 2403.18494 §1.3. Its own run shows the more-compressing vanilla model generalises worse 2403.18494 §3.3.
• What the measured compression means. IB theory reads compression as information lost about the input. The paper finds some lossy compression after the transition, but reads it mainly as saturation of the activations. It reports that most information is still present in all layers 2403.18494 §3.3.
• Binning scheme. The paper reports that absolute binning, used in some earlier IB studies, gave mixed results, and it prefers relative binning 2403.18494 §3.3. Those earlier studies are not on cards.
• Two uses of IB. One card uses IB as a phase and compression analysis of training 2403.18494 §3.3. The other uses it as a theorem that motivates augmentation 2106.05819 §3.1. Neither card tests the other's claim.

The debates above involve other works that have no card here. They are named in the papers' bibliographies and are not summarised on this page.

Papers

• Learning in PINNs: Phase transition, diffusion equilibrium, and generalization: applies IB phases and layer compression to full-batch Adam on PINNs, and proposes residual-based attention.
• Adversarial Graph Augmentation to Improve Graph Contrastive Learning: uses a graph IB theorem to motivate AD-GCL, a learned edge-drop augmentation for graph contrastive learning.