Gradient clipping

Overview

Clipping rescales a gradient vector so its norm is at most a threshold tau, and leaves it unchanged if it is already shorter. A zero gradient stays zero 2509.14952 §3. The step then uses the clipped vector instead of the raw stochastic gradient. The motivation given in the only card is heavy-tailed gradient noise: now and then one sample yields a gradient far larger than usual, and clipping caps how far that sample can move the iterate 2509.14952 §3.1.

Only one paper with a card lists this concept, so this page describes one method and its stated analysis. It is not a survey of clipping in general.

Variants

Clipped SGD on the inner problems (N2SBA)

Each inner step draws stochastic gradients and clips them before the descent step. Two inner loops run this way: one for the inner variable y and one for the penalty-related variable z 2509.14952 §3.1 eq. 4. The inner iterates are warm-started from the previous outer step alg. 1.

The clip level is not fixed. It is set per outer step t and inner step k, and the paper's theorem uses levels that decay with k, of order exp(-eta lambda mu k) R / eta, with separate radii R_y and R_z 2509.14952 §3.2 thm. 3.1. Card: Stochastic Bilevel Optimization with Heavy-Tailed Noise.

Clipped ascent for minimax (N2SGDA)

For a two-player problem where y is maximized, the inner step is a clipped ascent step instead of a descent step 2509.14952 §4 alg. 2. The same clip operator is used, with a clip level for each inner step alg. 2. The card records no decay rule for these levels.

Normalized outer step (not a clip)

The outer variable x is not clipped. It takes a step of fixed length in the direction of its gradient estimate, x minus eta_x times g over the norm of g 2509.14952 alg. 1. A clipped step keeps a gradient's size up to the cap, while a normalized step discards the size altogether. Pick the one whose guarantee you need; the paper analyses the two together and does not compare them on their own.

When to use which

Practical limits: a clipped step is biased, and the paper does not analyse the bias that clipping introduces 2509.14952 §3. The threshold shrinks as the inner loop proceeds, so inner steps get smaller over time 2509.14952 §3.2. The experiments search step-size grids, but the chosen clip levels are not reported 2509.14952 §5.1.

Implementation sketch

A clip operator and one clipped SGD step on an inner objective. The outer normalized step is not shown. Condensed from 2509.14952 §3 and alg. 1; the sample draws are simplified.

# clip(grad, tau): shrink grad so its norm is at most tau; zero stays zero  [§3]
def clip(grad, tau):
    size = norm(grad)
    if size == 0:
        return zeros_like(grad)
    return min(1.0, tau / size) * grad

# One clipped SGD step on an inner objective  [alg. 1, l. 325-328]
def clipped_inner_step(w, sample, eta, tau):
    # Stochastic gradient of the inner objective at w  [§3.1]
    grad = stochastic_grad(w, sample)
    # Clip first: one heavy-tailed sample moves w by at most eta * tau
    return w - eta * clip(grad, tau)

# Inner loop with a per-step clip level that decays with k  [§3.2] thm. 3.1
def inner_loop(w, samples, eta, taus):
    for k, sample in enumerate(samples):
        w = clipped_inner_step(w, sample, eta, taus[k])
    return w

Implementation notes: the paper's update divides by the norm of the outer gradient, which is undefined when that gradient is zero, so guard it in code 2509.14952 §3.1 alg. 1. The paper gives no numeric values for eta, tau or the decay rate.

Contradictions

None known. Only one source card lists this concept.

Papers

• Stochastic Bilevel Optimization with Heavy-Tailed Noise: clipped inner SGD and a normalized outer step for bilevel problems with heavy-tailed noise, with an SFO complexity bound and a minimax variant.