---
kind: concept
slug: learning-rate-schedule
title: Learning-rate schedules
summary: >-
  How to set the step size over a training run. Theory for one-pass SGD with a known horizon: decay from step 0 on easy tasks, or hold the stability limit and decay at the end on hard tasks. Practice: warm-up then cosine or one-cycle, step decay with a batch-scaled peak, flat-then-cosine, or halving on validation plateau. Constants are often not stated, so the peak must be tuned.
tags: [optimization, training, scaling]
papers: ["2602.06797", "1810.05726", "2002.04688", "2110.06207", "1808.07217", "2303.07399", "2311.17633"]
related: [neural-scaling-laws, gradient-descent-stability, transfer-learning]
status: draft
updated_at: 2026-11-01
---

## Overview

In theory, the problem is: given a horizon of N steps, choose the step size
η(k) at each step so that the final iterate has low excess risk. The one-pass
SGD setting is the one analysed in [arxiv:2602.06797 §3.1]. Its shared
structure is a trade-off. The total step size, called intrinsic time, drives
how much signal is learned. The step size at each step injects noise, and
later steps forget part of that noise [arxiv:2602.06797 §3.2, eq. 3]. The
invariant the theory variants keep is a stability cap on the step size,
η ≤ η_stab, with η_stab of order 1 [arxiv:2602.06797 eq. 8].

In practice, most recipes fix the run length in epochs and shape the rate
over it. One is a linear warm-up followed by cosine annealing with restarts
[arxiv:2110.06207 app. D]. Another is the one-cycle policy of a deep learning
library, which warms the rate up, anneals it, and moves momentum the other way
[arxiv:2002.04688 §3.1]. A third is step decay after a warm-up, with a peak
that scales with the number of workers [arxiv:1808.07217 app. A.3]. A fourth
holds the rate flat and then decays it with cosine to a fixed floor
[arxiv:2303.07399 §3.2]. A fifth sets the rate by a validation rule rather than
a horizon. One dataset paper trains a fine-tuned image classifier that way
[arxiv:1810.05726 §Deep Learning]. Its rule is the last variant below.

This page rests on seven papers. The theory comes from one paper: a linear
model on power-law features with constant label noise [arxiv:2602.06797
§3.1]. Its rates hold up to constants, and the paper states no numeric value
for the peak or the cap [arxiv:2602.06797 thm. 4.1]. The six practice
sources are a library paper, an open-set recognition paper, a dataset paper,
a distributed-training paper, a pose-estimation paper and a tutorial survey of
the Transformer [arxiv:2002.04688 §3.1] [arxiv:2110.06207 app. D]
[arxiv:1810.05726 §Deep Learning] [arxiv:1808.07217 app. A.3]
[arxiv:2303.07399 §3.2] [arxiv:2311.17633 §2.7]. None of them tests the
theory's schedules, so the recipes below are
not checks of the theory. Section numbers for 2602.06797 follow the LaTeX
source the card was compiled from, which is arXiv v1; the metadata records v3,
so check them against the v3 PDF before citing outside wikillm.

## Variants

### Power decay from step 0 (easy tasks)

For easy tasks, where s > 1 − 1/q, the step size decays from the first step
with a power law in the remaining fraction of the run [arxiv:2602.06797 thm.
4.1]. The peak scales with the horizon as N^(−(s−1+1/q)/(s+1/q)), and the
final-step loss reaches the minimax rate N^(−sq/(sq+1)) [arxiv:2602.06797 eq.
9]. Card: [Li et al. 2026, final-step-optimal schedules](/papers/2602.06797).

### Hold, then decay (hard tasks)

For hard tasks, where s < 1 − 1/q, the optimum holds the stability limit for
most of the run and decays only over its last fraction r_N. That fraction
shrinks with N but is only fixed up to a constant [arxiv:2602.06797 thm. 4.1
eq. 10]. The final-step loss is of order N^(−s) [arxiv:2602.06797 thm. 4.1].
Without the cap, the unconstrained peak diverges as N grows [arxiv:2602.06797
§8.1]. Same card as above: [paper](/papers/2602.06797).

### Discrete power decay for SGD

The schedule the implementer can run directly is η_k = η_0 (1 − k/N)^p.
For easy tasks it needs p > q − 1 and η_0 of order N^(−(sq−q+1)/(sq+1)). It
then reaches a final-step error of order N^(−sq/(sq+1)) with no log factor
[arxiv:2602.06797 thm. 6.3]. For hard tasks it takes η_0 of order 1 and
p > s/(1 − s), with error of order N^(−s) [arxiv:2602.06797 thm. 6.3]. The
authors say the exact exponent 2q − 1 from §4 is not required for the rate
[arxiv:2602.06797 §5.1].

### Horizon-normalised profiles

A schedule written as η̄ ζ(z/N), where ζ has unit integral and tends to zero
like (1 − x)^p at the end, is a family that can be tuned per horizon
[arxiv:2602.06797 §5.1 eq. 14]. The theory says that if p + 1 > q, some
member reaches the optimal rate once its level T is tuned. If p + 1 < q, the
rate stops improving as q grows; the paper calls this schedule-induced
saturation [arxiv:2602.06797 thm. 5.2 thm. 5.3 §5.1]. Cosine (p = 2) and
the 1 − √ shape (p = 1) are examples [arxiv:2602.06797 §5.1].

### Absolute-step profiles (avoid for the optimal rate)

A profile defined in absolute step counts, such as polynomial decay
η_0 (1 + z)^(−a), cannot reach the optimal rate for any non-increasing shape
in the regime the theorem states [arxiv:2602.06797 thm. 5.4 §5.2]. Define
the profile on z/N instead.

### Warm-up, then inverse-square-root decay (Transformer schedule)

The tutorial survey gives the warm-up schedule used for training the
Transformer. The rate is eta0 times the smaller of the inverse square root of
the step and the step times warm-up^(−1.5). It therefore rises linearly during
warm-up and then falls as the inverse square root of the step. The peak is
eta0 divided by the square root of the warm-up length, reached at the last
warm-up step [arxiv:2311.17633 §2.7]. The survey states no value for eta0,
for the warm-up length, or for the Adam settings it pairs with the schedule,
and it does not say which step index is the first [arxiv:2311.17633 §2.7].
The survey runs no experiments, so the card gives no evidence that this form
beats the others on this page [Xiao and Zhu 2023, Transformer
introduction](/papers/2311.17633).

### Warm-up, then cosine annealing with restarts (fixed epoch budget)

The open-set recognition paper trains a VGG32 backbone with a linear warm-up
and then cosine annealing with two warm restarts, at epochs 200 and 400 of a
600-epoch run. The base rate is 0.1, or 0.01 on TinyImageNet. The rate never
falls below the base rate times 1e-3 [arxiv:2110.06207 app. D]. The text
ends warm-up at epoch 20, but the released code uses a shorter warm-up, so the
card's Paper vs code section records the difference [Vaze et al. 2021,
open-set recognition](/papers/2110.06207). The paper does not name its
optimiser in the text; the code uses SGD with momentum 0.9, and Adam on
TinyImageNet [arxiv:2110.06207 app. D].

Its ablation changes the epoch count together with the scheduler: a 100-epoch
step-decay run is compared with 200- and 600-epoch cosine runs. The effect of
the schedule alone is therefore not isolated [arxiv:2110.06207 app. C].

### Step decay with a batch-scaled peak (data parallel)

With K workers, the peak rate is K times the single-worker rate. The rate
rises linearly to that peak over the first 5 epochs, then drops by a factor
of 10 at 50% and 75% of the samples [arxiv:1808.07217 app. A.3] [arxiv:1808.07217
app. A.4.1]. The single-worker rate is tuned per model. For its large-batch
baselines the paper also grid-searches the multiplier around K [arxiv:1808.07217
app. A.4.1].

The same schedule carries over to post-local SGD. That method runs mini-batch
SGD until the first decay and then switches to local SGD, with H local steps
between model averages [arxiv:1808.07217 §3]. The paper ties the switch to the
first decay. It gives no rule for other schedules, so a different schedule
needs its own switch point [arxiv:1808.07217 §3 footnote]. Card: [Lin et al.
2018, post-local SGD](/papers/1808.07217).

### Flat, then cosine to a floor (long fine-tune)

The pose estimator trains with AdamW at base rate 0.004 for batch 1024. A
linear warm-up runs over 1000 iterations from 1e-5 times the base. The rate
then stays constant until epoch 210 of a 420-epoch fine-tune, and decays by
cosine to 0.05 times the base [arxiv:2303.07399 §3.2] [arxiv:2303.07399 tab.
7]. The paper's step-by-step ablation adds this schedule on top of earlier
changes, so its gain is not measured against the baseline alone [arxiv:2303.07399
§3.2]. Card: [Jiang et al. 2023, RTMPose](/papers/2303.07399).

### One-cycle policy (warm-up and anneal, momentum mirrored)

The rate warms up to a peak over the first part of the run, then anneals
down. Momentum moves the other way over the same two phases [arxiv:2002.04688
§3.1] [fig. 8]. An example run sets a peak of 2e-2, with momentum (0.8, 0.7,
0.8), for one epoch on a text task [arxiv:2002.04688 §2.2]. The paper gives
no numeric defaults for the split point, the start and end ratios or the
momentum range. The card lists the library's code defaults
[Howard and Gugger 2020, fastai](/papers/2002.04688).

### Learning-rate range test (picks the peak, not a schedule)

Before the run, a short mock training raises the rate exponentially from 1e-7
to 10 over 100 iterations. A suggestion is then read off the loss curve
[arxiv:2002.04688 §3.1] [fig. 9]. The figure caption gives the rule as the
minimum-loss rate divided by 10. The released code's default picks a
different point, so the value it returns can differ from the plot [Howard and
Gugger 2020, fastai](/papers/2002.04688). Choose the suggestion rule
explicitly.

### Halve on validation plateau (practice)

The learning rate starts at 1e-4 with Adam. It is halved whenever the
validation loss has not improved for 16 epochs. Training stops after 32 epochs
without improvement, then restarts from the best weights with a lower initial
rate [arxiv:1810.05726 §Deep Learning]. The paper gives no value for the
restart rate. The rule has no horizon N and needs a validation set. It is the
training recipe for a fine-tuned classifier on a weed-image dataset, and the
paper reports no comparison of this schedule with alternatives
[arxiv:1810.05726 §Deep Learning]. Card: [Olsen et al. 2018, weed-image
dataset](/papers/1810.05726).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Easy task (s > 1 − 1/q) and you can tune the peak for each horizon N | Power decay from step 0, p > q − 1 | Reaches the minimax rate with no log factor [arxiv:2602.06797 thm. 6.3] |
| Hard task (s < 1 − 1/q), stability cap known | Hold the cap, then decay over a fraction r_N | The optimum has this shape [arxiv:2602.06797 thm. 4.1]; r_N needs a grid search, as the authors did [arxiv:2602.06797 §7.1] |
| Same peak reused across horizons | Not covered | The easy-task rate needs η_0 to scale with N [arxiv:2602.06797 thm. 6.3]. The experiments tuned the peak per N [arxiv:2602.06797 fig. 1] |
| Cosine decay (p = 2) in the theory's terms | Fine only if q < 3 | With p + 1 = 3, the theory predicts the rate saturates when q > 3 [arxiv:2602.06797 thm. 5.2]. The recipe below uses cosine without this analysis |
| Fixed epoch budget, image classifier, want a standard recipe | Warm-up, then cosine with restarts; base 0.1, or 0.01 on TinyImageNet | Used for a 600-epoch run on standard benchmarks [arxiv:2110.06207 app. D]; the schedule's own effect is not isolated [arxiv:2110.06207 app. C] |
| Fixed short run, want a fast default, peak rate unknown | One-cycle, with the range test for the peak | Used for short training examples [arxiv:2002.04688 §2.2] [fig. 9]; the library paper has no benchmark tables [Howard and Gugger 2020, fastai](/papers/2002.04688) |
| Transformer-style model, want the warm-up then inverse-square-root form | Warm-up then inverse-square-root decay; peak eta0 / sqrt(warm-up) | Formula given [arxiv:2311.17633 §2.7]; no experiments, and eta0 and warm-up must be chosen |
| Schedule written in absolute steps | Avoid, if the optimal rate matters | Cannot reach the optimal rate [arxiv:2602.06797 thm. 5.4] |
| Boundary case s = 1 − 1/q | Undetermined | Scaling alone does not decide it [arxiv:2602.06797 §4] |
| Training with a validation set, no known horizon N, Adam | Halve on validation plateau, stop after 32 stalled epochs | Simple to run; the paper used it but compared no alternatives [arxiv:1810.05726 §Deep Learning] |
| Data-parallel SGD, global batch grows with the number of workers | Peak × K, 5-epoch linear warm-up, ×0.1 at 50% and 75% | The large-batch baseline recipe [arxiv:1808.07217 app. A.3] [arxiv:1808.07217 app. A.4.1]; the multiplier needed a grid search per baseline [arxiv:1808.07217 app. A.4.1] |
| Distributed SGD that should cut communication, first decay known | Post-local SGD: mini-batch until the first decay, then local steps | Savings come only after the switch; the first phase still syncs every step [arxiv:1808.07217 §3] [arxiv:1808.07217 §4.2] |
| Long AdamW run where the rate should stay high, then anneal to a floor | Flat, then cosine to 0.05 times the base | Used for a 420-epoch fine-tune [arxiv:2303.07399 tab. 7]; the ablation is cumulative, so the schedule's own effect is not isolated [arxiv:2303.07399 §3.2] |

Task regime depends on s and q, which the paper treats as inputs with no
default [arxiv:2602.06797 §3.1]. The page has no rule for estimating them from
data.

Author-reported experiments use one fixed grid of decay ratios per setting,
and the best ratio moves with the setting. In a 2-layer ReLU MLP with peak
rate 0.1, the best ratio was 0.5 for γ = 0.6 and 1 for γ = 10
[arxiv:2602.06797 §7.1 fig. 2]. ResNet-18 on CIFAR-Full, at batch 20 and peak
rate 0.2, had its best ratio at 0.4. On CIFAR-Binary it was 1 [arxiv:2602.06797
§7.2 fig. 3]. Changing peak rate or batch size moves these ratios
[arxiv:2602.06797 app. D.2 app. D.3], so a grid search is needed per setting.
These runs use stable-linear decay, not the (1 − u)^(2q−1) shape of §4, and
have 3 to 32 runs per point [arxiv:2602.06797 §7]. The paper describes them as
consistent with the theory, not as tight fits [arxiv:2602.06797 §7]. They do
not separate task dependence from the other settings [arxiv:2602.06797 app.
D.2 app. D.3].

## Implementation sketch

The paper gives no code and no finite-N rule for δ_N [arxiv:2602.06797 §4].
The sketches below leave the peak and the cap as inputs. Tune them.

```pseudocode
# Power decay over a fixed horizon  [thm. 6.3]
# Inputs: N steps, exponent p, prefactor eta0 (not stated: tune it)
def power_decay_lr(k, N, eta0, p):
    # Step k in [0, N); the rate reaches 0 at k = N
    return eta0 * (1 - k / N) ** p
```

```pseudocode
# Hold the stability cap, then decay  [thm. 4.1]
# eta_stab is order 1 [eq. 8]: set it from tuning
# decay_ratio r is order N^(-something) with an unstated constant: grid it
def hold_then_decay_lr(k, N, eta_stab, decay_ratio, exponent):
    # Hold the cap for the first (1 - r) fraction of the run
    start = N * (1 - decay_ratio)
    if k <= start:
        return eta_stab
    # Position inside the decay window, from 0 to 1
    progress = (k - start) / (N - start)
    # Optimal shape uses exponent 2q - 1  [thm. 4.1]
    return eta_stab * (1 - progress) ** exponent
```

```pseudocode
# Linear warm-up, then cosine with restarts  [app. D]
# Base rate 0.1 (0.01 on TinyImageNet); floor is base * 1e-3
# Warm-up length is an input: the paper's text and its code differ
# Restart period 200 epochs; this is the schedule the text describes,
# not the first-epoch behaviour of the released code
def warmup_cosine_restart_lr(e, base, warmup_epochs):
    # Linear ramp up to the base rate during warm-up
    if e < warmup_epochs:
        return base * (e + 1) / warmup_epochs
    # Position inside the current 200-epoch cycle, from 0 to 1
    t = (e % 200) / 200
    floor = base * 1e-3
    return floor + 0.5 * (base - floor) * (1 + cos(pi * t))
```

```pseudocode
# Warm-up, then inverse-square-root decay  [§2.7]
# eta0 and warmup are not stated in the survey: tune them
# Steps start at 1 here to avoid step ** -0.5 at step 0; this is our choice
def transformer_warmup_lr(step, eta0, warmup):
    step = max(step, 1)
    # Linear growth to the peak at step == warmup, then inverse-sqrt decay
    return eta0 * min(step ** -0.5, step * warmup ** -1.5)
```

## Contradictions

Within the theory paper, the easy/hard boundary is handled two ways. §4 says
scaling alone leaves the boundary s = 1 − 1/q undetermined [arxiv:2602.06797
§4]. §6 writes the easy regime as s ≥ 1 − 1/q [arxiv:2602.06797 thm. 6.3].

Across papers: none known. The DeepWeeds rule and the recipes above are not
tests of the theory's schedules, so they do not contradict them. The tutorial
survey gives no numbers for its schedule and reports no experiments, so it
adds no conflicting result.

Within the distributed-training paper, the main text and the appendix disagree.
The text averages the local models after each round [arxiv:1808.07217 §3].
The appendix pseudocode for its algorithms applies the rate again to the
averaged difference [arxiv:1808.07217 alg. 1] [arxiv:1808.07217 alg. 2]. Read
literally, that double-scales the update.

Within one paper's release, the text and the code differ on details. The
open-set recognition paper's warm-up length differs between its text and its
code, and its code runs the first epoch at the floor rate, which the text does
not describe [Vaze et al. 2021, open-set recognition](/papers/2110.06207). The fastai paper's range-test suggestion
rule differs between its figure caption and its code default [Howard and
Gugger 2020, fastai](/papers/2002.04688). These are implementation
differences, not conflicting results.

## Papers

- [Li et al. 2026, final-step-optimal schedules](/papers/2602.06797): derives the final-step-optimal schedule for one-pass SGD and the power-decay schedule with its rates.
- [Howard and Gugger 2020, fastai](/papers/2002.04688): a library paper with the one-cycle policy and a learning-rate range test; it reports worked examples, not benchmark tables.
- [Vaze et al. 2021, open-set recognition](/papers/2110.06207): a closed-set training recipe with warm-up and cosine restarts, and an ablation that changes epochs and scheduler together.
- [Olsen et al. 2018, weed-image dataset](/papers/1810.05726): a dataset paper whose training recipe halves the Adam learning rate on a validation plateau, with no comparison of schedules.
- [Lin et al. 2018, post-local SGD](/papers/1808.07217): a step-decay recipe with a batch-scaled peak and warm-up, and a switch to local SGD at the first decay.
- [Jiang et al. 2023, RTMPose](/papers/2303.07399): a pose estimator trained with AdamW and a flat-then-cosine rate, with a fixed warm-up.
- [Xiao and Zhu 2023, Transformer introduction](/papers/2311.17633): a tutorial survey whose training section gives the warm-up then inverse-square-root schedule; no experiments and no numeric defaults for it.