Learning-rate schedules

Overview

In theory, the problem is: given a horizon of N steps, choose the step size η(k) at each step so that the final iterate has low excess risk. The one-pass SGD setting is the one analysed in 2602.06797 §3.1. Its shared structure is a trade-off. The total step size, called intrinsic time, drives how much signal is learned. The step size at each step injects noise, and later steps forget part of that noise 2602.06797 §3.2, eq. 3. The invariant the theory variants keep is a stability cap on the step size, η ≤ η_stab, with η_stab of order 1 2602.06797 eq. 8.

In practice, most recipes fix the run length in epochs and shape the rate over it. One is a linear warm-up followed by cosine annealing with restarts 2110.06207 app. D. Another is the one-cycle policy of a deep learning library, which warms the rate up, anneals it, and moves momentum the other way 2002.04688 §3.1. A third is step decay after a warm-up, with a peak that scales with the number of workers 1808.07217 app. A.3. A fourth holds the rate flat and then decays it with cosine to a fixed floor 2303.07399 §3.2. A fifth sets the rate by a validation rule rather than a horizon. One dataset paper trains a fine-tuned image classifier that way 1810.05726 §Deep Learning. Its rule is the last variant below.

This page rests on seven papers. The theory comes from one paper: a linear model on power-law features with constant label noise [arxiv:2602.06797 §3.1]. Its rates hold up to constants, and the paper states no numeric value for the peak or the cap 2602.06797 thm. 4.1. The six practice sources are a library paper, an open-set recognition paper, a dataset paper, a distributed-training paper, a pose-estimation paper and a tutorial survey of the Transformer 2002.04688 §3.1 2110.06207 app. D 1810.05726 §Deep Learning 1808.07217 app. A.3 2303.07399 §3.2 2311.17633 §2.7. None of them tests the theory's schedules, so the recipes below are not checks of the theory. Section numbers for 2602.06797 follow the LaTeX source the card was compiled from, which is arXiv v1; the metadata records v3, so check them against the v3 PDF before citing outside wikillm.

Variants

Power decay from step 0 (easy tasks)

For easy tasks, where s > 1 − 1/q, the step size decays from the first step with a power law in the remaining fraction of the run [arxiv:2602.06797 thm. 4.1]. The peak scales with the horizon as N^(−(s−1+1/q)/(s+1/q)), and the final-step loss reaches the minimax rate N^(−sq/(sq+1)) [arxiv:2602.06797 eq. 9]. Card: Li et al. 2026, final-step-optimal schedules.

Hold, then decay (hard tasks)

For hard tasks, where s < 1 − 1/q, the optimum holds the stability limit for most of the run and decays only over its last fraction r_N. That fraction shrinks with N but is only fixed up to a constant [arxiv:2602.06797 thm. 4.1 eq. 10]. The final-step loss is of order N^(−s) 2602.06797 thm. 4.1. Without the cap, the unconstrained peak diverges as N grows [arxiv:2602.06797 §8.1]. Same card as above: paper.

Discrete power decay for SGD

The schedule the implementer can run directly is η_k = η_0 (1 − k/N)^p. For easy tasks it needs p > q − 1 and η_0 of order N^(−(sq−q+1)/(sq+1)). It then reaches a final-step error of order N^(−sq/(sq+1)) with no log factor 2602.06797 thm. 6.3. For hard tasks it takes η_0 of order 1 and p > s/(1 − s), with error of order N^(−s) 2602.06797 thm. 6.3. The authors say the exact exponent 2q − 1 from §4 is not required for the rate 2602.06797 §5.1.

Horizon-normalised profiles

A schedule written as η̄ ζ(z/N), where ζ has unit integral and tends to zero like (1 − x)^p at the end, is a family that can be tuned per horizon 2602.06797 §5.1 eq. 14. The theory says that if p + 1 > q, some member reaches the optimal rate once its level T is tuned. If p + 1 < q, the rate stops improving as q grows; the paper calls this schedule-induced saturation 2602.06797 thm. 5.2 thm. 5.3 §5.1. Cosine (p = 2) and the 1 − √ shape (p = 1) are examples 2602.06797 §5.1.

Absolute-step profiles (avoid for the optimal rate)

A profile defined in absolute step counts, such as polynomial decay η_0 (1 + z)^(−a), cannot reach the optimal rate for any non-increasing shape in the regime the theorem states 2602.06797 thm. 5.4 §5.2. Define the profile on z/N instead.

Warm-up, then inverse-square-root decay (Transformer schedule)

The tutorial survey gives the warm-up schedule used for training the Transformer. The rate is eta0 times the smaller of the inverse square root of the step and the step times warm-up^(−1.5). It therefore rises linearly during warm-up and then falls as the inverse square root of the step. The peak is eta0 divided by the square root of the warm-up length, reached at the last warm-up step 2311.17633 §2.7. The survey states no value for eta0, for the warm-up length, or for the Adam settings it pairs with the schedule, and it does not say which step index is the first 2311.17633 §2.7. The survey runs no experiments, so the card gives no evidence that this form beats the others on this page Xiao and Zhu 2023, Transformer introduction.

Warm-up, then cosine annealing with restarts (fixed epoch budget)

The open-set recognition paper trains a VGG32 backbone with a linear warm-up and then cosine annealing with two warm restarts, at epochs 200 and 400 of a 600-epoch run. The base rate is 0.1, or 0.01 on TinyImageNet. The rate never falls below the base rate times 1e-3 2110.06207 app. D. The text ends warm-up at epoch 20, but the released code uses a shorter warm-up, so the card's Paper vs code section records the difference Vaze et al. 2021, open-set recognition. The paper does not name its optimiser in the text; the code uses SGD with momentum 0.9, and Adam on TinyImageNet 2110.06207 app. D.

Its ablation changes the epoch count together with the scheduler: a 100-epoch step-decay run is compared with 200- and 600-epoch cosine runs. The effect of the schedule alone is therefore not isolated 2110.06207 app. C.

Step decay with a batch-scaled peak (data parallel)

With K workers, the peak rate is K times the single-worker rate. The rate rises linearly to that peak over the first 5 epochs, then drops by a factor of 10 at 50% and 75% of the samples 1808.07217 app. A.3 [arxiv:1808.07217 app. A.4.1]. The single-worker rate is tuned per model. For its large-batch baselines the paper also grid-searches the multiplier around K [arxiv:1808.07217 app. A.4.1].

The same schedule carries over to post-local SGD. That method runs mini-batch SGD until the first decay and then switches to local SGD, with H local steps between model averages 1808.07217 §3. The paper ties the switch to the first decay. It gives no rule for other schedules, so a different schedule needs its own switch point 1808.07217 §3 footnote. Card: Lin et al. 2018, post-local SGD.

Flat, then cosine to a floor (long fine-tune)

The pose estimator trains with AdamW at base rate 0.004 for batch 1024. A linear warm-up runs over 1000 iterations from 1e-5 times the base. The rate then stays constant until epoch 210 of a 420-epoch fine-tune, and decays by cosine to 0.05 times the base 2303.07399 §3.2 [arxiv:2303.07399 tab. 7]. The paper's step-by-step ablation adds this schedule on top of earlier changes, so its gain is not measured against the baseline alone [arxiv:2303.07399 §3.2]. Card: Jiang et al. 2023, RTMPose.

One-cycle policy (warm-up and anneal, momentum mirrored)

The rate warms up to a peak over the first part of the run, then anneals down. Momentum moves the other way over the same two phases [arxiv:2002.04688 §3.1] fig. 8. An example run sets a peak of 2e-2, with momentum (0.8, 0.7, 0.8), for one epoch on a text task 2002.04688 §2.2. The paper gives no numeric defaults for the split point, the start and end ratios or the momentum range. The card lists the library's code defaults Howard and Gugger 2020, fastai.

Learning-rate range test (picks the peak, not a schedule)

Before the run, a short mock training raises the rate exponentially from 1e-7 to 10 over 100 iterations. A suggestion is then read off the loss curve 2002.04688 §3.1 fig. 9. The figure caption gives the rule as the minimum-loss rate divided by 10. The released code's default picks a different point, so the value it returns can differ from the plot Howard and Gugger 2020, fastai. Choose the suggestion rule explicitly.

Halve on validation plateau (practice)

The learning rate starts at 1e-4 with Adam. It is halved whenever the validation loss has not improved for 16 epochs. Training stops after 32 epochs without improvement, then restarts from the best weights with a lower initial rate 1810.05726 §Deep Learning. The paper gives no value for the restart rate. The rule has no horizon N and needs a validation set. It is the training recipe for a fine-tuned classifier on a weed-image dataset, and the paper reports no comparison of this schedule with alternatives 1810.05726 §Deep Learning. Card: Olsen et al. 2018, weed-image dataset.

When to use which

Task regime depends on s and q, which the paper treats as inputs with no default 2602.06797 §3.1. The page has no rule for estimating them from data.

Author-reported experiments use one fixed grid of decay ratios per setting, and the best ratio moves with the setting. In a 2-layer ReLU MLP with peak rate 0.1, the best ratio was 0.5 for γ = 0.6 and 1 for γ = 10 2602.06797 §7.1 fig. 2. ResNet-18 on CIFAR-Full, at batch 20 and peak rate 0.2, had its best ratio at 0.4. On CIFAR-Binary it was 1 [arxiv:2602.06797 §7.2 fig. 3]. Changing peak rate or batch size moves these ratios 2602.06797 app. D.2 app. D.3, so a grid search is needed per setting. These runs use stable-linear decay, not the (1 − u)^(2q−1) shape of §4, and have 3 to 32 runs per point 2602.06797 §7. The paper describes them as consistent with the theory, not as tight fits 2602.06797 §7. They do not separate task dependence from the other settings [arxiv:2602.06797 app. D.2 app. D.3].

Implementation sketch

The paper gives no code and no finite-N rule for δ_N 2602.06797 §4. The sketches below leave the peak and the cap as inputs. Tune them.

# Power decay over a fixed horizon  [thm. 6.3]
# Inputs: N steps, exponent p, prefactor eta0 (not stated: tune it)
def power_decay_lr(k, N, eta0, p):
    # Step k in [0, N); the rate reaches 0 at k = N
    return eta0 * (1 - k / N) ** p

# Hold the stability cap, then decay  [thm. 4.1]
# eta_stab is order 1 [eq. 8]: set it from tuning
# decay_ratio r is order N^(-something) with an unstated constant: grid it
def hold_then_decay_lr(k, N, eta_stab, decay_ratio, exponent):
    # Hold the cap for the first (1 - r) fraction of the run
    start = N * (1 - decay_ratio)
    if k <= start:
        return eta_stab
    # Position inside the decay window, from 0 to 1
    progress = (k - start) / (N - start)
    # Optimal shape uses exponent 2q - 1  [thm. 4.1]
    return eta_stab * (1 - progress) ** exponent

# Linear warm-up, then cosine with restarts  [app. D]
# Base rate 0.1 (0.01 on TinyImageNet); floor is base * 1e-3
# Warm-up length is an input: the paper's text and its code differ
# Restart period 200 epochs; this is the schedule the text describes,
# not the first-epoch behaviour of the released code
def warmup_cosine_restart_lr(e, base, warmup_epochs):
    # Linear ramp up to the base rate during warm-up
    if e < warmup_epochs:
        return base * (e + 1) / warmup_epochs
    # Position inside the current 200-epoch cycle, from 0 to 1
    t = (e % 200) / 200
    floor = base * 1e-3
    return floor + 0.5 * (base - floor) * (1 + cos(pi * t))

# Warm-up, then inverse-square-root decay  [§2.7]
# eta0 and warmup are not stated in the survey: tune them
# Steps start at 1 here to avoid step ** -0.5 at step 0; this is our choice
def transformer_warmup_lr(step, eta0, warmup):
    step = max(step, 1)
    # Linear growth to the peak at step == warmup, then inverse-sqrt decay
    return eta0 * min(step ** -0.5, step * warmup ** -1.5)

Contradictions

Within the theory paper, the easy/hard boundary is handled two ways. §4 says scaling alone leaves the boundary s = 1 − 1/q undetermined [arxiv:2602.06797 §4]. §6 writes the easy regime as s ≥ 1 − 1/q 2602.06797 thm. 6.3.

Across papers: none known. The DeepWeeds rule and the recipes above are not tests of the theory's schedules, so they do not contradict them. The tutorial survey gives no numbers for its schedule and reports no experiments, so it adds no conflicting result.

Within the distributed-training paper, the main text and the appendix disagree. The text averages the local models after each round 1808.07217 §3. The appendix pseudocode for its algorithms applies the rate again to the averaged difference 1808.07217 alg. 1 1808.07217 alg. 2. Read literally, that double-scales the update.

Within one paper's release, the text and the code differ on details. The open-set recognition paper's warm-up length differs between its text and its code, and its code runs the first epoch at the floor rate, which the text does not describe Vaze et al. 2021, open-set recognition. The fastai paper's range-test suggestion rule differs between its figure caption and its code default Howard and Gugger 2020, fastai. These are implementation differences, not conflicting results.

Papers

• Li et al. 2026, final-step-optimal schedules: derives the final-step-optimal schedule for one-pass SGD and the power-decay schedule with its rates.
• Howard and Gugger 2020, fastai: a library paper with the one-cycle policy and a learning-rate range test; it reports worked examples, not benchmark tables.
• Vaze et al. 2021, open-set recognition: a closed-set training recipe with warm-up and cosine restarts, and an ablation that changes epochs and scheduler together.
• Olsen et al. 2018, weed-image dataset: a dataset paper whose training recipe halves the Adam learning rate on a validation plateau, with no comparison of schedules.
• Lin et al. 2018, post-local SGD: a step-decay recipe with a batch-scaled peak and warm-up, and a switch to local SGD at the first decay.
• Jiang et al. 2023, RTMPose: a pose estimator trained with AdamW and a flat-then-cosine rate, with a fixed warm-up.
• Xiao and Zhu 2023, Transformer introduction: a tutorial survey whose training section gives the warm-up then inverse-square-root schedule; no experiments and no numeric defaults for it.