Gradient conflict resolution

Overview

When one network learns from several losses, their gradients can point in different directions or differ widely in size. Following their average can then help one task while hurting another, often the hardest one 2110.14048 §1 2110.14048 §2.1.

The methods on this page change how the task gradients are merged into one update. The shared idea is to keep the average gradient as the starting point and move only a bounded distance from it, toward a direction that improves the worst task 2110.14048 §3.1 2110.14048 eq. 3.

Only one paper card covers this slug so far, so the page describes one method, CAGrad. The other methods the paper compares against (MGDA, PCGrad, GradDrop) are named in its tables but have no cards here, so they are not described.

Variants

CAGrad

The update is the average gradient plus a term of fixed length c‖g_0‖ along the normalised blend of task gradients 2110.14048 §3.1. The blend weights are found by a small problem over the tasks, not over the parameters 2110.14048 §3.1 app. A.3. The one hyperparameter c lies in [0, 1) for the convergence guarantee. At c = 0 the rule is plain gradient descent on the average loss 2110.14048 §3.1 2110.14048 alg. 1. Card: Conflict-Averse Gradient Descent for Multi-task Learning.

Author-reported results, where lower Δm is better 2110.14048 tab. 1:

Results on the RL and semi-supervised benchmarks are in 2110.14048 tab. 3 and 2110.14048 tab. 4. On Meta-World MT10 the paper reports success 0.83 for CAGrad against 0.72 for PCGrad, and on MT50 0.52 against 0.50 2110.14048 tab. 3. CAGrad uses no task metadata. CARE, which does use task language metadata, reaches 0.84 on MT10 2110.14048 tab. 3.

CAGrad-Fast (sampled tasks)

Each step, a random subset S of tasks is sampled. Their gradients are computed, and the unsampled tasks are replaced by their average. The same problem is then solved 2110.14048 §3.3 2110.14048 eq. 4. The paper uses |S| = 4 on the 10-task RL benchmark and |S| = 8 on the 50-task one [arxiv:2110.14048 app. B.3]. Per-step times are 4.8 s for CAGrad-Fast and 10.3 s for full CAGrad on the 10-task setup, and 11.4 s against 27.8 s on the 50-task setup 2110.14048 tab. 7.

When to use which

The margins are small in places. CityScapes segmentation is 75.16 mIoU for CAGrad against 75.18 for MTAN 2110.14048 tab. 2, and the significance test uses p = 0.1 with three seeds 2110.14048 tab. 1. Treat the differences as the authors' result on their own setups, not a general ranking.

Implementation sketch

Restated from the paper's algorithm, with the citations kept 2110.14048 alg. 1:

# Average of the task gradients, the main objective  [§2.1]
g0 = mean(grads)
# Search radius, c times the norm of g0  [§3.1] [eq. 3]
radius = c * norm(g0)
# Task weights on the simplex; the paper leaves the solver open  [§3.1]
w = solve_weights(grads, g0, radius)
# Blended task gradient  [§3.1]
g_w = sum(w_i * g_i for w_i, g_i in zip(w, grads))
# Update direction: average plus a bounded push toward g_w  [§3.1] [alg. 1]
d = g0 + radius * g_w / norm(g_w)
# Plain step, as in the paper's algorithm  [alg. 1]
theta = theta - alpha * d

For CAGrad-Fast, replace the full gradient list with the sampled subset and replace the unsampled part by its average, as in 2110.14048 eq. 4. Guard the division by the norm of g_w with a small constant; the paper does not address g_w = 0 2110.14048 §3.1 (see Pitfalls).

Contradictions

• Convergence for c ≥ 1. The main text says fixed points are Pareto-stationary for c ≥ 1 2110.14048 §3.2. The appendix says any c ≥ 0 2110.14048 app. A.3. Use the appendix version only with its proof in hand.
• CityScapes c. The text says c = 0.2 2110.14048 §5.2. The appendix and the reported table row say c = 0.4 2110.14048 app. B.2 2110.14048 tab. 2. The authors' code runs c = 0.4 and c = 0.6 run.sh. The repository does not show c = 0.2.
• Selection of c. The supervised text picks c by average training performance 2110.14048 app. B.2. The appendix sweep on CityScapes gives Δm 10.74 at c = 0.8, lower than the 11.64 reported at c = 0.4 2110.14048 tab. 6 2110.14048 tab. 2.

Pitfalls

• The theorem and the experiments differ. Theorem 1 covers the plain step θ − αd with α ≤ 1/H 2110.14048 §3.2 2110.14048 alg. 1. The supervised, toy and RL runs feed the direction into Adam or SAC optimisers 2110.14048 §5.2 2110.14048 §5.3.
• Undisclosed scaling in the code. The paper's update has no extra scale 2110.14048 alg. 1. The authors' code divides it by 1 + c² in the supervised setups utils.py. At c = 0.4 this shrinks the step by about 14%. Results reproduced from the paper alone may differ.
• Degenerate blend. If g_w = 0 the update is undefined. The paper does not say what to do; the code adds small constants 2110.14048 §3.1.
• Compute per step. Each step needs one backward pass per task. The RL task-weight solve adds up to 20 inner steps 2110.14048 app. B.3.
• Best-epoch reporting in RL. The RL setup reports the highest average test score seen during training 2110.14048 app. B.3. This is optimistic, and no held-out selection is reported.
• Unstated weights. The semi-supervised auxiliary weights are not given in the paper 2110.14048 §5.4.

Papers

• Conflict-Averse Gradient Descent for Multi-task Learning: introduces CAGrad and its sampled variant CAGrad-Fast.