Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging

In plain words

Teaching a language model to be helpful, honest and harmless usually means training it on a mix of preference data. This paper asks whether it is better to mix the data, or to train one model per goal and then merge the finished weights into one model. It compares the two approaches on the same datasets and finds that merging can match or beat mixing.

The paper also proposes its own merge method, called RESM. It looks at how large each model's weight changes are, and it gives less weight to models whose changes are dominated by a few very large values. It also decides how many singular directions to keep in each layer, based on how sparse that layer's changes are. Both choices aim to reduce the interference between the goals.

TL;DR

• Compares 12 training-free merging methods and 3 data-mixture methods for 3H alignment of two instruction-tuned 7–8B models §5 tab. 2.
• RESM is a task-singular-vector merge with an outlier weight per model and layer and a per-layer rank from a sparsity measure §4 alg. 1.
• Author-reported: best normalised 3H gain of +14.97% on Llama-3-8B-Instruct and +3.36% on Mistral-7B-Instruct-v0.2 tab. 3 tab. 4.
• Code is not released. The implementation details below come from the paper; several equations are ambiguous or internally inconsistent (see Pitfalls).

Problem

Helpfulness, honesty and harmlessness (3H) pull against each other: tuning for one can hurt another §1. The usual fix is data mixing, where the preference datasets are weighted or filtered before one model is trained. This needs expert judgement and can give conflicting training signals §1. Model merging combines the parameters of several specialised models, but its use for 3H was little studied before this paper §1 §2.

Two problems are named for merging 3H models: preference noise accumulating in the merged update, and a single rank threshold that ignores how sparse each layer is §3 §4.

Method

Setup. For a base model θ⁰ and n aligned models θⁱ, the alignment task vector for layer l is Δᵢₗ = θⁱₗ − θ⁰ₗ §3. Task-singular-vector merging (TSVM) takes an SVD of each Δ, keeps the top k singular components and orthogonalises the singular vectors across models before summing §3 eq. 1 eq. 2 eq. 3.

RESM keeps this structure and changes two things §4 eq. 4:

1. Each model's singular values are multiplied by a scalar weight α, computed from its outlier mass in that layer. The weight is normalised across models.
2. The number of kept singular components k_l is set per layer from a sparsity measure, not fixed for all layers.

Steps for one layer, as in alg. 1:

1. Compute Δᵢₗ for every model i alg. 1, Step 1.
2. For each row r, compute the mean and standard deviation of |Δ| over columns eq. 5 eq. 6. Then count the entries with |Δ| at least μ + 3σ and sum their absolute values, giving an outlier mass per model eq. 7. Normalise these masses across models to get αₗ⁽ⁱ⁾ eq. 7 alg. 1, Step 2.
3. Compute a sparsity consensus Ωₗ over all models, which is the share of entries with |Δ| below ε eq. 8. Set kₗ = ⌊dₗ(γ₀ + γΩₗ)⌋ eq. 9 alg. 1, Step 3.
4. For each model, take the SVD, orthogonalise U and V eq. 1 eq. 2, scale S by αₗ⁽ⁱ⁾, keep the first kₗ components alg. 1, Step 4.
5. Sum the truncated terms and add them to θ⁰ₗ eq. 4 alg. 1.

# RESM for one layer l  [alg. 1]
# Input:  base (d_out, d_in); models (list of n aligned matrices)
# Output: merged layer (d_out, d_in)

def resm_merge_layer(base, models, gamma0, gamma, eps):
    # Alignment task vector per model  [alg. 1, Step 1]
    deltas = [w - base for w in models]
    # Outlier mass per model: entries of |delta| at or above the row's
    # mean + 3 sigma (both statistics taken on |delta|)  [eq. 5-7]
    masses = []
    for delta in deltas:
        mu, sigma = row_mean_and_std(abs(delta))
        keep = abs(delta) >= (mu + 3 * sigma)[:, None]
        masses.append(sum(abs(delta)[keep]))
    # Normalise across models, so the alphas sum to 1  [eq. 7]
    alphas = [m / sum(masses) for m in masses]
    # Sparsity consensus over all models and entries  [eq. 8]
    omega = mean([abs(delta) < eps for delta in deltas])
    # Rank for this layer, shared by all models  [eq. 9]
    # d: d_l, not defined for non-square matrices (see Pitfalls)
    k = floor(d * (gamma0 + gamma * omega))
    merged = zeros_like(base)
    for delta, alpha in zip(deltas, alphas):
        # SVD of the task vector  [alg. 1, Step 4]
        u, s, vt = svd(delta)
        # Orthogonal factors, as written in [eq. 1] [eq. 2]
        u_perp = orthogonal_projection(u)
        vt_perp = orthogonal_projection(vt)
        # Outlier weight scales the singular values  [alg. 1, Step 4]
        s_scaled = alpha * s
        # Keep the first k components  [alg. 1, Step 4]
        merged += u_perp[:, :k] @ diag(s_scaled[:k]) @ vt_perp[:k, :]
    # Add the summed truncated task vectors to the base  [eq. 4]
    return base + merged

Key equations, as printed in the paper:

\theta^{\text{RESM}}_l = \theta^0_l + \sum_{i=1}^{n} U^{(i)}_{l\perp}[:,:k_l]\,\bigl(\alpha^{(i)}_l S^{(i)}_l\bigr)\,V^{(i)\top}_{l\perp}[:k_l,:] eq. 4

\mu^{(i)}_{l,r} = \mathbb{E}_c\bigl[|\Delta^{(i)}_{l,r,c}|\bigr], \qquad \sigma^{(i)}_{l,r} = \sqrt{\mathbb{E}_c\bigl[|\Delta^{(i)}_{l,r,c}|^2\bigr] - (\mu^{(i)}_{l,r})^2} eq. 5 eq. 6

\alpha^{(i)}_l = \frac{\sum_{r}\bigl\|\mathrm{Threshold}(\Delta^{(i)}_{l,r,:},\ \mu^{(i)}_{l,r}+3\sigma^{(i)}_{l,r})\bigr\|_1}{\sum_{j=1}^{n}\sum_{r}\bigl\|\mathrm{Threshold}(\Delta^{(j)}_{l,r,:},\ \mu^{(j)}_{l,r}+3\sigma^{(j)}_{l,r})\bigr\|_1} eq. 7

\Omega_l = \frac{1}{n\,d_l^2}\sum_{i=1}^{n}\sum_{r,c}\mathbb{I}\bigl(|\Delta^{(i)}_{l,r,c}|<\epsilon\bigr), \qquad k_l = \bigl\lfloor d_l(\gamma_0+\gamma\,\Omega_l)\bigr\rfloor eq. 8 eq. 9

The paper describes Threshold(M, τ) as zeroing entries whose absolute value is below τ §4.

Overall reasoning given in the paper: the 3σ rule is meant to drop small, noise-like changes and to balance models with large updates §4; the rank rule keeps more components in sparse layers and fewer in dense layers §4.

Hyperparameters

Results

Author-reported. Setup: 3H DPO on ten preference datasets grouped into five annotation dimensions, in the static setting §5 tab. 1. Columns follow the paper: Helpful_Avg, Honest_Avg and Harmless_Avg are the per-dimension averages, and Norm_Gain is the mean of the three per-dimension relative gains against the base instruct model tab. 3.

Other numbers reported in the paper:

• Other merge methods on Llama-3, static setting tab. 3: DELLA-Ties +11.10%, Breadcrumbs-Ties +10.15%, Rewarded Soup +9.60%, Weight Average +7.39%, Task Arithmetic +2.83%, Model Stock +1.68%.
• Outlier pruning applied on top of TSVM, Llama-3 tab. 5: TSVM+Wanda +13.17%, TSVM+OWL +13.47%, RESM +14.97%.
• Ablations, static setting tab. 6 (Llama-3) and tab. 7 (Mistral): without outlier weighting +13.60% and +2.89%; without rank selection +13.71% and +3.12%; full RESM +14.97% and +3.36%.
• Continual setting, sequential DPO tab. 10 (Llama-3) and tab. 11 (Mistral): RESM +14.56% and +2.83%; TSVM +13.47% and +0.27%; the final sequential stage +12.21% and +0.35%. RESM merges from the Instruct checkpoint, not from an intermediate stage §5 app. C.
• Seed stability fig. 5: a Mistral comparison of DARE-Ties and RESM across seeds; sparsity sensitivity over the factor γ fig. 6.

Paper vs code

No official code.

Searched, in the order the skill lists:

• Links in the paper: the abstract points only to a Hugging Face model page (Jinluan), which hosts models, not code. The appendix says code and trained models will be released "upon acceptance" app. C.
• Project page: none linked.
• GitHub search: "RESM merging" and "Reweighting Enhanced Task Singular Merging" both returned no matching repositories. The GitHub account of the first author lists no public repositories.
• The LaTeX source contains a commented-out line pointing to a repository under an older title (AdaR1) and a different author list. Its owner is not an author of this version, and the repository page could not be fetched (gateway timeout), so it was not used.

No code is available to compare against the paper.

Pitfalls

• The outlier weight is one scalar per model and layer. Step 2 yields αₗ⁽ⁱ⁾, which multiplies all singular values of model i in layer l alg. 1. It does not mask individual singular values, even though the text speaks of masking outliers "within S" §3 fig. 2. An implementer who needs a per-direction weighting will not get one from this algorithm.

• Outlier threshold: the appendix and eq. 7 disagree. Eq. 7 keeps entries with |Δ| at or above μ + 3σ, where μ and σ are computed on |Δ| eq. 5 eq. 6. Appendix A describes keeping Δ above μ + 3σ and Δ below μ − 3σ app. A. The second reading has no meaning for |Δ|. The appendix's claim that about 0.3% of entries survive assumes a Gaussian with the threshold applied to Δ itself; the paper does not compute the surviving fraction for eq. 7.

• Per-model orthogonalisation is a no-op as written. Each model's U and V come from its own SVD, so they are already orthonormal. Projecting an orthonormal matrix onto orthonormal matrices returns the same matrix eq. 1 eq. 2 alg. 1, Step 4. The decorrelation across models that TSVM relies on requires stacking the models, which the paper does not spell out. How the paper's TSVM baseline builds U and V is also not restated here.

• Scaling of the sum is unstated. Eq. 4 sums the models with no 1/n factor, but §3 says "wₗ is the scaling factor" for its merge equation. Eq. 4 has no such factor. Whether the paper's runs scale the sum is unknown from the text.

• d_l is undefined for non-square matrices. The commented-out text treats Δ as d_l × d_l, and Ω uses d_l² eq. 8. Eq. 9 uses d_l as a rank scale. The paper never says what d_l means for rectangular projection matrices. This is the main open question for applying RESM to real LLM layers.

• ε = 0.1 has no stated units or scale. Ω counts entries with |Δ| < ε eq. 8. If most task-vector entries are much smaller than 0.1, Ω is close to 1 and k_l is close to 0.8·d_l for almost every layer, which would switch the rank adaptation off. The paper does not report typical |Δ| magnitudes. Check the distribution on your model before trusting γ₀ = 0.2, γ = 0.6.

• Rank bounds. Because Ω ∈ [0, 1], k_l lies between ⌊0.2 d_l⌋ and ⌊0.8 d_l⌋ eq. 9. The same k_l is used for all models in a layer, since Ω pools all models.

• Normalisation can divide by zero. Eq. 7 divides by the outlier mass summed over all models. If that sum is zero in a layer, α is undefined. The paper does not address this case.

• Ordering of rank selection and orthogonalisation. The appendix says orthogonalisation should come before rank selection, and calls this established by "theoretical analysis and empirical validation" app. A. The appendix gives qualitative reasons only, and no table compares the two orders.

• Conflict-bound appendix has an arithmetic error. The bound on the conflict probability uses 2/√k + 2/(√e·√k), which equals 3.21/√k. The paper then states it is at most 2.5/√k app. A. The same symbol ε is used for the conflict threshold (0.3 in the text) and for 1/√k in the proof. The bound is about random unit vectors, not the measured task vectors. The NeurIPS checklist calls this a complete proof app. A; it is a heuristic estimate.

• Stale equation numbers in the appendix. Appendix A refers to "Eq. (3)" for the outlier weight and "Eq. (6)" for the rank. By source order these are eq. 7 and eq. 9 in this card. Check against the compiled PDF before citing.

• The TSVM rank is not given. The baseline's k_fixed is described but its value is not stated in the text, so the main baseline cannot be reproduced from the paper alone §3.

• Merge baselines are not specified. The data-mixture and merge baselines come from MergeKit with no per-method parameters in the paper §5. Reproducing the table needs the MergeKit configs, which are not published.

• Single runs, no error bars. Tables 3 to 7 report one value per cell with no error bars or seed count. Differences in the ablations are small: on Mistral, the full model is 0.47 points above the no-outlier variant. Only fig. 5 shows multiple seeds, for DARE-Ties and RESM on Mistral.

• Judged metrics depend on closed judges. Honesty (HaluEval-Wild), safety (SaladBench), refusal (OR-Bench) and MT-Bench are scored by LLM judges. The judge versions and prompts are not given. The helpfulness benchmarks have no stated evaluation harness or settings app. C.

• How the averages are built. Norm_Gain is the mean of the three per-dimension relative gains. I checked this against the RESM Llama-3 row: (1.41 + 8.79 + 34.72) / 3 = 14.97. Helpful_Avg is an unweighted mean of eight columns, including MT-Bench on a 0–10 scale next to percentage scores. I checked this against the Llama-3 instruct row (58.79). Compare methods using Norm_Gain and the per-dimension numbers, not the absolute Helpful_Avg.

• Abstract ranges and text ratios. The abstract gives 2%–5% over data mixture and 1%–3% over merging. From the tables, the Norm_Gain differences are 4.81 and 1.83 points against Hummer, and 1.97 and 0.91 points against TSVM. The body says RESM is "close to 1.5x to 2.1x" better than data mixture §5. The Norm_Gain ratios are 1.47x (Llama-3) and 2.20x (Mistral).

• Table gaps and rounding. In the Mistral continual table, the MMLU cell for Stage 3 is blank tab. 11. Hummer's Llama-3 Norm_Gain is +10.16% in tab. 3 and +10.17% in tab. 6.

• Model selection for individual models. The paper trains each dimension model over a β and lr grid and reports the "best" one, without saying which metric defines best app. C.

• Interpretation vs data. The statement that catastrophic forgetting "paradoxically" improves 3H in sequential DPO is an interpretation of the continual table §5 tab. 10 tab. 11. The table values are the evidence.

• Dataset licences are not listed. The paper uses ten public preference datasets (HelpSteer, Py-Dpo, Distilabel-Orca, Distilabel-Capybara, UltraSafety, Truthy-DPO, GRATH, UltraFeedback, PKU-SafeRLHF, Nectar) and says the licence question is not applicable tab. 1. Check each dataset's licence before training on it.

• Scope limits (App. D). The paper covers two backbones, training-free merges only, and 3H only. The authors list no broader trust or safety claims app. D.

Lineage

• Builds on: TSVM, the task-singular-vector merge whose layer-wise SVD and orthogonalisation RESM extends §3 eq. 1 eq. 2 eq. 3.
• Builds on: the task-vector and sparsification merges (Task Arithmetic, TIES, DARE, DELLA, Breadcrumbs, Model Stock) used as baselines §2 tab. 2.
• Compared against: data-mixture methods (Heuristic, ArmoRM, Hummer) tab. 2 and linear merges (Weight Average, Rewarded Soup) tab. 3.
• Outlier-pruning comparison: Wanda and OWL applied on top of TSVM tab. 5.
• Related but not compared: MoE-based H3 Fusion, which routes inputs and needs training data app. B.
• Superseded by: not known.