---
kind: concept
slug: function-preserving-network-transformation
title: Function-preserving network transformation
summary: >-
  Edit a trained network so it computes the same function, then train on from there. Path-level edits can split one layer into parallel branches, not only widen or deepen it. The paper's search reuses weights, so each candidate needs only a short training run. Trade-off: edits are exact only in theory, and pooling edges need an extra distillation fit.
tags: [architecture-search, network-transformation]
papers: ["1806.02639"]
related: [neural-architecture-search, knowledge-distillation, policy-gradient]
status: draft
updated_at: 2026-11-08
---

## Overview

Training a candidate architecture from scratch costs a full run per candidate.
A function-preserving transformation starts from a trained parent and changes
its structure in a way that leaves the output unchanged at the moment of the
edit [arxiv:1806.02639 §3.1]. The child therefore starts as good as the parent,
and only needs further training to exploit the new capacity.

The invariant every variant shares is that the transformed network computes the
parent's function. The variants differ in which structural edits they allow.
Layer-level edits change width or depth. Path-level edits change how the signal
flows, for example by splitting one layer into several parallel branches
[arxiv:1806.02639 §1] [arxiv:1806.02639 §3.1].

## Variants

### Layer-level edits

Operations that widen a layer, deepen a network or compress it, in the style of
Net2Net. The paper names Net2DeeperNet and Net2WiderNet as the building blocks
of its branch edits [arxiv:1806.02639 §2.3]. The layer-level search it extends
keeps a recurrent controller but cannot change connection topology, so a
chain-shaped start stays chain-shaped [arxiv:1806.02639 §1] [arxiv:1806.02639 §2.3].
Card: [Path-level EAS](/papers/1806.02639) describes this lineage. Use it as
background, not as a method of its own.

### Path-level edits (addition or concatenation)

A layer is replaced by a multi-branch motif, merged either by addition or by
concatenation [arxiv:1806.02639 §3.1]:

- *Addition:* each branch holds a copy of the layer, and each branch output is
  divided by the branch count N. The sum then equals the original output
  [arxiv:1806.02639 §3.1].
- *Concatenation:* the input is copied to every branch, the output filters are
  split into N groups, and the branch outputs are concatenated [arxiv:1806.02639 §3.1].

For an identity layer, addition uses N identity branches, and concatenation
splits the input channels, with identity mappings in each branch
[arxiv:1806.02639 §3.1]. Net2Net-style operations are then applied inside the
branches, so that they can differ in depth and kernel type
[arxiv:1806.02639 §3.1] [arxiv:1806.02639 §2.3].

The paper uses this as the edit inside a tree-structured search space. Nodes
choose a merge scheme and a branch count, edges choose one of seven primitive
operations, and a tree-LSTM controller trained with REINFORCE picks them
[arxiv:1806.02639 §3.2] [arxiv:1806.02639 §3.3]. The best reported cell is
TreeCell-A, with 2.30% CIFAR-10 test error at 14.3M parameters, author-reported
[arxiv:1806.02639 tab. 1] [Path-level EAS](/papers/1806.02639).

### Pooling edges (distillation fit)

Pooling cannot be initialised as an identity, so a pooling edge breaks exact
preservation. The paper fits the student's weights to the teacher's output
logits by knowledge distillation after the edit. It calls this cost
negligible [arxiv:1806.02639 §3.2].

## When to use which

| Situation | Use | Why |
|---|---|---|
| The parent is a chain and the target is a different topology | Path-level edits | Layer-level edits cannot change connection topology [arxiv:1806.02639 §1] [arxiv:1806.02639 §2.3] |
| Only width or depth must change | Layer-level edits | Path-level edits are only needed for branch structure [arxiv:1806.02639 §1] |
| Search must be cheaper than from-scratch search | Reuse weights per candidate | The paper reports about 200 GPU-hours for 500 candidates, against 48,000 for NASNet-A. Author-reported [arxiv:1806.02639 §4.3] |
| Edit includes pooling | Add a distillation fit | Pooling has no identity initialisation [arxiv:1806.02639 §3.2] |
| No labelled validation split is available | Not stated | The paper holds out 5,000 training images for validation [arxiv:1806.02639 §4.1] |
| Parent weights are not accessible | Not applicable | The method starts from trained weights [arxiv:1806.02639 §3.1] |

The reported gains apply to the paper's CIFAR-10 and mobile ImageNet setting,
with DenseNet and PyramidNet parents [arxiv:1806.02639 tab. 1] [arxiv:1806.02639 tab. 2].
The paper gives single-run numbers without variance [arxiv:1806.02639 §4.2].
Results on other domains are not reported there.

## Implementation sketch

The two merge constructions from the paper, for one convolution layer. Each
branch is a copy of the layer with its own scaling; the rest of the search
loop is in the paper [arxiv:1806.02639 alg. 1].

```pseudocode
# Split a layer into N branches merged by addition  [arxiv:1806.02639 §3.1]
def split_by_addition(layer, n):
    # 1. Copy the layer once per branch, keeping its weights  [§3.1]
    branches = [copy(layer) for _ in range(n)]
    # 2. Divide each branch output by N so the sum matches the original  [§3.1]
    for b in branches:
        b.output_scale = 1 / n
    # 3. Merge the branches by summing their outputs  [§3.1]
    return AddMerge(branches)


# Split a layer into N branches merged by concatenation  [arxiv:1806.02639 §3.1]
def split_by_concat(layer, n):
    # 1. Give each branch one of N groups of the output filters  [§3.1]
    groups = split_filters(layer.weights, n)
    # 2. Each branch receives the same input, and gets its group's weights  [§3.1]
    branches = [layer_with(g) for g in groups]
    # 3. Concatenate the branch outputs along the channel axis  [§3.1]
    return ConcatMerge(branches)
```

Where the branches differ in depth or kernel type, apply the Net2Net-style
operations to each branch after the split [arxiv:1806.02639 §3.1].

## Contradictions

None known. Only one card covers this slug, so no cross-paper comparison is
possible. Known gaps between the paper and its released code, such as copied
edges receiving small noise, are described on [Path-level EAS](/papers/1806.02639)
and do not contradict the paper's own claims.

## Papers

- [Path-level EAS](/papers/1806.02639): extends layer-level function-preserving
  edits with path-level branch edits and a tree-structured RL search.