Function-preserving network transformation

Overview

Training a candidate architecture from scratch costs a full run per candidate. A function-preserving transformation starts from a trained parent and changes its structure in a way that leaves the output unchanged at the moment of the edit 1806.02639 §3.1. The child therefore starts as good as the parent, and only needs further training to exploit the new capacity.

The invariant every variant shares is that the transformed network computes the parent's function. The variants differ in which structural edits they allow. Layer-level edits change width or depth. Path-level edits change how the signal flows, for example by splitting one layer into several parallel branches 1806.02639 §1 1806.02639 §3.1.

Variants

Layer-level edits

Operations that widen a layer, deepen a network or compress it, in the style of Net2Net. The paper names Net2DeeperNet and Net2WiderNet as the building blocks of its branch edits 1806.02639 §2.3. The layer-level search it extends keeps a recurrent controller but cannot change connection topology, so a chain-shaped start stays chain-shaped 1806.02639 §1 1806.02639 §2.3. Card: Path-level EAS describes this lineage. Use it as background, not as a method of its own.

Path-level edits (addition or concatenation)

A layer is replaced by a multi-branch motif, merged either by addition or by concatenation 1806.02639 §3.1:

• Addition: each branch holds a copy of the layer, and each branch output is divided by the branch count N. The sum then equals the original output 1806.02639 §3.1.
• Concatenation: the input is copied to every branch, the output filters are split into N groups, and the branch outputs are concatenated 1806.02639 §3.1.

For an identity layer, addition uses N identity branches, and concatenation splits the input channels, with identity mappings in each branch 1806.02639 §3.1. Net2Net-style operations are then applied inside the branches, so that they can differ in depth and kernel type 1806.02639 §3.1 1806.02639 §2.3.

The paper uses this as the edit inside a tree-structured search space. Nodes choose a merge scheme and a branch count, edges choose one of seven primitive operations, and a tree-LSTM controller trained with REINFORCE picks them 1806.02639 §3.2 1806.02639 §3.3. The best reported cell is TreeCell-A, with 2.30% CIFAR-10 test error at 14.3M parameters, author-reported 1806.02639 tab. 1 Path-level EAS.

Pooling edges (distillation fit)

Pooling cannot be initialised as an identity, so a pooling edge breaks exact preservation. The paper fits the student's weights to the teacher's output logits by knowledge distillation after the edit. It calls this cost negligible 1806.02639 §3.2.

When to use which

The reported gains apply to the paper's CIFAR-10 and mobile ImageNet setting, with DenseNet and PyramidNet parents 1806.02639 tab. 1 1806.02639 tab. 2. The paper gives single-run numbers without variance 1806.02639 §4.2. Results on other domains are not reported there.

Implementation sketch

The two merge constructions from the paper, for one convolution layer. Each branch is a copy of the layer with its own scaling; the rest of the search loop is in the paper 1806.02639 alg. 1.

# Split a layer into N branches merged by addition  [arxiv:1806.02639 §3.1]
def split_by_addition(layer, n):
    # 1. Copy the layer once per branch, keeping its weights  [§3.1]
    branches = [copy(layer) for _ in range(n)]
    # 2. Divide each branch output by N so the sum matches the original  [§3.1]
    for b in branches:
        b.output_scale = 1 / n
    # 3. Merge the branches by summing their outputs  [§3.1]
    return AddMerge(branches)


# Split a layer into N branches merged by concatenation  [arxiv:1806.02639 §3.1]
def split_by_concat(layer, n):
    # 1. Give each branch one of N groups of the output filters  [§3.1]
    groups = split_filters(layer.weights, n)
    # 2. Each branch receives the same input, and gets its group's weights  [§3.1]
    branches = [layer_with(g) for g in groups]
    # 3. Concatenate the branch outputs along the channel axis  [§3.1]
    return ConcatMerge(branches)

Where the branches differ in depth or kernel type, apply the Net2Net-style operations to each branch after the split 1806.02639 §3.1.

Contradictions

None known. Only one card covers this slug, so no cross-paper comparison is possible. Known gaps between the paper and its released code, such as copied edges receiving small noise, are described on Path-level EAS and do not contradict the paper's own claims.

Papers

• Path-level EAS: extends layer-level function-preserving edits with path-level branch edits and a tree-structured RL search.