Accelerate Intermittent Deep Inference

In plain words

Some sensors run on a tiny chip that gets its power from light, vibration or radio waves. The power is weak and unstable, so the chip shuts off again and again in the middle of a calculation, and each time it must pick up where it stopped. Running a neural network on such a chip is hard, because the network has to fit in a very small working memory and has to finish in small pieces between power failures.

The paper builds on an existing method that searches for a network design which can survive these power cuts. It keeps that search but makes the networks it starts from smaller: a big, already-compact network is first distilled into a smaller one, and the search then looks for designs that fit the chip's memory. The author's main idea for speed is to let each power-on period do more work, by processing more data at once.

The results are reported by the author on a test board and on a PC, and the paper says the method reaches higher accuracy than the baseline at a modest cost in latency. Treat these as the author's claims: the paper has no published code and few numeric details.

TL;DR

• Targets microcontrollers that run on harvested energy, with a 2 KB runtime SRAM cap on an MSP430FR5994 §5.1 §5.3.
• Extends the iNAS design search with a distilled and shrunk MCUNet teacher and a larger batch of work per power cycle §4.2 §4.3.
• Author-reported: higher accuracy than HW-NAS on CIFAR-10 and Tiny ImageNet, a larger network at about 2 KB peak memory, and 12% more latency than iNAS on CIFAR-10 §5.4 §5.5 §5.6.
• No official code was found. Many training and search settings are not stated.

Problem

Networks for battery-less devices must run in short bursts between power failures. Each burst has a fixed energy budget, and progress must be saved to non-volatile memory and recovered after each outage §2.1 §2.3.

Neural architecture search (NAS) for such devices has to add a check for these power cycles. The paper says earlier optimized NAS assumed continuous power, and earlier intermittent designs balanced data reuse and recovery costs but often reached low accuracy §1. It also says NAS usually maximizes data reuse without modelling intermittent behaviour §1.

The paper's concrete target is a network that runs in about 2 KB of runtime SRAM on an MSP430FR5994, while keeping accuracy high §5.1 §5.3.

Method

The paper describes its method in prose. It has no numbered equations, tables or algorithms. Section numbers below follow the paper's own numbering (§1 to §7; subsections counted within each section). In the two-column PDF these appear as Roman and letter numerals, so §3.2 is Section III-B. Figure numbers follow order of appearance; fig. 1 to fig. 14 are the paper's figures.

Baseline: iNAS §4.1 fig. 6. An RNN controller proposes a child network. For each child, the design explorer (iNAS-Exp) searches preservation and execution designs, and a performance model (iNAS-PMod) estimates energy per power cycle and end-to-end latency §4.1.

• The child is trained only if a feasible design meets the latency requirement §4.1.
• The controller is updated with a reward built from the child's accuracy, an exponential moving average of earlier feasible accuracies, and the latency relative to the requirement §4.1.
• The output is the most accurate child with its lowest-latency design §4.1.
• The paper states that search time grows linearly with the number of feasible children §4.1.

Accelerated intermittent NAS §4.2 fig. 7. The paper lists three changes to iNAS:

1. Distil a large teacher before the search. The teacher is MCUNetV1-Int2, which uses float16 §5.3. The student is pruned by more than 80% of its parameters and its weights are shared, then it is shrunk along kernel, row/column (depth) and channel dimensions §4.2.
2. Use a more capable controller, such as an evolutionary or Bayesian one. This is proposed, not evaluated §4.2.
3. After the search, a solution dumper writes C code and one header file with all weights and the network structure. It uses the TI DSPLib library on the MSP430 §4.2.

Larger batch per power cycle (hyper-tile) §3.1 §4.3 fig. 1 fig. 4. Vanilla iNAS uses fine tiles, which means more data fetches. The paper proposes a larger batch size, called hyper-tile, so each power cycle does more computation. It claims this allows much larger CNNs with only a small SRAM increase §3.1 §4.3. No batch size is given.

Real-time scheduling §3.1 fig. 3. The paper says it adds a scheduling framework with worst-case execution time and deadline checks for intermittent DNN tasks §3.1. The text does not give the schedulability test, so this part cannot be implemented from the paper alone.

Techniques discussed in §3.2 fig. 5. §3.2 describes these in general terms, not as implemented steps: float16 weights compiled into a header file, progressive shrinking (width, kernel, channel), a TinyNAS and TinyEngine co-design, and storing trained weights to CSV files for layer-by-layer execution §3.2. Only the distillation and header-file steps appear in the implementation §4.2.

Pseudocode of the implemented pipeline, reconstructed from §4.1 and §4.2:

# Accelerated intermittent NAS  [§4.1] [§4.2] [fig. 7]
# Input:  teacher network (MCUNetV1-Int2, float16); device budget
# Output: child network with its intermittent execution design

def accelerated_intermittent_nas(teacher, device_budget):
    # 1. Distil and shrink the teacher into a search space  [§4.2]
    search_space = distill_and_shrink(teacher)

    # 2. Search with the iNAS controller  [§4.1]
    for iteration in range(num_iterations):          # count not stated
        child = controller.propose(search_space)

        # Best preservation and execution design for this child  [§4.1]
        design = inas_exp(child, device_budget)
        if design is None:
            continue

        # Train only if the design meets the latency requirement  [§4.1]
        if design.latency >= latency_requirement:
            continue
        accuracy = train(child)

        # Reward as written in the paper, sign of latency included  [§4.1]
        reward = (accuracy - moving_average_accuracy) + (
            design.latency / latency_requirement
        )
        controller.update(reward)

    # 3. Keep the most accurate child with its lowest-latency design  [§4.1]
    best_child, best_design = best_feasible_result()

    # 4. Write one header with weights and structure for the MCU  [§4.2]
    return dump_solution(best_child, best_design)

Hyperparameters

Results

Author-reported. Setup: CIFAR-10 and Tiny ImageNet, accuracy measured on a PC (CPU only), peak memory and latency measured on the MSP430FR5994 §5.1 §5.4 §5.6. Figure values are not transcribed here. The text gives these headline numbers:

Other statements in the text: iNAS reaches accuracy comparable to HW-NAS §5.4. The authors say iNAS is better for smaller networks under intermittent execution §6.

Paper vs code

No official code. Searches: the paper and LaTeX source have no code link; the GitHub repository search for "accelerated intermittent deep inference", "aiNAS intermittent", "intermittent deep inference", "intermittent deep inference msp430" and "intermittent nas" found no repository from the authors; a user search for the author name found none.

Pitfalls

• Few settings are stated. The paper names no batch size, search budget, latency requirement, energy budget, optimizer or learning rate. An implementer must pick these values, and results cannot be compared directly.
• Reward sign. The paper says the reward adds the latency term divided by the requirement §4.1. Such a reward rewards slower designs. This may be a typo, but the paper does not say so; check it before reuse.
• Name variants. The title is "Accelerate Intermittent Deep Inference" and the abstract says "Accelerated". Figures and text also use "Accelerated insNAS", "Accelerated intermittent NAS" and "hyper-tile" for the same method §3.1 §4.2 fig. 4 fig. 7.
• Hyper-tile has no size. The method rests on a larger batch per power cycle, but the paper gives no value and no formula for the batch size S §4.3 §2.3. The memory claim "magnitude larger CNNs with only a small increase" §3.1 has no number attached.
• Distillation detail is missing. "Share the weights with the least accuracy decrease based on the biases" §4.2 is not defined, and the pruning criterion is not given.
• Storage format conflicts. §3.2 mentions CSV weight files; §4.2 writes a single header file §3.2 §4.2. The implemented version is the header file, but the text is not consistent.
• Figure caption mismatch. fig. 5 is captioned "Weight Sharing Model after Knowledge Distillation", while the nearby text describes CSV weight storage and layer-by-layer execution §3.2.
• Accuracy and latency come from different setups. Accuracy is measured on a PC (CPU only), not on the MCU §5.4. Latency is measured only on CIFAR-10 with the MSP430 §5.6. Power-failure emulation is not described: the paper says a voltage monitor can detect power loss and save state §5.1, but not how the intermittent power was simulated or how often it failed.
• Undefined percentages. "+19.1% accuracy" is not stated as percentage points or as a relative change. Absolute values are only in fig. 11.
• Size claim is ambiguous. The text gives "145X" as network size relative to MCUNet at a comparable peak memory §5.5, and "0.75 million parameters" in §6. The baseline for 145x is not stated in one place.
• Scheduling section is not operational. §3.1 refers to worst-case execution time and schedulability but gives no test, bound or formula.
• Unevaluated proposals. Better NAS controllers §4.2 and the hyper-tile idea on larger SRAM §7 are proposals. They are not tested in the paper.
• Overclaims to check. "The first to build ... more than 60%" accuracy on Tiny ImageNet §6 is a claim with no comparison table in the paper.
• Author-reported only. There is one author and no code. Results should be reproduced before any product decision.

Lineage

• Builds on: iNAS, intermittent-aware NAS with iNAS-Exp and iNAS-PMod §2.3 §4.1. Not in this wiki yet.
• Builds on: HW-NAS, the hardware-aware NAS used as the accuracy baseline §5.4 §4.1.
• Uses: MCUNetV1-Int2 as the teacher §5.3 (cited in the paper as Lin2020MCUNetTD).
• Compared against: HW-NAS, iNAS and MCUNet §5.4 §5.5 §5.6. The Figure 2 caption also names OFA fig. 2.
• Superseded by: not known from content in this wiki.