---
kind: concept
slug: computation-offloading
title: Computation offloading
summary: >-
  Deciding, per time slot, how much of a device's work to send to nearby base stations instead of running it locally,
  so that long-run energy stays low and queues stay stable. One source uses Lyapunov optimisation with an actor-critic
  agent, evaluated only in simulation. A TinyML review argues for keeping inference on the device instead.
tags: [edge-computing, reinforcement-learning, optimization, iot]
papers: ["2011.08430", "2211.04448"]
related: [policy-gradient, lyapunov-optimization]
status: draft
updated_at: 2026-10-25
---

## Overview

A device with limited CPU and battery can run a task itself or send its input
over a wireless link to an edge server. Sending costs transmit energy, running
at the edge costs edge energy, and running locally costs CPU energy. The
decision must be made for every time slot, while task arrivals and channel
quality change at random [arxiv:2011.08430 §2.2, §3.1].

The shared goal is to minimise long-run energy per completed bit, subject to
capacity limits and to queues that do not grow without bound [arxiv:2011.08430
§2.2, eq. 13, eq. 14]. The invariant the source relies on is that the long-run
objective and the stability constraints can be turned into a sequence of
per-slot decisions, using a Lyapunov drift-plus-penalty bound [arxiv:2011.08430
§3.2, eq. 17].

Offloading is not the only option. The TinyML review in this wiki treats
running the model on the device as the goal, and names delay and
confidentiality risks when data is sent to a distant site for analysis
[arxiv:2211.04448 §I]. It does not evaluate offloading, so it is a contrast
to the variant below rather than a second variant of it.

## Variants

### Lyapunov-based per-slot offloading with an actor-critic agent

The network has one macro base station and several small-cell base stations,
with each device attached to a covering station [arxiv:2011.08430 §2.1, §2.2].
Each slot the controller chooses bandwidths, transmit powers, offloaded amounts
and CPU shares [arxiv:2011.08430 §4.1]. The paper writes the per-slot choice as
minimising a Lyapunov penalty that combines energy, edge-queue and device-queue
terms [arxiv:2011.08430 eq. 20]. The source then trains a reinforcement-learning
agent to produce these decisions, instead of re-solving the per-slot problem
when the channel changes [arxiv:2011.08430 §3.2, §4.1]. Training uses
asynchronous actor-critic: learning agents at the small cells roll out copies
of the policy and send returns to a global agent at the macro cell
[arxiv:2011.08430 §4.2]. The method is described in [Deep Reinforcement Learning
for Stochastic Computation Offloading in Digital Twin Networks](/papers/2011.08430).

### Keep the work on the device

The TinyML review describes the opposite end of the design space. The model is
trained on a resource-rich machine and then ported to a microcontroller, where
inference runs on each sensor sample [arxiv:2211.04448 §II] [fig. 1]. The
review gives no numbers for this approach, and its main constraints are the
power and memory limits of the device [arxiv:2211.04448 §I, §IV.B]. Use it as
background for the on-device case, not as evidence about offloading.

### Other variants

No other card in this wiki covers offloading yet. Compare against the
two-variant joint-optimisation baseline and DQN that the source uses, but treat
those as baselines rather than variants [arxiv:2011.08430 §5].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Arrivals are random and the channel changes every slot, with many devices | Per-slot Lyapunov decision with a learned policy, as in the source | The source reports that the learned policy avoids re-solving per slot and that its cost is lower than DQN's in its simulations [arxiv:2011.08430 §4.2, §5, fig. 7] |
| Few devices and a fixed model of the channel | Not established by this source | The source reports small effect of the number of small cells when devices are few, and gives no comparison with a non-learning solver for that case [arxiv:2011.08430 fig. 5, §5] |
| Input data is sensitive, or delay to a distant site matters, and the model fits on the device | Keep inference on the device, as the TinyML review describes | The review names delay and confidentiality risks of sending data away [arxiv:2211.04448 §I]; its constraints are power and memory [arxiv:2211.04448 §IV.B] |
| You need a result you can reproduce exactly | Not established by this source | The source has no public code, and several equations are inconsistent (see below) [arxiv:2011.08430 §5] |
| Actions are discrete (for example, pick one station) | Not established by this source | The source's DQN baseline is attributed to action discretisation, and the source does not study discrete action spaces directly [arxiv:2011.08430 §4.2] |

## Implementation sketch

The common core is the per-slot loop: observe queues and channel state, pick an
action, then update the queues. The source's learning part is an
actor-critic update on the resulting return.

```pseudocode
# Observe state: channel rates, CPU capacities, powers, bandwidths, queues
# [arxiv:2011.08430 eq. 21]
state = observe_state(twin)

# Choose continuous actions: bandwidth, power, offloaded amount, CPU share
# [arxiv:2011.08430 §4.1]
action = actor(state)

# Reward is the negative of the per-slot Lyapunov penalty [arxiv:2011.08430 eq. 22]
reward = -penalty(state, action)

# Advance device and edge queues with the standard update equations
# [arxiv:2011.08430 eq. 5, eq. 6]
state = update_queues(state, action, arrivals())

# Learner: n-step return, advantage, and actor and critic updates
# [arxiv:2011.08430 eq. 23 to eq. 27]
returns = discounted_returns(rewards, bootstrap_value)
advantage = returns - critic(states)
actor_step(advantage * grad_log_policy(states, actions))
critic_step(squared_error_gradient(critic, states, returns))
```

The source reports a learning rate of 0.001, which it describes as best in its
sweep [arxiv:2011.08430 fig. 8, §5]. Network sizes are in the card's
Hyperparameters table.

### Implementation pitfalls

These come from the source's equations and its text. Check them before
implementing.

- The energy weight V cancels in the per-slot objective as written, because the
  bracket that multiplies it is zero [arxiv:2011.08430 eq. 17, eq. 20, eq. 22].
- The perturbation term in the drift bound uses a quantity that is not defined
  in the paper [arxiv:2011.08430 eq. 19].
- The paper does not say whether the per-slot problem is solved by an
  optimiser or by the actor network [arxiv:2011.08430 alg. 1, §4].
- The actor outputs eight softmax values, which is a categorical distribution,
  while the actions are described as continuous. The mapping from outputs to
  actions is not given [arxiv:2011.08430 §4.1, §5].
- The critic update adds the gradient of a squared error, which is a descent
  direction, while the actor step is an ascent step. Check the sign [arxiv:2011.08430 eq. 27].

## Contradictions

Across papers: none known. The two sources do not compare offloading with
on-device inference, so their numbers cannot be set against each other.
Within the offloading source, the text and figures disagree on the number of
devices used in one discussion (N=40 against the setup's N=20)
[arxiv:2011.08430 §5].

## Papers

- [Deep Reinforcement Learning for Stochastic Computation Offloading in Digital Twin Networks](/papers/2011.08430): applies Lyapunov drift-plus-penalty with asynchronous actor-critic to energy-efficient offloading in an industrial digital-twin network, with results shown only in simulation figures.
- [A review of TinyML](/papers/2211.04448): surveys machine learning on microcontroller-class devices and their deployment workflow; a review with no experiments, so it is a contrast to offloading rather than a variant of it.