Computation offloading

Overview

A device with limited CPU and battery can run a task itself or send its input over a wireless link to an edge server. Sending costs transmit energy, running at the edge costs edge energy, and running locally costs CPU energy. The decision must be made for every time slot, while task arrivals and channel quality change at random 2011.08430 §2.2, §3.1.

The shared goal is to minimise long-run energy per completed bit, subject to capacity limits and to queues that do not grow without bound [arxiv:2011.08430 §2.2, eq. 13, eq. 14]. The invariant the source relies on is that the long-run objective and the stability constraints can be turned into a sequence of per-slot decisions, using a Lyapunov drift-plus-penalty bound [arxiv:2011.08430 §3.2, eq. 17].

Offloading is not the only option. The TinyML review in this wiki treats running the model on the device as the goal, and names delay and confidentiality risks when data is sent to a distant site for analysis 2211.04448 §I. It does not evaluate offloading, so it is a contrast to the variant below rather than a second variant of it.

Variants

Lyapunov-based per-slot offloading with an actor-critic agent

The network has one macro base station and several small-cell base stations, with each device attached to a covering station 2011.08430 §2.1, §2.2. Each slot the controller chooses bandwidths, transmit powers, offloaded amounts and CPU shares 2011.08430 §4.1. The paper writes the per-slot choice as minimising a Lyapunov penalty that combines energy, edge-queue and device-queue terms 2011.08430 eq. 20. The source then trains a reinforcement-learning agent to produce these decisions, instead of re-solving the per-slot problem when the channel changes 2011.08430 §3.2, §4.1. Training uses asynchronous actor-critic: learning agents at the small cells roll out copies of the policy and send returns to a global agent at the macro cell 2011.08430 §4.2. The method is described in Deep Reinforcement Learning for Stochastic Computation Offloading in Digital Twin Networks.

Keep the work on the device

The TinyML review describes the opposite end of the design space. The model is trained on a resource-rich machine and then ported to a microcontroller, where inference runs on each sensor sample 2211.04448 §II fig. 1. The review gives no numbers for this approach, and its main constraints are the power and memory limits of the device 2211.04448 §I, §IV.B. Use it as background for the on-device case, not as evidence about offloading.

Other variants

No other card in this wiki covers offloading yet. Compare against the two-variant joint-optimisation baseline and DQN that the source uses, but treat those as baselines rather than variants 2011.08430 §5.

When to use which

Implementation sketch

The common core is the per-slot loop: observe queues and channel state, pick an action, then update the queues. The source's learning part is an actor-critic update on the resulting return.

# Observe state: channel rates, CPU capacities, powers, bandwidths, queues
# [arxiv:2011.08430 eq. 21]
state = observe_state(twin)

# Choose continuous actions: bandwidth, power, offloaded amount, CPU share
# [arxiv:2011.08430 §4.1]
action = actor(state)

# Reward is the negative of the per-slot Lyapunov penalty [arxiv:2011.08430 eq. 22]
reward = -penalty(state, action)

# Advance device and edge queues with the standard update equations
# [arxiv:2011.08430 eq. 5, eq. 6]
state = update_queues(state, action, arrivals())

# Learner: n-step return, advantage, and actor and critic updates
# [arxiv:2011.08430 eq. 23 to eq. 27]
returns = discounted_returns(rewards, bootstrap_value)
advantage = returns - critic(states)
actor_step(advantage * grad_log_policy(states, actions))
critic_step(squared_error_gradient(critic, states, returns))

The source reports a learning rate of 0.001, which it describes as best in its sweep 2011.08430 fig. 8, §5. Network sizes are in the card's Hyperparameters table.

Implementation pitfalls

These come from the source's equations and its text. Check them before implementing.

• The energy weight V cancels in the per-slot objective as written, because the bracket that multiplies it is zero 2011.08430 eq. 17, eq. 20, eq. 22.
• The perturbation term in the drift bound uses a quantity that is not defined in the paper 2011.08430 eq. 19.
• The paper does not say whether the per-slot problem is solved by an optimiser or by the actor network 2011.08430 alg. 1, §4.
• The actor outputs eight softmax values, which is a categorical distribution, while the actions are described as continuous. The mapping from outputs to actions is not given 2011.08430 §4.1, §5.
• The critic update adds the gradient of a squared error, which is a descent direction, while the actor step is an ascent step. Check the sign 2011.08430 eq. 27.

Contradictions

Across papers: none known. The two sources do not compare offloading with on-device inference, so their numbers cannot be set against each other. Within the offloading source, the text and figures disagree on the number of devices used in one discussion (N=40 against the setup's N=20) 2011.08430 §5.

Papers

• Deep Reinforcement Learning for Stochastic Computation Offloading in Digital Twin Networks: applies Lyapunov drift-plus-penalty with asynchronous actor-critic to energy-efficient offloading in an industrial digital-twin network, with results shown only in simulation figures.
• A review of TinyML: surveys machine learning on microcontroller-class devices and their deployment workflow; a review with no experiments, so it is a contrast to offloading rather than a variant of it.