Paper · 2020-11-17 · cs.LG, eess.SP
Edge Intelligence for Energy-efficient Computation Offloading and Resource Allocation in 5G Beyond
Minimises device energy in a 5G end-edge-cloud network: each task runs locally, on a small-cell base station or on the macro base station, and servers split CPU among offloaded tasks. Uses DDPG on mixed discrete-continuous actions, with offloading decisions rounded by bipartite matching. Author-reported: lower energy than local-only and full-offloading heuristics; no numbers in the text.
Computation offloadingActor-criticExperience replayBipartite matching for set-prediction lossescomputation-offloadingedge-computingdeep-reinforcement-learningddpg5genergy-efficiencyresource-allocation
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Phones and other small devices have little battery and little computing power. They can run a task themselves, send it to a nearby small base station with its own server, or send it to a bigger base station with a cloud server behind it. Each choice costs energy in a different way: computing locally uses the device's battery, while sending data over the air also drains it.
The paper trains a learning agent to make these choices for many devices at once, and to split the servers' computing capacity among the tasks sent to them. The agent learns from trial runs in a simulated network, which means it does not need a complete, exact picture of the wireless channel. Because the choice of where to run a task is a yes-or-no decision while the capacity split is a continuous amount, the authors add a rounding step that turns the agent's continuous outputs into clean yes-or-no decisions. The authors report that this approach uses less energy than two simple fixed policies.
TL;DR
- Problem: choose, per device, local computing, a small-cell base station (SBS) or the macro base station (MBS), and split each server's CPU among its tasks, to minimise total energy under a per-task deadline §4.
- Method: formulate as an MDP and train DDPG; the actor outputs continuous offloading and CPU-share values, which are rounded to binary decisions with a bipartite matching §5.
- Result: author-reported lower energy than local-only and full-offloading baselines in simulation; the text gives no numeric values for these results §6.2.
- Code: none found.
Problem
Each device has one task, described by its input size d_i, its CPU demand c_i and a deadline τ_i §3. Every task runs in exactly one place: locally, on one SBS, or on the MBS §3. The objective is the sum of the energies of the chosen options eq. 8. The constraints are the deadline, the one-place rule, the CPU capacity of each server, and bounds on the CPU shares §4. The authors state that the integer constraints make the problem NP-hard, and they therefore use DRL instead of an exact solver §4.
The stated assumption is that the optimisers in prior work need complete and accurate network information, which is hard to get in a time-varying wireless network §2.2.
Method
System model §3. There is one MBS and M SBSs. N devices sit in the MBS coverage area. Each SBS has an edge server, and the MBS has a cloud server with more CPU §3. A device uses the nearest SBS whose coverage radius γ_j covers it, and otherwise uses the MBS §3.1. Uplink rate to an SBS uses OFDMA-style interference from other cells eq. 1. Rate to the MBS has no interference term eq. 2. The downlink delay of the result is ignored, because the authors assume the result is much smaller than the input §3.2.
Per-option cost §3.2.
- Local: delay is cycles divided by the device CPU speed eq. 3. Energy is ς · c · f² with ς the effective switched capacitance eq. 4.
- SBS j: delay is uplink time plus compute time on the CPU share f_ij eq. 5. Energy is transmit power times uplink time plus c times e_j eq. 6.
- MBS: the same form as SBS, with rate R_i0, CPU share f_i0 and energy coefficient e_0 eq. 7, §3.2.
Constraints §4. Feasibility means: the chosen option meets τ_i [c1]; each device picks exactly one option [c2, c7]; the CPU shares on each SBS and on the MBS sum to at most their capacity [c4]; and each share lies between 0 and the server capacity [c5]. The labels c1 to c7 are the ones used in the source text; their rendered numbers are not confirmed.
MDP §4. The paper defines a 4-tuple (S, A, P, R) and writes the state, action and reward as follows.
- State: input sizes, remaining CPU demand, deadlines, the uplink rates to each SBS and the MBS, and the free CPU of each server §4, item 1.
- Action: the local choice x, the SBS choice y, the MBS choice z, and the CPU shares f^s and f^m §4, item 2.
- Reward: minus the expected energy of the chosen options [§4, item 3, eq. 8 form]. When a task misses its deadline the agent gets a negative constant penalty §4. The simulation uses a penalty of 100 · N §6.1.
- Next state: remaining demand and deadline are reduced by the work done in one slot; the SBS or MBS free CPU is reduced by the shares assigned §4, item 4.
- Objective: maximise the discounted sum of rewards over T slots with discount ε in [0, 1] §4.
DDPG training [§5.1, Algorithm 1 [alg. 1]]. The actor π(s | θ_π) maps a
state to a proto-action. Exploration adds Ornstein–Uhlenbeck noise §5.1. The
critic Q(s, a | θ_Q) is trained on a one-step target built from target copies of
the actor and critic §5.1, target eq.. The actor is updated along the sampled
policy gradient through the critic §5.1, gradient eq.. Target copies are
updated by a soft rule with weight ω in [0, 1] §5.1, target-update eq..
Transitions are stored in a replay memory and sampled in mini-batches §5.1.
# DDPG training loop [alg. 1] [§5.1]
# Input: environment env, replay memory replay (fixed size, not stated)
# Output: trained actor
def train(env, episodes, steps_per_episode):
# Initialise actor, critic and their target copies [alg. 1 lines 1-2]
actor = build_network()
critic = build_network()
actor_target = copy(actor)
critic_target = copy(critic)
# Replay memory; its size is not stated [alg. 1 line 2]
replay = empty_buffer()
for episode in range(episodes):
# Reset the environment at the start of each episode [alg. 1 line 3]
state = env.reset()
for t in range(steps_per_episode):
# Actor output plus exploration noise [§5.1 action eq.]
action = actor(state) + ou_noise()
# Immediate reward and next state from the environment [§4]
reward, next_state = env.step(action)
# Store the transition; the oldest is dropped when full [§5.1]
replay.add(state, action, reward, next_state)
# Sample a mini-batch of V transitions [§5.1] [§6.1: V = 32]
batch = replay.sample(32)
# One-step target from the target networks [§5.1 target eq.]
targets = reward + discount * critic_target(
next_state, actor_target(next_state))
# Critic: minimise squared error to the target [§5.1 loss eq.]
critic_loss = mean((targets - critic(state, action)) ** 2)
critic.update(critic_loss, learning_rate_critic)
# Actor: ascend critic gradient through actor [§5.1 pg eq.]
actor_objective = mean(critic(state, actor(state)))
actor.update(-actor_objective, learning_rate_actor)
# Soft update of the target networks [alg. 1 lines 8-9] [§5.1]
actor_target.params = (omega * actor.params
+ (1 - omega) * actor_target.params)
critic_target.params = (omega * critic.params
+ (1 - omega) * critic_target.params)
state = next_stateThe two target assignments are parameter-wise updates; the paper writes them
per parameter §5.1. The soft-update weight omega is not stated numerically.
Action refinement §5.2. The actor returns continuous values for the offloading variables w = [x, y, z]. These must become binary decisions. The method has three steps: take the continuous values; build a weighted bipartite graph between devices and virtual slots of each strategy; then find a complete maximum-weight matching with the Hungarian algorithm §5.2 steps 1–3. A matched edge sets the decision to 1, and any other entry to 0 §5.2 step 3.
# Round continuous offloading values to binary decisions [§5.2]
# Input: weights[j][i] = continuous w_ij for strategy j, device i
# Output: binary decisions, same shape
def refine(weights):
# Strategy 0 is local, 1..M are SBSs, M+1 is the MBS [§5.2 step 2]
graph_edges = []
for j in strategies(weights):
# Slot count J_j = ceil(sum of w_ij over devices) [§5.2 step 2]
count = ceil(sum(weights[j]))
# Connect devices to virtual slots (Algorithm 2 below) [alg. 2]
graph_edges += build_edges(weights[j], count, strategy=j)
# Maximum-weight complete bipartite matching [§5.2 step 3] [Hungarian]
matching = hungarian_max_weight(graph_edges)
# Matched edge -> 1, every other entry -> 0 [§5.2 step 3]
return binarise(weights, matching)Algorithm 2 builds the edges. The reading below follows the paper's wrap-around description: each virtual slot has capacity one, and a device's weight fills slots in device order.
# Edges between devices and virtual slots of one strategy [alg. 2]
# Input: weights[i] = w_ij for devices i, count = J_j
# Output: list of (device, slot, edge weight)
def build_edges(weights, count, strategy):
# One slot only: every positive weight goes to it [alg. 2 lines 2-4]
if count <= 1:
return [(i, 1, w) for i, w in enumerate(weights) if w > 0]
edges = []
# Slot being filled, and the capacity still free in it [alg. 2 line 5]
slot = 1
room = 1.0
for i, w in enumerate(weights):
remaining = w
while remaining > 0:
# Put as much of the device's weight into the slot as fits [alg. 2]
part = min(room, remaining)
edges.append((i, slot, part))
remaining -= part
room -= part
# Move to the next slot once this one is full [alg. 2 lines 8-11]
if room <= 0 and slot < count:
slot += 1
room = 1.0
return edgesThe pseudocode above is a re-reading of the paper's text and Algorithm 2, not a transcription of an implementation; the paper has no code.
Compute §5.1 complexity. The paper gives the per-update cost of the networks as the sum of products of adjacent layer widths for actor and critic §5.1 complexity eq.. The action refinement is O(|V| · |E|) for |V| nodes and |E| edges §5.2.
Hyperparameters
Values are as stated in the paper (simulation setup §6.1 unless noted).
| Name | Value | Where | Notes |
|---|---|---|---|
| Devices N (default) | 100 | §6.1 | Every device has one task |
| Device counts in sweeps | 20, 60, 100 | §6.3 fig. 7 | Table of training delay uses 20, 40, 60 tab. 3 |
| SBS count M | 10 | §6.1 | One MBS |
| Max device transmit power | 100 mW | §6.1 | |
| Noise power σ² | 10⁻¹¹ mW | §6.1 | |
| Bandwidth, MBS / SBS | 10 MHz / 5 MHz | §6.1 | |
| Energy per CPU cycle e_j = e_0 | 1 W/GHz | §6.1 | Reference to a chip-power paper; unit as stated |
| CPU speed, device / SBS / MBS | 0.5 / 10 / 50 GHz | §6.1 | Cycles per second |
| Task input size d_i | U[5, 50] MB | §6.1 | Uniform |
| Required CPU per task | U[0.5, 5] GHz | §6.1 | Stated "per bit"; see Pitfalls |
| Deadline τ_i | 1 s | §6.1 | |
| Output activation | (tanh(x) + 1) / 2 | §6.1 | Layer it applies to not stated |
| Mini-batch size V | 32 | §6.1 | |
| Training episodes | 6000 | §6.1 | |
| Steps per episode | 20 | §6.1 | |
| Penalty for a missed deadline | 100 · N | §6.1 | §4 calls it "a negative constant" |
| Actor learning rate α_π | swept 10⁻²…10⁻⁵; 10⁻³ best | §6.3 fig. 5 | Converges only at 10⁻³ and 10⁻⁴ |
| Critic learning rate α_Q | swept 10⁻²…10⁻⁵; 10⁻³ best | §6.3 fig. 5 | Same |
| Discount factor ε | swept 0.5–0.7; 0.6 best | §6.3 fig. 6 | Values outside 0.5–0.7 not tested |
| Soft-update weight ω | in [0, 1] | §5.1 | Value not stated |
| Replay memory size | not stated | ||
| Network width, depth, optimiser | not stated | Authors say TensorFlow §6.1 | |
| Exploration noise parameters | not stated | §5.1 | Only "Ornstein–Uhlenbeck" |
| Time-slot length Δt | not stated | §4 | |
| Path-loss exponent α | not stated | §3.1 | Symbol only |
| Effective switched capacitance ς | not stated | §3.2 | Symbol only |
| SBS coverage radius γ | not stated | §3.1 | Symbol only |
| Channel-gain model | 3GPP models cited | §6.1 | Parameters not given |
| Time-varying experiment | 10 devices, 5 SBS; MBS 2 MHz, SBS 1 MHz; bandwidth × 10 after 3000 episodes | §6.3 fig. 8 | |
| Hardware | MacBook Pro, two Intel Core i5 at 2.6 GHz | §6.1 | |
| Software | Python and TensorFlow | §6.1 |
Task types used in the simulation tab. 2: Type 1 is 50 MB with 5 GHz (large data, compute-heavy); Type 2 is 50 MB with 0.5 GHz (large data, light compute); Type 3 is 5 MB with 5 GHz (small data, compute-heavy) tab. 2.
Results
Author-reported. Setup: simulated network with one MBS and ten SBSs; the "real-world dataset" named in the abstract is not identified in §6 (see Pitfalls) §6.1. Baselines: local computing and full offloading to the MBS §6.1. No numeric headline values appear in the text; the comparisons are read from figures.
| Setting | Metric | Baseline | This paper | Source |
|---|---|---|---|---|
| Energy vs number of devices (N = 100 default) | System energy consumption | Local computing; full offloading | Lower than both; full offloading is worst; local energy rises fastest with N. Values not stated in text | fig. 3 §6.2 |
| Per-device energy by task type | Energy per device | Local; full offloading | Lowest for types 1 and 3. For type 2 local computing is lowest and the proposal is slightly above local but far below full offloading | fig. 4 §6.2 |
| Learning rate sweep | Normalised cumulative reward | Not applicable | Converges only at 10⁻³ and 10⁻⁴; best at 10⁻³ for both rates | fig. 5 §6.3 |
| Discount factor sweep (0.5–0.7) | Normalised cumulative reward | Not applicable | Converges for all tested values; 0.6 best | fig. 6 §6.3 |
| Number of devices (20, 60, 100) | Normalised cumulative reward | Not applicable | Increases with N; the gain from 60 to 100 is smaller than from 20 to 60 | fig. 7 §6.3 |
| Training delay | Seconds (as labelled) | Not applicable | N=20: 35.17 s at 3000 episodes, 71.35 s at 6000; N=40: 38.31 s, 77.87 s; N=60: 51.40 s, 100.18 s | tab. 3 §6.3 |
| Bandwidth raised ten-fold after 3000 episodes | Normalised cumulative reward | Not applicable | Reward rises sharply and stabilises higher | fig. 8 §6.3 |
Missing: no variance across seeds, no number of runs, no confidence intervals, and no comparison with any learning-based baseline such as a DQN §6.
Paper vs code
No official code.
Searched: (1) the paper's abstract, body and bibliography [raw/src/2011.08442, reference.bib] and the arXiv metadata, which have no project or code link. The bibliography's only GitHub link is github.com/floodsung/DDPG, a third-party DDPG implementation, which is not the authors' and is not used. (2) GitHub API repository search for "end-edge-cloud offloading" and "DDPG offloading edge": no author-owned repository. (3) Author profile github.com/yueyuedai (a username that matches an author's contact e-mail handle in the source; the profile does not name its owner); it shows one offloading-related repository, UA-MEC, whose README names a different paper ("Joint user association and offloading") and which has no license, so it is not used. The profile's owner is not confirmed to be the author.
Pitfalls
- Dataset is unnamed. The abstract says "real-world dataset" [abstract], but §6 generates tasks from uniform distributions §6.1. The LaTeX source has a commented-out paragraph about MNIST, which is not part of the compiled paper. Do not assume a dataset.
- Units of CPU demand. Required CPU is "per bit" in U[0.5, 5] GHz §6.1, but eqs. 3–6 use total cycles c_i. The conversion from cycles per bit to total cycles (for example multiplying by the input size in bits) is not stated. Choose one explicitly and say so in the implementation.
- Units of energy. Eq. 6 adds p·d/R (watt-seconds) to c·e_j with e_j in W/GHz. The sum is consistent only when c is in gigacycles. Eq. 4 with f in GHz and c in cycles also needs the same care.
- Notation in the rate. Eq. 1 writes the interference from other BSs with h_{i'j'} and r^s_{i'j'} for every j' ≠ j, while the text says the interference comes from "other SBSs". The text also says device i "can communicate with SBS i" where j is meant §3.1. Treat the summation range as per the equation.
- Index conventions differ. The bipartite section uses j = 0 for local, 1..M for SBSs, and M+1 for the MBS §5.2 step 2. The system model uses b_0 for the MBS §3. Map them once.
- Algorithm 2 boundary. The source says the loop runs over s = 1 … J_j − 1, so the last slot only receives overflow. Whether the final slot should take any remaining weight is not stated. The sketch above fills slots until the weight is used up, which is one reading.
- Feasibility after rounding. The rounding step does not show that its output meets the deadline, the one-place rule, or the CPU capacities §5.2. The text does not say how infeasible actions are handled beyond the deadline penalty §4, §6.1. The reward and the penalty must cover this.
- Projection of CPU shares. The actor outputs f^s and f^m as continuous values and the activation maps to (0, 1) §6.1. How these values are scaled to the server capacity F, and how the capacity constraint c4 is enforced, is not stated.
- Target updates. The text says the target networks are updated "after a period of training" §5.1, while Algorithm 1 applies the soft update at every step alg. 1 lines 8–9. The step-wise version is used in the pseudocode above; ω is not stated.
- Critic inputs. §5.1 also feeds the next state into the critic. The equations and Algorithm 1 use Q(s_t, a_t) as the critic's input, as in standard DDPG. Implement the standard form.
- Bellman equation. The equation placed under "Primary Critic DNN" has Q on both sides §5.1. The usable target is the one-step equation with the target networks.
- Algorithm title. Algorithm 1 is titled "DDPG-based Content Sharing algorithm" alg. 1 and the text refers to "edge caching and content delivery policy" §5.1. These phrases appear to be left over from another paper. The algorithm described is the offloading one.
- Reward sign and penalty. §4 describes the reward as minus the expected energy and calls the deadline penalty "a negative constant"; §6.1 sets it to −100 · N. Confirm the sign and the constant before comparing numbers.
- Figure-only results. All comparison numbers are in figures [fig. 3, fig. 4, fig. 5 to fig. 8], and their axes are normalised without a stated normalisation. Do not reconstruct numeric results from the figures without the original data.
- Benchmarks are weak. The two baselines are heuristics; there is no comparison with DQN-style or other learning methods, and no seed variance is reported §6.
- Training delay units. Table 3 gives values in seconds without saying whether they are totals or per episode tab. 3.
Lineage
- Builds on: DDPG (Lillicrap et al. 2015), cited for the exploration noise and the actor-critic update §5.1; no card in this wiki yet.
- Builds on: the rounding technique (Shmoys and Tardos, 1993) and the Hungarian algorithm (Kuhn, 1955), both cited in §5.2; no cards in this wiki.
- Uses the 3GPP channel models, cited in §6.1.
- Compared against: local computing and full offloading to the MBS §6.1. No learning-based baseline.
- Superseded by: not stated.
Source and license
Adapted from “Edge Intelligence for Energy-efficient Computation Offloading and Resource Allocation in 5G Beyond” by Yueyue Dai, Ke Zhang, Sabita Maharjan, Yan Zhang (arXiv:2011.08442), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.