wikillm

Paper · 2021-12-21 · cs.CV, stat.ML

Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition

ESE-FN recognises elderly activities from RGB video and skeleton sequences. Two fusion nets add an expansion step to Squeeze-and-Excitation: M-Net fuses across modalities and C-Net fuses across channels. A multi-modal loss ties the fused prediction to the better single modality. Author-reported: 95.9% on ETRI-Activity3D (previous best 93.7%) and 92.4% CS on NTU RGB+D.

Channel attentionMultimodal fusionAttention mechanismactivity-recognitionmultimodal-fusionattentionsqueeze-and-excitation

arXivPDFMarkdown11 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Watching an older person and telling apart "combing hair" from "blowing hair", or "making a call" from "playing with a phone", is hard. The two actions move almost the same way. What differs is a small object in the hand, or a little movement of one hand. A camera video shows the objects but is cluttered by the background. A skeleton, the stick figure of body joints, shows the posture cleanly but not the objects.

This paper combines both inputs. It learns how much to trust each input, and which parts of the combined signal matter. It adds a step that first widens the signal and then squeezes it back down, so that the weighting can look at both fine local detail and the overall picture. A training loss also asks the combined prediction to be no worse than the better single input. The authors report that this gives the best accuracy among the methods they compare on their elderly dataset, and it also does well on a standard general dataset.

TL;DR

ESE-FN fuses RGB video features and skeleton features for elderly activity recognition. It adds an "expansion" step to Squeeze-and-Excitation attention, applied first across modalities (M-Net) and then across channels (C-Net). A multi-modal loss ties the fused prediction to the best single-modality prediction. Author-reported: 95.9% on ETRI-Activity3D against 93.7% for the previous best method they list. Skip it if you need a released implementation: none was found.

Problem

Elderly activities mix whole-body actions with human-object interactions, and many of those interactions are small and local. Some pairs of activities have nearly the same motion trajectory and amplitude, so the difference is only an object or a few local movements §1 fig. 2. The authors argue that RGB video alone lacks temporal information for actions with a clear time structure, and that optical flow, the usual remedy, is too slow to compute §2.1. Skeleton data carries posture and motion but no object information, so it struggles with actions that differ only by the object used §2.2.

Simply concatenating or averaging the two modalities discards the ability to keep useful information and suppress modality-specific noise §1 §2.3. The paper's premise is that fusion needs learned attention, and that this attention should act both across modalities and across channels §2.3 §3.2.

Method

The pipeline has four parts: feature extraction for each modality, modal-wise fusion (M-Net), channel-wise fusion (C-Net), and a multi-modal loss (ML) §3.2 fig. 3.

Background: Squeeze-and-Excitation. The starting point is the SE block of SENet. It averages each channel over space (squeeze), passes the resulting vector through two fully connected layers with a sigmoid, and multiplies the feature maps by the result (excitation) §3.1. In the paper's notation:

W=\sigma(F_1({\text {Relu}}(F_2(Y)))) eq. 2

The authors note that global average pooling sees the whole map at once and can be dominated by background. The new blocks therefore add an expansion step before the squeeze, so that local and global dependencies are both captured §3.1 §3.3.

Feature extraction. Each video is split into T clips, and one frame is sampled at random from each clip §3.2. An RGB backbone maps the frame set to a feature vector f_r, and a skeleton backbone maps the skeleton sequence to f_s eq. 4 eq. 5. The two vectors can have different sizes, so each passes through its own MLP (H_r, H_s) to a common size d, and the results are stacked along a modality axis of length n (the number of modalities) eq. 6 §3.2.

M-Net (modal-wise fusion). The stacked features are transposed and passed through M-Net eq. 7. Inside M-Net:

  1. Several stacked convolutions with different kernel sizes expand the features across the modality axis, using local context eq. 10 §3.3.
  2. Average pooling over the d_m axis squeezes the result to one value per modality eq. 11.
  3. Two fully connected layers with a ReLU between them, then a sigmoid, give a weight per modality eq. 12.
  4. The input features are multiplied by these weights eq. 13.

C-Net (channel-wise fusion). The output of M-Net is transposed and passed to C-Net eq. 8. Inside C-Net:

  1. A single convolution expands the channel axis to a length n1 larger than n. The authors use one layer here, because after M-Net the differences between modalities are smaller and a lighter expansion suffices eq. 14 §3.4.
  2. Average pooling over the n1 axis gives one value per channel eq. 15.
  3. The same two-layer fully connected gate as in step 3 of M-Net gives a channel weight eq. 16.
  4. The features are multiplied by the channel weights eq. 17.

The fused multi-modal feature f_rs is obtained by summing the C-Net output over the modality axis §3.2. The text does not give the exact formula for this sum.

Multi-modal loss (ML). Three cross-entropy losses are computed: one from the RGB feature alone (L_r), one from the skeleton feature alone (L_s) and one from the fused feature (L_rs) §3.5. The training objective combines them as

{\mathcal L}=\alpha \times {\mathcal L}_{rs}+\beta \times ({\text {min}}({\mathcal L}_{r},{\mathcal L}_{s})-{\mathcal L}_{rs}) eq. 18

with α = 0.7 and β = 0.3 in the experiments §4.3 fig. 7. The network parameters are chosen to minimise this loss eq. 9 §3.2.

Pseudocode
# ESE-FN forward pass and training loss  [§3.2] [eq. 4-9] [eq. 18]
# Inputs: a video split into T clips, its skeleton sequence, one label.
# Output: scalar loss to minimise.

def ese_fn_loss(video, skeleton, label):
    # 1. Sample one random frame per clip; T = 64 on ETRI  [§3.2] [§4.2]
    frames = sample_one_frame_per_clip(video, T)
    # 2. Extract one feature vector per modality  [eq. 4] [eq. 5]
    f_rgb = rgb_backbone(frames)              # size d1 not stated
    f_skel = skeleton_backbone(skeleton)      # size d2 not stated
    # 3. Map both to size d and stack along the modality axis  [eq. 6]
    f = concat_modal(mlp_rgb(f_rgb), mlp_skel(f_skel))   # (d, n)
    # 4. Modal-wise fusion on the transposed features  [eq. 7] [§3.3]
    h_m = m_net(transpose(f))
    # 5. Channel-wise fusion on the transposed output  [eq. 8] [§3.4]
    h_mc = c_net(transpose(h_m))
    # 6. Sum over the modality axis to get the fused feature  [§3.2]
    f_rs = sum_over_modalities(h_mc)
    # 7. One cross-entropy loss per feature  [§3.5]
    loss_r = cross_entropy(head_r(f_rgb), label)
    loss_s = cross_entropy(head_s(f_skel), label)
    loss_rs = cross_entropy(head_rs(f_rs), label)
    # 8. Combine with alpha = 0.7 and beta = 0.3  [eq. 18] [§4.3]
    return alpha * loss_rs + beta * (min(loss_r, loss_s) - loss_rs)
Pseudocode
# M-Net: modal-wise expansion, squeeze, excitation  [§3.3] [eq. 10-13]
# Input: transposed features, one row per modality.

def m_net(features):
    # 1. Expand across the modality axis with stacked convolutions  [eq. 10]
    expanded = conv_stack_m(features)         # (m, d_m, 1)
    # 2. Squeeze: average over the d_m axis  [eq. 11]
    squeezed = mean_over_axis(expanded, d_m)  # (m, 1)
    # 3. Excite: FC, ReLU, FC, sigmoid gives one weight per row  [eq. 12]
    weights = sigmoid(fc_3(relu(fc_4(squeezed))))
    # 4. Scale the input by the weights  [eq. 13]
    return features * weights                 # shapes: see Pitfalls
Pseudocode
# C-Net: channel-wise expansion, squeeze, excitation  [§3.4] [eq. 14-17]
# Input: M-Net output, transposed as in eq. 8.

def c_net(features):
    # 1. Expand the channel axis with one convolution  [eq. 14]
    expanded = conv_c(features)               # (d, n1, 1), n1 > n
    # 2. Squeeze: average over the n1 axis, one value per channel  [eq. 15]
    squeezed = mean_over_axis(expanded, n1)   # (d, 1)
    # 3. Excite: FC, ReLU, FC, sigmoid gives one weight per channel  [eq. 16]
    weights = sigmoid(fc_5(relu(fc_6(squeezed))))
    # 4. Scale the input channels by the weights  [eq. 17]
    return features * weights

Hyperparameters

Values are stated in the paper unless marked "not stated". The ablation baselines and the diagnostic study use some different settings; see Pitfalls.

NameValueWhereNotes
Clips per video, T64§4.2one random frame per clip; the diagnostic study uses T = 16 §4.3
RGB backboneResNeXt101 (default); ResNeXt18 also tested§4.2 tab. 1pre-trained, then fine-tuned
Skeleton backboneShift-GCN§4.2pre-trained, then fine-tuned
Backbone optimiserSGD, momentum 0.9§4.2same for both backbones
Backbone learning rate0.1§4.2schedule not stated
Backbone weight decay1e-3 (ResNeXt); 1e-4 (Shift-GCN)§4.2
Backbone batch size128 (ResNeXt18); 32 (ResNeXt101 and Shift-GCN)§4.2
Backbone epochs120 (ResNeXt); 140 (Shift-GCN)§4.2
ESE-FN optimiserSGD, momentum 0.9§4.2
ESE-FN learning rate0.1 (basic)§4.2schedule not stated
ESE-FN weight decay1e-4§4.2
ESE-FN batch size32§4.2
ESE-FN epochs30§4.2 fig. 6the paper says the curves level off around 30 epochs
α0.7§4.3 fig. 7 eq. 18searched over {0.3, 0.5, 0.7, 0.9}
β0.3§4.3 fig. 7 eq. 18searched over {0, 0.3, 0.6, 0.9}
Per-branch losscross-entropy§3.5
Hardware and frameworkPyTorch, Titan RTX GPU§4.2
Conv kernel sizes and layer counts (M-Net, C-Net)not stated§3.3 §3.4 fig. 4 fig. 5M-Net uses "several" convolutions with different kernels; C-Net uses one
Dimensions d, d1, d2, d_m, n1, mnot stated§3.2 §3.3 §3.4n1 > n is given; the values are not
FC layer widths in the gatesnot statedeq. 12 eq. 16
Learning-rate schedule, dropout, augmentation beyond frame samplingnot stated§4.2
Random seeds, repeats, error barsnot stated§4

Results

Author-reported. The main elderly benchmark is ETRI-Activity3D: 112,620 samples from 100 people, in 55 action classes, split by person ID §4.1.

SettingMetricBaselineThis paperSource
ETRI-Activity3D, RGB+Skeleton, ResNeXt101 + Shift-GCNaccuracyFSA-CNN 93.795.9tab. 4
ETRI-Activity3D, RGB+Skeleton, VGG16 + Shift-GCNaccuracyFSA-CNN 93.793.4tab. 4
Same benchmark, other methods listedaccuracyEvolution Pose Map 93.6; c-ConvNet 91.3; Deep Bilinear Learning 88.4not applicabletab. 4
NTU RGB+D, cross-subject (CS)accuracyEvolution Pose Map 91.792.4tab. 5 §4.6
NTU RGB+D, cross-view (CV)accuracyShift-GCN 96.596.7tab. 5 §4.6

The paper's own ablations, on ETRI-Activity3D:

SettingAccuracySource
B1: RGB only93.5tab. 2
B2: skeleton only88.6tab. 2
B3: RGB + skeleton, concatenated, softmax94.0tab. 2
B4: no C-Net (M-Net and ML)95.3tab. 2
B5: no M-Net (C-Net and ML)94.5tab. 2
B6: no ML (M-Net and C-Net, loss L_rs only)95.7tab. 2
B7: full ESE-FN95.9tab. 2
A1: SENet for channel fusion only94.3tab. 3
A2: SENet for modal fusion only94.9tab. 3
A3: SENet for modal and channel fusion95.3tab. 3
A4: C-Net only, with expansion94.5tab. 3
A5: M-Net only, with expansion95.3tab. 3
A6: full ESE-FN (M-Net and C-Net)95.9tab. 3

Backbone accuracies on ETRI-Activity3D, each trained alone with a softmax: ResNeXt18 87.1% (15.60M parameters), ResNeXt101 93.5% (48.16M), Shift-GCN 88.6% (0.42M) tab. 1.

On NTU RGB+D, the table lists CS and CV accuracies for about 20 methods. ESE-FN is shown at 92.4 (CS) and 96.7 (CV), the best in that table. Other entries include MS-G3D at 91.5 and 96.2 and 2s-AGCN at 88.5 and 95.1 tab. 5. The other methods' numbers are taken from a previous paper and were not re-run §4.6.

Paper vs code

No official code.

I checked the paper's LaTeX source, which has no code link, URL or repository mention, and searched GitHub for "ESE-FN", "expansion squeeze excitation fusion" and "elderly activity recognition ETRI". The first query found one repository, owen199711/ESE-FN. It is not used: its README does not name the paper or its authors, it has no license, and its folders (for example studentManage/ and web/) do not look like a research release. Nothing ties it to the authors. The same card therefore has no code field.

Pitfalls

  • Eq. 18 is a linear combination, not a penalty on the absolute difference. Expanded, it reads (α − β) × L_rs + β × min(L_r, L_s). With α = 0.7 and β = 0.3, that is 0.4 × L_rs + 0.3 × min(L_r, L_s) (our algebra from eq. 18 and §4.3). The difference term is not an absolute or squared distance; it is linear in L_rs, so the "consistency" effect comes from the coefficients. The text does not say whether the min term is detached from the gradient, or whether min is taken per sample or per batch.
  • α = 1, β = 0 versus the diagnostic text. The diagnostic study says that β = 0 reduces the multi-modal loss to L_rs §4.3. The B6 ablation sets α = 1 and β = 0 §4.4. Only the α = 1 version is exactly L_rs. With the chosen α = 0.7 and β = 0, the loss is 0.7 × L_rs, which only matches the plain objective up to a learning-rate scale. Implement α and β as stated, and keep this in mind when comparing to the baseline.
  • α and β were tuned on one configuration, then reused. The sweep uses ResNeXt18 with T = 16 §4.3. The final model uses ResNeXt101 with T = 64 §4.2. The paper says the choice is made "without loss of generality", but does not test that. The split used for the sweep is not stated, so the reported best point could be a test-set choice.
  • Shape inconsistencies in M-Net. The text gives both m × d_m > n × d §3.3 and m < n × d §3.3 in the same subsection, with m not defined. Eq. 13 multiplies the input by the M-ESEA weights, but the weights come out of the squeeze of the expanded tensor, so their shape does not obviously match the input (n, d, 1). The exact broadcasting is not stated. The pseudocode above marks this rather than resolving it.
  • Which axis is averaged differs between the two nets. M-Net squeezes over d_m eq. 11; C-Net squeezes over n1 eq. 15. Both are described as "global average pooling". Check the axis in code before copying the design.
  • Fusion by summation is underspecified. The fused feature is "summed in the modal-wise way" §3.2, without a formula. Whether it is a sum or a mean, and over which axis after C-Net, is not stated in the text.
  • Backbone training versus fusion training. Backbones are fine-tuned first, then used as feature extractors §4.2. Eq. 9 says all ESE-FN parameters are optimised §3.2. The text does not say whether the backbones are frozen or trained end-to-end during the fusion stage, which changes the cost and the results.
  • Test-time prediction head. The text does not state which head gives the reported accuracy. The ablation numbers in tab. 2 imply the fused branch, but this is an inference from the table, not a statement in the paper.
  • Single runs, no variance. No seeds, repeats or standard deviations are reported, while the full model's gaps over the ablation baselines are 0.2 to 1.9 percentage points §4.4 tab. 2. Those gaps may be inside run-to-run noise.
  • Comparisons are not controlled. The competing methods in tab. 4 use different modalities (some use depth) and different backbones, and the numbers are copied from earlier papers §4.5. The ESE-FN row with VGG16 is 93.4, below the 93.7 it is compared to.
  • Pre-computed comparisons on NTU. The NTU comparison copies its numbers from a previous paper §4.6. The text does not say how the RGB stream is fed for NTU, and no NTU settings for T, backbone or hyperparameters are given.
  • Typos in source. The NTU table caption says "NUT RGB+D" tab. 5. The paper also says "Shif-GCN" in one place and "ST-GAN" for ST-GCN §2.2. These do not affect the numbers.

Lineage

  • Builds on: Squeeze-and-Excitation Networks, the SE block (named in the text as SENet) §2.1 §3.1 eq. 1-3. ESE-FN adds an expansion step before the squeeze and applies the block across modalities and across channels §3.3 §3.4.
  • Backbones: ResNeXt (RGB) and Shift-GCN (skeleton) §4.2.
  • Compared against: FSA-CNN tab. 4; Evolution Pose Map; c-ConvNet; Deep Bilinear Learning tab. 4; and, on NTU, IndRNN, Beyond Joint, SK-CNN, ST-GCN, Shift-GCN, MS-G3D and others tab. 5.
  • Dataset: ETRI-Activity3D §4.1; NTU RGB+D §4.6.
  • Superseded by: not known from content already in this wiki.

Source and license

Adapted from “Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition” by Xiangbo Shu, Jiawen Yang, Rui Yan, Yan Song (arXiv:2112.10992), used under CC BY-SA 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. This page is licensed under CC BY-SA 4.0.