Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition

In plain words

A skeleton-based action recognizer looks at a sequence of body poses, where each joint (wrist, elbow, knee and so on) is a point in space. A graph convolution lets each joint pool information from the joints it is connected to. The catch is that the connections, the "topology", are usually the same for every feature the network computes. Some features may care about the hands, others about the legs, so one shared wiring is a poor fit.

This paper keeps one shared wiring as a starting point, then gives every feature channel a small correction to it. The correction depends on the particular action being watched, so the wiring adapts to the input. The method is named CTR-GC, and the full network built from it is CTR-GCN. It also adds a temporal module that looks at several time scales. The authors report strong results on three public benchmarks.

TL;DR

CTR-GC learns a shared joint adjacency matrix and refines it, per sample and per channel, with pairwise joint correlations computed from pooled features. Each channel then aggregates joint features with its own refined matrix. In the authors' comparison, CTR-GCN is best on NTU RGB+D 120 (X-Sub and X-Set), NTU RGB+D X-Sub and NW-UCLA, but not on NTU RGB+D X-View tab. 5 tab. 6 tab. 7.

Problem

Graph convolutions for skeletons use one topology for all channels in most earlier work §2.2. The authors argue this forces every channel to aggregate joints the same way, which limits feature extraction §1. Methods that give each channel group its own parameterised topology, such as DC-GCN, become heavy and hard to optimise when set channel-wise, and their topologies are the same for every sample §1 §2.2. The paper asks whether channel-wise, sample-dependent topologies can be modelled cheaply §1.

Method

Framing. A skeleton is a graph whose vertices are the N joints, with an adjacency matrix \mathbf{A}. A standard topology-shared graph convolution transforms each joint feature and sums neighbours weighted by a_{ij} §3.1 eq. 1. The paper treats this as the case of one topology for all channels, and rewrites it as a feature transform followed by an aggregation §3.2 eq. 2.

Feature transform. The input \mathbf{X} \in \mathbb{R}^{N \times C} is multiplied by a learned weight matrix \mathbf{W} §3.2 eq. 3.

Correlation modelling. For each pair of joints, the method pools the features of each joint with two learned linear maps, \psi and \phi. It then computes a per-channel correlation with \mathcal{M}_1, which applies a Tanh to the difference of the two pooled vectors §3.2 eq. 4. An alternative \mathcal{M}_2 uses a small MLP on the concatenation eq. 5. A final linear map \xi lifts the result to C' channels, giving a channel-specific correlation \mathbf{q}_{ij} for each pair eq. 6. The paper notes that \mathbf{Q} is not forced to be symmetric §3.2.

Refinement. The channel-wise topology is the shared matrix plus a scaled correlation, \mathbf{R} = \mathbf{A} + \alpha \mathbf{Q}, where \alpha is a trainable scalar §3.2 eq. 7. \mathbf{A} is learned by backpropagation and shared across channels §3.2.

Aggregation. Channel c is aggregated with its own matrix \mathbf{R}_c, and the C' outputs are concatenated §3.2 eq. 8 fig. 2.

Spatial module. Three CTR-GC branches run in parallel and their outputs are summed. Each branch pools the input over time before it computes correlations, so the topology is inferred from the whole clip §3.4. The neighbourhood of each joint is the whole skeleton graph §3.4.

Temporal module. It follows multi-scale temporal modelling from MS-G3D §3.4. It has four branches, each beginning with a 1x1 convolution that reduces channels. Two branches use temporal convolutions with different dilations, one uses max pooling, and the fourth is a plain 1x1 branch. The four outputs are concatenated §3.4 fig. 3.

Network. CTR-GCN stacks ten basic blocks, followed by global average pooling and a softmax classifier §3.4 [supp. Detailed Architecture]. Each block has a spatial module, a temporal module and a residual connection §3.4 fig. 3. Channel widths are 64 for blocks 1 to 4, 128 for blocks 5 to 7 and 256 for blocks 8 to 10 §3.4. Temporal stride 2 is used in blocks 5 and 8 [supp. Detailed Architecture].

Unified analysis. The paper writes dynamic convolution as a per-neighbour weight eq. 9, and writes each graph convolution as a generalised weight per neighbour eq. 10 eq. 11 eq. 12 eq. 13 eq. 14. It then classifies graph convolutions by how many constraints their weights obey §3.3 tab. 1. The paper argues that CTR-GC is the least constrained class, which implies more representation capacity §3.3. This analysis is qualitative and is not tested directly.

The equations, as numbered in the source §3.2 to §3.3:

\mathbf{z}_i = \sum_{v_j \in \mathcal{N}(v_i)} a_{ij}\,\mathbf{x}_j\mathbf{W} eq. 1

\mathbf{Z} = \mathcal{A}\big(\mathcal{T}(\mathbf{X}),\ \mathcal{R}(\mathcal{M}(\mathbf{X}), \mathbf{A})\big) eq. 2

\widetilde{\mathbf{X}} = \mathcal{T}(\mathbf{X}) = \mathbf{X}\mathbf{W} eq. 3

\mathcal{M}_1(\psi(\mathbf{x}_i), \phi(\mathbf{x}_j)) = \sigma\big(\psi(\mathbf{x}_i) - \phi(\mathbf{x}_j)\big) eq. 4

\mathcal{M}_2(\psi(\mathbf{x}_i), \phi(\mathbf{x}_j)) = \mathrm{MLP}\big(\psi(\mathbf{x}_i) \,\|\, \phi(\mathbf{x}_j)\big) eq. 5

\mathbf{q}_{ij} = \xi\Big(\mathcal{M}\big(\psi(\mathbf{x}_i), \phi(\mathbf{x}_j)\big)\Big) eq. 6

\mathbf{R} = \mathcal{R}(\mathbf{Q}, \mathbf{A}) = \mathbf{A} + \alpha \cdot \mathbf{Q} eq. 7

\mathbf{Z} = \big[\mathbf{R}_1\tilde{\mathbf{x}}_{:,1} \,\|\, \cdots \,\|\, \mathbf{R}_{C'}\tilde{\mathbf{x}}_{:,C'}\big] eq. 8

\mathbf{z}_i^k = \sum_{p_j \in \mathcal{N}(p_i)} \mathbf{x}_j^k \mathbf{W}_j^k eq. 9

\mathbf{z}_i^k = \sum_{v_j \in \mathcal{N}(v_i)} a_{ij}\,\mathbf{x}_j^k \mathbf{W} = \sum_{v_j \in \mathcal{N}(v_i)} \mathbf{x}_j^k (a_{ij}\mathbf{W}) eq. 10

\mathbf{z}_i^k = \sum_{v_j \in \mathcal{N}(v_i)} a_{ij}^k\,\mathbf{x}_j^k \mathbf{W} eq. 11

\mathbf{z}_i^k = \sum_{v_j \in \mathcal{N}(v_i)} \mathbf{p}_{ij} \odot (\mathbf{x}_j^k \mathbf{W}) eq. 12

\mathbf{z}_i^k = \sum_{v_j \in \mathcal{N}(v_i)} \mathbf{x}_j^k \big[p_{ij1}\mathbf{w}_{:,1}, \cdots, p_{ijC'}\mathbf{w}_{:,C'}\big] eq. 13

\mathbf{z}_i^k = \sum_{v_j \in \mathcal{N}(v_i)} \mathbf{x}_j^k \big[r_{ij1}^k \mathbf{w}_{:,1}, \cdots, r_{ijC'}^k \mathbf{w}_{:,C'}\big] eq. 14

Equations 12 and 13 are one align block in the source, numbered as two. Equation 14 is the form the paper assigns to CTR-GC §3.3 [supp. Formula Derivation].

The forward pass of one CTR-GC, in the paper's notation:

# CTR-GC forward pass for one clip  [§3.2] [eq. 2] [eq. 3]
# Input: features (N, C, T, V); V joints, T frames
# Output: features (N, C', T, V)

def ctr_gc(features, shared_adjacency, alpha):
    # Linear transform T(X) = XW  [eq. 3]
    transformed = linear_per_joint(features)          # (N, C', T, V)

    # Pool over time, then reduce channels with psi and phi  [§3.4]
    queries = mean_over_time(phi(features))           # (N, r, V)
    keys = mean_over_time(psi(features))              # (N, r, V)

    # Pairwise correlation per channel, M1 = tanh(difference)  [eq. 4]
    pairs = tanh(queries[:, :, :, None] - keys[:, :, None, :])
    # (N, r, V, V)

    # Lift to C' channels with xi  [eq. 6]
    specific = xi(pairs)                              # (N, C', V, V)

    # Refine the shared topology per channel  [eq. 7]
    topology = shared_adjacency + alpha * specific    # (N, C', V, V)

    # Aggregate each channel with its own matrix  [eq. 8]
    return channel_wise_matmul(topology, transformed)

The spatial module and the block, in the paper's described order:

# Spatial module: three CTR-GC branches, summed  [§3.4]
# Each branch has its own adjacency and alpha  [code: model/ctrgcn.py#L204-L207]
def spatial_module(features, adjacencies, alphas):
    total = 0
    for branch in range(3):
        total = total + ctr_gc(features, adjacencies[branch], alphas[branch])
    return total

# Basic block: spatial, temporal, residual  [§3.4] [fig. 3]
# Batch norm and the spatial residual sit inside the spatial module
# [code: model/ctrgcn.py#L224-L249]
def basic_block(features, stride):
    spatial = relu(spatial_module_norm(features) + spatial_residual(features))
    temporal = multi_scale_temporal(spatial, stride)
    return relu(temporal + block_residual(features, stride))

The temporal module, as implemented in the reference code:

# Multi-scale temporal module: four branches, concatenated  [§3.4]
# Kernel size 5 and dilations 1 and 2 are not stated in the paper
# [code: model/ctrgcn.py#L72-L124, L252-L257]
def multi_scale_temporal(features, stride):
    branches = []
    # Two dilated temporal convolutions, each after a 1x1 reduction
    for dilation in [1, 2]:
        branches.append(temporal_conv(features, 5, dilation, stride))
    # Max pooling over time, after a 1x1 reduction
    branches.append(max_pool_time(features, 3, stride))
    # Plain 1x1 convolution branch
    branches.append(conv1x1(features, stride))
    # Concatenate along the channel axis
    return concatenate(branches, axis=channels)

Hyperparameters

Results

Author-reported. Setup: each dataset's standard cross-subject, cross-view or cross-setup split. NTU RGB+D has 56,880 sequences in 60 classes, NTU RGB+D 120 has 113,945 samples in 120 classes, and NW-UCLA has 1,494 clips in 10 classes §4.1. The setting of the final CTR-GCN row is not stated in the text tab. 5 tab. 6 tab. 7 §4.6.

The Paper's table 2 and table 3 both list the baseline at 83.4%, with 1.22M and 1.21M parameters respectively. The modality used for the ablation tables is not stated.

Additional results: temporal pooling in place of temporal convolution gives 72.8% against 84.9% [supp. Ablation Study]. The number of CTR-GCs per block gives 84.3% (1), 84.7% (2), 84.9% (3), 85.2% (4), 85.4% (5) and 85.0% (6). The paper uses 3 "for fair comparison" with other methods, although 5 scores higher in its own ablation [supp. Ablation Study].

Paper vs code

The official repository is linked from the paper's abstract footnote. It is at commit e13d7582e281d06711eeecb380a472b278ae1663 and is licensed CC BY-NC 4.0 (non-commercial). Only paths and logic are cited below, with no code quoted. Citations use paths relative to the repository root.

• Refinement term. The paper writes \mathbf{R} = \mathbf{A} + \alpha\mathbf{Q} eq. 7. The code multiplies the Tanh-correlation lift by \alpha and adds the shared matrix ctrgcn.py. This matches.
• Alpha initialisation. The paper calls \alpha a trainable scalar and gives no initial value §3.2. The code initialises it to zero, so \mathbf{R} equals \mathbf{A} at the start ctrgcn.py.
• Shared adjacency initialisation. The paper says \mathbf{A} is learned §3.2. The code initialises it from a normalised spatial partition with three subsets (self, inward, outward) tools.py ctrgcn.py.
• Three branches. The paper describes three CTR-GCs in parallel §3.4. The code creates one CTR-GC per adjacency subset, each with its own projections, and sums them ctrgcn.py ctrgcn.py. The paper's equations do not show separate projections per branch.
• Correlation variants. The paper describes M1, M1+ and M2 §3.2 tab. 3. The released CTR-GC implements only the subtraction-and-Tanh form [code: model/ctrgcn.py#L172-L177].
• First layer. For input with 3 or 9 channels, the code fixes the reduced width at 8 and not at C/r ctrgcn.py. The paper does not address the first layer.
• Residual connections. The paper says each block has a residual §3.4. The first block has no block-level residual in the code ctrgcn.py ctrgcn.py.
• Input normalisation. The code applies a batch norm over all input joint coordinates before block 1 ctrgcn.py ctrgcn.py. The paper does not describe it.
• Unused softmax. The code defines a softmax over the joint axis but does not apply it in the forward pass ctrgcn.py ctrgcn.py.
• Temporal kernel and dilations. The paper gives dilations but no kernel size §3.4. The code uses kernel 5 with dilations 1 and 2 [code: model/ctrgcn.py#L76-L78] ctrgcn.py.
• Weight decay, schedule for NW-UCLA. The paper gives weight decay 0.0004 and decays at epochs 35 and 55 §4.2. The NW-UCLA config uses weight decay 0.0001 and decays only at epoch 50 default.yaml default.yaml.
• Fusion weights. The paper says four streams are fused §4.6. The code weights them 0.6, 0.6, 0.4 and 0.4 ensemble.py. For three streams it uses 0.6, 0.6 and 0.4 ensemble.py.
• Bone and motion modalities. The paper does not define them. The code defines bone as the difference from a parent joint feeder_ntu.py and has a separate velocity option feeder_ntu.py.
• Warm-up. The paper says a warm-up strategy is used for 5 epochs §4.2. The code ramps the rate linearly over those epochs main.py.

No discrepancies were found in the channel widths, the number of blocks, the input lengths or the batch sizes for NTU RGB+D and NTU RGB+D 120.

Pitfalls

• Ambiguous setting of the headline numbers. The text says the joint-bone fusion achieves the best NTU RGB+D 120 results §4.6, but the final CTR-GCN row is not labelled with its stream setting tab. 5. Only the joint+bone and bone-only rows are labelled. The fusion weights in the code are not stated in the paper.
• NTU X-View is not a win. The paper says its method is best under nearly all benchmarks §4.6. In the NTU RGB+D table, DDGCN has 97.1 on X-View and CTR-GCN has 96.8 tab. 6.
• Initial value of \alpha. If \alpha starts at zero, the correlations have no effect until training moves it. Implementers who initialise it to a non-zero value will change the early training dynamics [code: model/ctrgcn.py#L223].
• Shared matrix initialisation. Starting from a normalised spatial partition is a design choice the paper does not state. A dense random start may not reproduce the reported numbers tools.py.
• Parameter-count differences. The baseline has 1.22M parameters in one table and 1.21M in another tab. 2 tab. 3. The CTR-GCN count is 1.46M in the ablations tab. 2 tab. 4. The parameter count of the multi-stream model is not given.
• Equation numbering. Numbers here are counted from the equation environments in the source. The supplement's references to equations 8, 12, 13 and 14 match this numbering [supp. Formula Derivation].
• Cost. The table reports about 1.97G FLOPs for CTR-GC against about 1.65G for ST-GC tab. 4. The paper says the refinement adds few parameters §1. The channel-wise topology \mathbf{R} has shape N by N by C' for each sample §3.2, so its memory cost grows with channels. The paper does not break this cost out.
• Training details left out. The paper does not specify the augmentation or the pre-processing from Zhang et al. §4.2, or the bone and motion definitions. These are only in the code (see Paper vs code).
• Correlation symmetry. The paper notes that the channel-specific correlations are not forced to be symmetric §3.2. An implementer who symmetrises them will change the method.
• Analysis is qualitative. The constraint analysis §3.3 tab. 1 is not verified by experiment beyond the ablations.
• Dead code in release. The softmax defined in the release is never applied in the forward pass (see Paper vs code). Do not assume the refinement is normalised.

Lineage

• Builds on: the static topology-shared graph convolution of Yan et al. (ST-GCN), which is the paper's baseline §4.3; the self-attention based dynamic topology of Shi et al. (2s-AGCN) §2.2; the channel-group topologies of Cheng et al. (DC-GCN) §2.2; and the multi-scale temporal module of Liu et al. (MS-G3D) §3.4. The analysis borrows the dynamic convolution framing of Chen et al. and Yang et al. (CondConv) §3.3. These are cited by author name only, with no arXiv ids checked.
• Compared against: Dynamic GCN, DDGCN, MS-G3D, Shift-GCN, SGN, PA-ResGCN and others tab. 5 tab. 6 tab. 7.
• Superseded by: not known from content in this wiki.