Deep Fusion Clustering Network

In plain words

Many datasets describe each item in two ways: by its own features, and by its links to other items. This paper groups such items into clusters without any labels, using both views. One small network reads the features alone, and a second one reads the features together with the links. The method blends the two summaries of each item into one fused summary. From that summary it builds a sharper grouping of the items and uses it as a training target, so that the two networks and the blend are trained together and help each other. The authors report better scores than earlier methods on six public datasets.

TL;DR

Deep Fusion Clustering Network (DFCN) trains an autoencoder (AE) and an improved graph autoencoder (IGAE) together. A learned fusion module (SAIF) mixes their embeddings, and a Student-t target distribution from the fused embedding guides all three parts through a triplet KL loss. Clusters come from K-means on the fused embedding. Results are author-reported. The official code differs from the paper in several places, so read Paper vs code before reproducing.

Problem

Clustering without labels needs a training signal. Methods that use only the features (AE-based) ignore the graph structure. Graph-convolution methods mainly reconstruct the adjacency and, according to the authors, under-use node attributes and can over-smooth § Introduction § Related Work. Methods that build their target distribution from one source alone get a less robust signal § Introduction § Related Work: Target Distribution Generation. The input is an attributed graph with N nodes, attribute matrix X and adjacency A, and the number of clusters K is given § Notations alg. 1.

Method

The network has four parts: an AE, an IGAE, a fusion module (SAIF) and the training objective § The Proposed Method. Figure 1 shows the structure fig. 1.

1. Encoders. The AE encodes X into Z_AE. The IGAE, a stack of graph layers, encodes X over the normalised adjacency into Z_IGAE § Fusion-based Autoencoders eq. 1.
2. Shared decoders. Both decoders reconstruct their inputs from the fused embedding Z-tilde, not from their own latent code § Fusion-based Autoencoders.
3. IGAE losses. The IGAE decoder reconstructs the weighted attributes à X and the adjacency, with weight gamma on the adjacency term eq. 2 eq. 3 eq. 4 eq. 5. The AE loss is a mean squared error § Joint loss and Optimization.
4. SAIF fusion § Structure and Attribute Information Fusion fig. 2:
   • a linear mix of the two embeddings with weight alpha eq. 6;
   • one graph-smoothing step, Z_L = Ã Z_I eq. 7;
   • a dense softmax over the inner products of Z_L, giving S, and a global embedding Z_G = S Z_L eq. 8;
   • a skip connection Z-tilde = beta Z_G + Z_L eq. 9.
5. Target distribution § Triplet Self-supervised Strategy. Soft assignments Q of Z-tilde to K centres use a Student-t kernel with v degrees of freedom eq. 10. The target P sharpens Q and renormalises it over clusters eq. 11. Q' and Q'' are the same soft assignments computed from Z_IGAE and Z_AE.
6. Triplet KL loss. The KL divergence from P to the arithmetic mean of Q, Q' and Q'' eq. 12.
7. Objective. Reconstruction terms plus lambda times the KL term eq. 13.
8. Clusters. K-means on the final Z-tilde alg. 1.

Equations are numbered in order of appearance in the LaTeX source. The source algorithm refers to them as Eq. 6 to 13, which agrees with this numbering.

# DFCN training loop  [alg. 1]
# Inputs: features X (N, d), normalised adjacency A_norm (N, N), K,
#         gamma, lambda_, target update interval T, iterations I
# Output: K-means labels from the final fused embedding

def train_dfcn(X, A_norm, K, gamma, lambda_, T, I):
    # Initialise AE, IGAE and fusion parameters  [alg. 1]
    model = init_networks()
    # Initial centres: K-means on the fused embedding  [alg. 1]
    centres = kmeans(model.fused(X, A_norm), K)
    for i in range(1, I + 1):
        # Encode X with the AE and with the IGAE  [eq. 1]
        Z_ae = model.ae_encoder(X)                 # (N, d')
        Z_igae = model.igae_encoder(X, A_norm)     # (N, d')
        # Learned mix of the two embeddings  [eq. 6]
        Z_I = alpha * Z_ae + (1 - alpha) * Z_igae
        # One step of graph smoothing  [eq. 7]
        Z_L = A_norm @ Z_I
        # Dense softmax correlation between nodes  [eq. 8]
        S = softmax(Z_L @ Z_L.T, axis=1)           # (N, N)
        # Global refinement with a skip connection  [eq. 9]
        Z_tilde = beta * (S @ Z_L) + Z_L
        # Student-t soft assignments of the three embeddings  [eq. 10]
        Q = student_t(Z_tilde, centres)
        Q_igae = student_t(Z_igae, centres)
        Q_ae = student_t(Z_ae, centres)
        # Target distribution, refreshed every T iterations  [eq. 11]
        if i % T == 0:
            P = target(Q)
        # Decode from the fused embedding and compute reconstruction terms
        X_hat, Z_hat, A_hat = model.decode(Z_tilde, A_norm)
        L_igae = L_w(Z_hat, A_norm @ X) + gamma * L_a(A_hat, A_norm)
        L_ae = mse(X_hat, X)
        # Triplet KL loss and total objective  [eq. 12] [eq. 13]
        L_kl = KL(P, (Q + Q_igae + Q_ae) / 3)
        loss = L_ae + L_igae + lambda_ * L_kl
        update(model, loss)
    return kmeans(model.fused(X, A_norm), K)

Key equations, as printed in the source:

L_{IGAE} = L_w + \gamma L_a eq. 3

L_w = \frac{1}{2N}\|\widetilde{\mathbf{A}}\mathbf{X} - \widehat{\mathbf{Z}}\|_F^2 eq. 4

L_a = \frac{1}{2N}\|\widetilde{\mathbf{A}} - \widehat{\mathbf{A}}\|_F^2 eq. 5

\mathbf{Z}_I = \alpha \mathbf{Z}_{AE} + (1-\alpha)\mathbf{Z}_{IGAE} eq. 6

\mathbf{S}_{ij} = \frac{e^{(\mathbf{Z}_L\mathbf{Z}_L^{T})_{ij}}}{\sum_{k=1}^{N} e^{(\mathbf{Z}_L\mathbf{Z}_L^{T})_{ik}}} eq. 8

\widetilde{\mathbf{Z}} = \beta \mathbf{Z}_G + \mathbf{Z}_L eq. 9

q_{ij} = \frac{(1+\| \tilde{z}_{i} - u_{j} \|^2/v)^{-\frac{v+1}{2}}}{\sum_{j'}(1+\| \tilde{z}_{i} - u_{j'} \|^2/v)^{-\frac{v+1}{2}}} eq. 10

p_{ij} = \frac{q^{2}_{ij}/\sum_{i}q_{ij}}{\sum_{j'}(q^{2}_{ij'}/\sum_{i}q_{ij'})} eq. 11

L_{KL}= \sum_{i}\sum_{j}p_{ij}\log\frac{p_{ij}}{(q_{ij}+q^{'}_{ij}+q^{''}_{ij})/3} eq. 12

L= L_{AE}+L_{IGAE} +\lambda L_{KL} eq. 13

Eq. 1 and Eq. 2 are the layer rules of the encoder and decoder: each layer multiplies by the normalised adjacency, then by a weight matrix, then applies a non-linearity (sigma) such as ReLU or tanh eq. 1 eq. 2. The paper does not name the exact non-linearity per dataset § Improved Graph Autoencoder.

Hyperparameters

Values in the text are from the paper. Code defaults are from the repository. Where the paper is silent, the table says so.

Results

Author-reported. Setup: six public datasets, 10 runs with mean and standard deviation for DFCN, on a single NVIDIA 2080Ti GPU in PyTorch [§ Training Procedure] § Benchmark Datasets tab. 2. Datasets (samples, classes, dimension): USPS 9298, 10, 256; HHAR 10299, 6, 561; REUT 10000, 4, 2000; ACM 3025, 3, 1870; DBLP 4058, 4, 334; CITE 3327, 6, 3703 tab. 2.

Headline, ACC (DFCN vs the strongest baseline on that dataset, tab. 3):

• USPS: 79.5 vs 78.1 (SDCN)
• HHAR: 87.1 vs 84.3 (SDCN)
• REUT: 77.7 vs 79.3 (SDCN_Q). DFCN is not the best here on ACC.
• ACM: 90.9 vs 90.5 (SDCN)
• DBLP: 76.0 vs 68.1 (SDCN)
• CITE: 69.5 vs 66.0 (SDCN)

Baseline results other than ARGA are copied from the SDCN paper, and ARGA was re-run by the authors § Parameters Setting. Source colours mark the best in red and the runner-up in blue; these are not reproduced here.

Table 3, clustering performance (mean ± std) tab. 3

The paper's text gives DBLP gains over SDCN of 7.9, 4.2, 7.8 and 8.0 points for ACC, NMI, ARI and F1 § Comparison with the State-of-the-art Methods. These are differences of table values, in points, not relative percentages.

Table 4, ablation of the both-source target distribution (mean ± std) tab. 4

On CITE, the +AE variant has a higher F1 (64.4) than DFCN (64.3) tab. 4. The paper's text uses this table to argue that both sources are needed § Influence of Exploiting Both-source Information.

Other reported results. The IGAE with both reconstruction targets beats the adjacency-only variant fig. 3. The SAIF ablation reports 0.5 to 5.0 points of gain for the cross-modality fusion over the baseline, and the triplet strategy beats the single-source variant on all datasets fig. 4. These figure values are described in the text only; the figure itself was not transcribed. The paper reports that lambda around 10 works across datasets fig. 5. The t-SNE plots are qualitative fig. 6. The paper's own summary is that DFCN "consistently outperforms" the baselines; in Table 3 this does not hold for ACC on REUT.

Paper vs code

Official code: github.com/WxTu/DFCN at commit 4bdd381, owned by the first author. It was linked from the paper's introduction § Introduction. The code is organised in DFCN.py, IGAE.py, AE.py, train.py, main.py, opt.py, utils.py and load_data.py. The repository ships DBLP features and labels, pretrained and final checkpoints for all six datasets, and graph files for all six README.md. The raw features for the other five datasets are not in the repository; the README points to external downloads README.md.

Differences between the paper and the code, each with both citations. The code is the reference for what the released model does.

1. Input reduction. Every dataset is reduced with PCA before the network (30, 50, 100, 100, 50, 100 dimensions). The paper's live text does not mention this; the PCA sentence is commented out in the source. Paper gives Table 2 dimensions tab. 2; the code's input is the PCA output main.py main.py.
2. Stop rule and reported number. The paper says optimisation stops when a validation loss plateaus § Parameters Setting. The code computes ACC with the ground-truth labels at every epoch, saves the model when ACC improves train.py, and reports the maximum ACC over epochs with the NMI, ARI and F1 from that epoch main.py. This is label-based model selection.
3. Repeats. The paper reports 10 runs with mean and standard deviation § Training Procedure. The repository runs one seed (default 0) opt.py and has no repeat loop.
4. Batch. The paper's batch size is 256 § Parameters Setting. The code computes the full graph in each step train.py.
5. KL term, geometric mean. The paper averages Q, Q' and Q'' arithmetically inside the logarithm eq. 12. The code averages their logarithms, which is a geometric mean train.py.
6. KL term, normalisation. The paper's KL is a sum over nodes and clusters eq. 12. The code uses batch-mean reduction, which divides by N train.py. The code's KL term is therefore the paper's term divided by N, so lambda = 10 in the code corresponds to a much smaller weight than lambda = 10 in the paper's form.
7. Target update interval. Algorithm 1 refreshes P every T iterations alg. 1. T is not given. The code recomputes P every epoch train.py.
8. Reconstruction scale. The paper's L_w and L_a have a 1/(2N) factor and Frobenius norms eq. 4 eq. 5. The code uses mean squared error over all entries train.py. Relative to the paper's form, the code's L_w is 2/d times and its L_a is 2/N times the paper's, where d is the input dimension after PCA. A gamma of 0.1 therefore has a different effect.
9. Adjacency normalisation. The paper uses D^(-1/2)(A+I)D^(-1/2) § Notations. The code row-normalises, computing D^(-1)(A+I) load_data.py load_data.py.
10. Mix weight alpha. The paper's alpha is a single learnable scalar eq. 6. The code uses a learnable matrix with one value per node and per embedding dimension, initialised to 0.5 DFCN.py.
11. Second mix weight, fixed. The code sets b = 1 - a once at construction DFCN.py. Because b is computed once and stored, it does not follow the updates to a. Read from the code and not run: as written, the IGAE weight stays at 0.5 while the AE weight is trained.
12. Cluster centres. The paper describes the centres u as pre-calculated eq. 10. The code initialises them with K-means but registers them as a trainable parameter and updates them with the optimiser DFCN.py train.py.
13. Layer order in the IGAE. The paper applies the non-linearity after the adjacency multiplication eq. 1. The code applies it to the features before the multiplication, ÷sigma(XW) IGAE.py. The output layer of the decoder also has an activation, except on HHAR IGAE.py.
14. Adjacency reconstruction. The paper's text describes an inner product over multi-level representations without giving the combination § Improved Graph Autoencoder. The code sums two sigmoid inner products, so its values range from 0 to 2, against a target à with values up to 1 DFCN.py IGAE.py IGAE.py.
15. Pretraining and joint stage. The 30 and 100 iteration stages are described in the paper § Training Procedure. The repository does not contain the pretraining script; the README points to demo files README.md. Train() loads a pretrained checkpoint and runs epoch steps, default 200 train.py opt.py.
16. Evaluated embedding. The ACC printed at each epoch is computed on the embedding from before that epoch's update: K-means runs on the Z-tilde from the forward pass, which precedes optimizer.step() train.py train.py.
17. Code licence conflict. The LICENSE file is Apache-2.0. The README says the code "is released for academic research use only" and asks commercial users to contact the author README.md. This card cites code by path and line and quotes none of it. Check the licence before reusing the code.

No further discrepancies were checked beyond the files above.

Pitfalls

• Label-based epoch selection. Reported numbers, as printed by the code, come from the epoch with the best ground-truth ACC main.py. A label-free stop rule, which the paper describes, will give different numbers. The size of that difference is not measured here.
• Iteration versus epoch. The paper counts "iterations" for pretraining and training § Training Procedure but also uses a batch size of 256 § Parameters Setting. The released code is full-graph, so an iteration there is one full-graph step. The paper does not say which one it means.
• Dense N x N matrices. S, the reconstructed adjacency and the target adjacency are all N x N eq. 8 DFCN.py train.py. At N = 10000 (REUT, tab. 2) one float32 matrix is about 400 MB, and the code keeps several. The cost grows quadratically with the number of nodes, and the paper does not mention it.
• Softmax without temperature. S is a softmax of raw inner products eq. 8 DFCN.py. The paper states no temperature. If the embeddings grow in norm, S becomes close to one-hot. Watch the scale of Z_L.
• Student-t degrees of freedom. The paper calls v a degree of freedom and never states its value eq. 10. The code uses 1.0 opt.py.
• Target as a constant. The code computes P from q.data, so P is a constant and gradients do not flow through it train.py. The paper does not say whether P is treated as a constant.
• Algorithm references. The algorithm says P is used to "refine" Q, Q' and Q'' "by Eq. 12" alg. 1. Eq. 12 is the KL loss; there is no separate refinement formula.
• Only lambda is swept. The paper shows a sensitivity sweep for lambda from 0.01 to 100 § Analysis of Hyper-parameter λ fig. 5. No sweep is given for gamma, which is fixed at 0.1 § Parameters Setting. The same two values are used on all six datasets.
• Comparison protocol. Baseline numbers come from the SDCN paper, and DFCN uses SDCN's code and data for preprocessing and testing § Parameters Setting. Differences in protocol with the baselines are therefore not controlled by the authors.
• Source of the heat-kernel graph. The affinity matrix follows SDCN § Benchmark Datasets. The bandwidth is not in the text. The code uses a fixed bandwidth of 1 on the PCA features load_data.py.
• Evaluation metric. ACC uses Kuhn-Munkres matching of clusters to classes § Evaluation Metric utils.py. The macro F1 is computed after that matching, so it depends on the matching, not on the raw labels.

Lineage

• Builds on: Structural Deep Clustering Network (SDCN, Bo et al., WWW 2020). The paper follows its heat-kernel graph construction and reuses its preprocessing code and data, and its comparison uses SDCN's reported numbers § Benchmark Datasets § Parameters Setting. The bibliography has no arXiv id for SDCN. Also DEC and IDEC for the Student-t target distribution § Triplet Self-supervised Strategy, and DANet for the zero-initialised skip weight § Structure and Attribute Information Fusion.
• Compared against: K-means, AE, DEC, IDEC, GAE/VGAE, ARGA, DAEGC, SDCN_Q and SDCN § Comparison with the State-of-the-art Methods.
• Superseded by: not stated.