wikillm

Paper · 2022-10-21 · cs.LG

Twin Contrastive Learning for Online Clustering

Twin contrastive learning (TCL) clusters unlabelled data one pass at a time. It applies contrastive losses to the rows (instances) and columns (clusters) of the batch cluster-probability matrix, adds a cluster-entropy term, and boosts with confident pseudo-labels. Author-reported ACC: 0.887 on CIFAR-10, 0.644 on ImageNet-Dogs.

Deep clusteringContrastive learningClustering evaluationclusteringunsupervised-learningimage-clusteringtext-clustering

arXivPDFCodeMarkdown19 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Clustering sorts unlabelled items into groups. This method uses a neural network that gives every item a score for each group. The trick is to read that score table in two directions. Each row is one item's guess at its group. Each column is one group's profile over the items. Training pulls together two augmented views of the same item, and it also makes each group look the same across its two views. Because each item gets its group from a single forward pass, the method suits data that arrives as a stream.

A second stage keeps only the model's most confident guesses, treats them as labels, and trains on them to fix earlier mistakes. The authors also mix mild and strong augmentations, which they found helps. The reported results are the authors' own.

TL;DR

For unlabelled images or sentences that need to be clustered online, TCL is a single end-to-end network with two contrastive heads, one per direction of the score table. It is worth reading if you need a clustering method that assigns each new item on its own, and you can supply the cluster count. The authors report results competitive with recent methods on five image sets and two text sets. The image results use a ResNet-34 trained from scratch for 1000 epochs §4.2. The released code covers images only.

Problem

Many deep clustering methods alternate between learning features and running an offline clustering step, so they need the whole dataset and tend to accumulate errors across the alternation §1 §2.2. The paper aims for clustering that makes an assignment for each instance from that instance alone, so that it can handle large or streaming data §1. Recent contrastive clustering methods mostly work in several stages, with pseudo-labels generated by a preliminary clustering step §1 §2.2. The cluster count M is a fixed input. The paper suggests estimating it with X-means or Louvain community detection, and says the method is robust to a rough estimate §4.6.6.

Method

TCL has three parts: a contrastive pair construction (CPC) module, an instance-level contrastive head (ICH), and a cluster-level contrastive head (CCH). They are trained jointly, then refined by a confidence-based boosting stage §3.

Pair construction §3.1. For each image or text item, two view families are sampled. The weak family T and the strong family T′ each give one view. For images, the weak family follows SimCLR §3.1 and the strong family uses RandAugment followed by Cutout §4.2 tab. 2. For text, the weak family is synonym replacement and the strong family is EDA §4.2. A shared backbone gives features h for both views §3.1.

Instance-level loss §3.2.1. A two-layer MLP g_I maps h to a 128-dimensional space. Cosine similarity eq. 2 feeds an InfoNCE loss over the 2N views of a mini-batch of N items eq. 3. The two views of one item form the positive pair. Every other view in the batch is a negative. The loss is averaged over all 2N views eq. 4.

Cluster-level loss §3.2.2. A second two-layer MLP g_C maps h to M dimensions, and a softmax turns the output into a soft assignment y §3.2.2. Stacking the assignments of the batch gives an N×M matrix Y. Its columns are M cluster vectors of length N under one view. The strong view gives a second matrix Y′, so there are 2M cluster vectors in total §3.2.2. The same cluster under the two views forms a positive pair, and the other 2M−2 vectors are negatives §3.2.2. The InfoNCE loss on cosine similarity is averaged over the 2M vectors eq. 5 eq. 6 eq. 7. The paper's text writes the average with 1/(2M) but sums over M in its intermediate form eq. 7; the final form used in training sums over 2M eq. 9.

Cluster entropy. To prevent all items from landing in a few clusters, the loss subtracts the entropy H_clu of the batch's cluster marginals eq. 8. Each marginal averages the soft assignments over the batch, and the entropy is taken over the 2M entries §3.2.2. This gives L_clu eq. 9.

Training objective. The training loss is the instance loss plus the cluster loss, with equal weights eq. 1 §3.2. The authors say a tuned weight would break the unsupervised setting, so they sum the two directly §3.2. Training then runs for a fixed number of epochs alg. 1.

Confidence-based boosting §3.3. For each batch, the model predicts on the raw, unaugmented images. For each predicted cluster k, the top γ·N/M items by confidence are kept eq. 10 eq. 11. The paper fixes γ to 0.5 §3.3. Their pseudo-labels are stored in a memory for later batches. Items whose confidence later drops below α = 0.99 lose their stored label eq. 16 §3.3. The boosting loss is L_scl + L_sl eq. 12.

  • L_scl eq. 13 eq. 14 is the instance loss from above with one change: pairs that share a pseudo-label are removed from the denominator. They are not treated as positives. The other view of the same item stays the positive.
  • L_sl eq. 15 is a cross-entropy on the strong view, for items that have a stored label. Each class is weighted by 1/N_c, where N_c is its size in the batch, so large clusters count less. The sum is divided by N_p, the number of labelled items.
Pseudocode
# Training step on one mini-batch of N raw images  [alg. 1] [§3.1]
# Returns the scalar training loss  [eq. 1]
def training_step(images):
    # Two augmented views per image: weak and strong  [§3.1]
    views_weak = weak_augment(images)            # (N, 3, 224, 224)
    views_strong = strong_augment(images)        # (N, 3, 224, 224)

    # Shared backbone f gives features h for both views  [§3.1]
    h_weak = backbone(views_weak)                # (N, 512) for ResNet-34
    h_strong = backbone(views_strong)            # (N, 512)

    # Instance head g_I, then cosine similarity  [§3.2.1] [eq. 2]
    z_weak = l2_normalize(instance_head(h_weak))
    z_strong = l2_normalize(instance_head(h_strong))

    # Cluster head g_C, then soft assignments  [§3.2.2]
    y_weak = softmax(cluster_head(h_weak))       # (N, M)
    y_strong = softmax(cluster_head(h_strong))   # (N, M)

    # Instance loss: InfoNCE over the 2N views, temperature tau_I  [eq. 4]
    loss_ins = info_nce_rows(z_weak, z_strong, tau_ins)

    # Cluster loss: InfoNCE over the 2M cluster vectors, tau_C  [eq. 7]
    loss_clu = info_nce_columns(y_weak, y_strong, tau_clu)

    # Entropy of the batch cluster marginals, subtracted  [eq. 8] [eq. 9]
    marginal_weak = y_weak.mean(0)               # (M,), sums to 1
    marginal_strong = y_strong.mean(0)           # (M,), sums to 1
    h_clu = entropy(marginal_weak) + entropy(marginal_strong)
    loss_clu = loss_clu - h_clu

    # Plain sum of the two losses, with no balancing weight  [eq. 1]
    return loss_ins + loss_clu
Pseudocode
# InfoNCE over rows: the positive of a row is its other view  [eq. 2-4]
def info_nce_rows(z_weak, z_strong, tau):
    # Stack both views into 2N rows of unit length  [§3.2.1]
    rows = concat(z_weak, z_strong)              # (2N, d)
    sims = rows @ rows.T / tau                   # (2N, 2N)
    # Remove each row's similarity with itself  [eq. 3]
    sims = fill_diagonal(sims, -inf)
    # Row i's positive is row i + N, modulo 2N  [eq. 3]
    positive = (arange(2N) + N) % (2N)
    return mean(cross_entropy(sims, positive))   # over 2N rows  [eq. 4]


# InfoNCE over columns: each column is one cluster across the batch  [eq. 5-7]
def info_nce_columns(y_weak, y_strong, tau):
    # Columns become 2M cluster vectors of length N  [§3.2.2]
    clusters = concat(y_weak.T, y_strong.T)      # (2M, N)
    clusters = l2_normalize(clusters)
    sims = clusters @ clusters.T / tau           # (2M, 2M)
    # Remove each cluster's similarity with itself  [eq. 6]
    sims = fill_diagonal(sims, -inf)
    # Cluster k's positive is cluster k under the other view  [§3.2.2]
    positive = (arange(2M) + M) % (2M)
    return mean(cross_entropy(sims, positive))   # over 2M vectors  [eq. 7]
Pseudocode
# Boosting: choose pseudo-labels from the raw image  [§3.3] [eq. 10-11]
def update_pseudo_labels(images_raw, memory, indices):
    # Predict on the raw image with the cluster head  [eq. 10]
    probs = softmax(cluster_head(backbone(images_raw)))  # (N, M)
    conf = probs.max(1)
    pred = probs.argmax(1)
    # Per predicted cluster, keep the top gamma * N / M by confidence  [eq. 11]
    quota = ceil(gamma * N / M)                  # paper gives no rounding
    for k in range(M):
        members = nonzero(pred == k)
        ranked = members[argsort(-conf[members])]
        for i in ranked[:quota]:
            # Keep an existing stored label; only fill empty entries
            if memory[indices[i]] == -1:
                memory[indices[i]] = k
    # Weed out stored labels whose confidence fell below alpha  [eq. 16]
    memory[indices[conf < alpha]] = -1
    return memory
Pseudocode
# Boost loss = L_scl + L_sl  [eq. 12]
def boost_loss(z_weak, z_strong, logits_strong, labels):
    # Labels for both views; -1 means no stored label  [§3.3]
    labels_both = concat(labels, labels)         # (2N,)
    labelled = labels_both != -1                 # (2N,)
    # L_scl: InfoNCE on the 2N views  [eq. 13] [eq. 14]
    # Labelled pairs with the same label leave the denominator, but
    # the positive (other view of the same image) stays in it
    same_label = labels_both[:, None] == labels_both[None, :]
    drop = same_label & labelled[:, None] & labelled[None, :]
    drop = drop & not_self & not_positive
    loss_scl = info_nce_masked(z_weak, z_strong, drop, tau_ins)

    # L_sl: weighted cross-entropy on the strong view  [eq. 15]
    stored = labels[labels != -1]                # the N_p labelled items
    counts = bincount(stored, minlength=M)       # N_c per cluster
    weights = 1 / counts[labels]                 # w_c ~ 1 / N_c  [§3.3]
    log_probs = log_softmax(logits_strong)       # (N, M)
    picked = log_probs[arange(N), labels]        # (N,)
    keep = labels != -1
    loss_sl = -sum(weights[keep] * picked[keep]) / keep.sum()

    return loss_scl + loss_sl

Hyperparameters

NameValueWhereNotes
Backbone (images)ResNet-34, randomly initialised§4.2 model.pyCode also has ResNet-18 and ResNet-50 variants.
Backbone (text)DistilBERT sentence model distilbert-base-nli-stsb-mean-tokens, pre-trained§4.2No text code in the repository.
Image sizeResize to 224×224§4.2The code's eval transform resizes to 256 and centre-crops to 224 transforms.py.
Feature h512-d for ResNet-34§4.2Code adds one Linear(512, 512) after pooling model.py.
Instance head g_I2-layer MLP, output 128§3.2.1 §4.2Hidden width not stated. Code adds BN and ReLU before the first Linear model.py.
Cluster head g_C2-layer MLP, output M§3.2.2Linear weights initialised with trunc-normal, std 0.02 model.py. Not stated in paper.
τ_I (instance)0.5 for all datasets§4.2ImageNet run: 0.2 §4.6.7.
τ_C (cluster)1.0 for all datasets§4.2ImageNet run: 0.2 §4.6.7.
Loss weightsL_ins + L_clu, both weight 1eq. 1 §3.2No balancing weight.
Entropy termsubtracted from L_clu, weight 1eq. 8 eq. 9
Batch size N256§4.2Per GPU in the code train.py. Negatives come from all GPUs loss.py.
OptimiserAdam; lr 1e-4; weight decay 1e-4 (images)§4.2train.py train.py
Optimiser (text)lr 5e-6 backbone, 5e-4 heads§4.2Not in code.
Epochs, training1000 (images), 500 (text)§4.2Code default 1000 train.py.
Epochs, boosting200 (images), 100 (text)§4.2Code default 200 boost.py.
Weak augmentation (images)SimCLR set: ResizedCrop, ColorJitter, Grayscale, HFlip, GaussianBlur. No GaussianBlur on CIFAR.§4.2Parameters are not in the paper; see code transforms.py.
Strong augmentation (images)4 random RandAugment ops with ranges in tab. 2, then Cutout 75×75§4.2 tab. 2See Paper vs code for the Cutout size and op ranges.
Weak augmentation (text)Synonym replacement: 20% of words, top-n from a RoBERTa augmenter§4.2
Strong augmentation (text)EDA: four operations, each with p = 0.2§4.2
Confident ratio γ0.5§3.3 §4.6.2Fixed in boost.py boost.py.
Threshold α0.99§3.3 §4.6.2Fixed in boost.py boost.py.
Boosting batch256§4.6.2 fig. 5
ICH dimension128§4.2 tab. 9
HardwareTITAN RTX 24G and RTX 3090; CUDA 11.0; PyTorch 1.8.0; Ubuntu 18.04§4.2README pins PyTorch 1.11 and CUDA 11.3.1 README.md.
ImageNet setupResNet-50 initialised from MoCo v2; first two blocks frozen; batch 4096; 100 training and 20 boosting epochs; τ_I = τ_C = 0.2; α = 0.99; γ = 0.5§4.6.7Freezing code is commented out; see Paper vs code.
Over-clusteringCluster count doubled (CIFAR-10: 20; CIFAR-100: 40 and 100)§4.6.6

Datasets tab. 1 §4.1 §4.2:

  • CIFAR-10: 60,000 images, both official splits combined; 10 classes.
  • CIFAR-100: 60,000 images, both splits combined; 20 super-classes are the ground truth §4.1.
  • STL-10: 13,000 labelled images from both splits; its 100,000 unlabelled images also feed the instance loss §4.2.
  • ImageNet-10: 13,000 training images; 10 classes.
  • ImageNet-Dogs: 19,500 training images; 15 classes.
  • StackOverflow: 20,000 samples; 20 classes.
  • Biomedical: 20,000 samples; 20 classes.

Results

All numbers are author-reported. Setup: ResNet-34 trained from scratch on images, 1000 training and 200 boosting epochs §4.2. The compared methods are listed in §4.3. The table below shows TCL, its no-head variant TCL_ICH, and a selection of baselines. The full 21-method table is tab. 3. Values are NMI / ACC / ARI, in the paper's units (fractions) tab. 3.

MethodCIFAR-10CIFAR-100 (20 super-classes)STL-10ImageNet-10ImageNet-Dogs
k-means0.087 / 0.229 / 0.0490.084 / 0.130 / 0.0280.125 / 0.192 / 0.0610.119 / 0.241 / 0.0570.055 / 0.105 / 0.020
DAC0.396 / 0.522 / 0.3060.185 / 0.238 / 0.0880.366 / 0.470 / 0.2570.394 / 0.527 / 0.3020.219 / 0.275 / 0.111
DCCM0.496 / 0.623 / 0.4080.285 / 0.327 / 0.1730.376 / 0.482 / 0.2620.608 / 0.710 / 0.5550.321 / 0.383 / 0.182
PICA0.591 / 0.696 / 0.5120.310 / 0.337 / 0.1710.611 / 0.713 / 0.5310.802 / 0.870 / 0.7610.352 / 0.352 / 0.201
CC0.705 / 0.790 / 0.6370.431 / 0.429 / 0.2660.746 / 0.850 / 0.7260.859 / 0.893 / 0.8220.445 / 0.429 / 0.274
SCAN0.796 / 0.861 / 0.7500.485 / 0.483 / 0.3140.703 / 0.818 / 0.661not reportednot reported
SPICE (self-trained)0.734 / 0.838 / 0.7050.448 / 0.468 / 0.2940.817 / 0.908 / 0.812(0.927) / (0.969) / (0.933)(0.498) / (0.546) / (0.362)
PCL0.802 / 0.874 / 0.7660.528 / 0.526 / 0.3630.410 / 0.718 / 0.6700.841 / 0.907 / 0.8220.440 / 0.412 / 0.299
TCL0.819 / 0.887 / 0.7800.529 / 0.531 / 0.3570.799 / 0.868 / 0.7570.875 / 0.895 / 0.8370.623 / 0.644 / 0.516
TCL_ICH (k-means on ICH)0.792 / 0.867 / 0.7370.522 / 0.517 / 0.3370.732 / 0.792 / 0.5640.869 / 0.891 / 0.8230.624 / 0.639 / 0.503

Parenthesised SPICE values are for ImageNet-10 and ImageNet-Dogs. The paper says SPICE uses an ImageNet pre-trained model for those two, while all other methods train from scratch §4.3 tab. 3. The paper says TCL beats SPICE on ImageNet-Dogs without ImageNet pre-training §4.5.1.

Text clustering tab. 4. Values are NMI / ACC.

MethodStackOverflowBiomedical
TF-IDF0.156 / 0.2030.254 / 0.280
Para2Vec0.279 / 0.3260.348 / 0.413
GSDPMM0.306 / 0.2940.320 / 0.281
RecNN0.402 / 0.4050.338 / 0.367
STCC0.490 / 0.5110.381 / 0.436
HAC-SD0.595 / 0.6480.335 / 0.401
ECIC0.734 / 0.7870.413 / 0.478
SCCL0.745 / 0.7550.415 / 0.462
TCL0.786 / 0.8820.429 / 0.498
TCL_ICH0.788 / 0.8070.423 / 0.470

Ablations (author-reported; the paper's ablations are reported without boosting unless noted §4.6).

Boosting tab. 5, CIFAR-10 / ImageNet-Dogs / StackOverflow, NMI / ACC / ARI:

BoostCIFAR-10ImageNet-DogsStackOverflow
None0.790 / 0.865 / 0.7520.518 / 0.549 / 0.3810.751 / 0.860 / 0.731
SL only0.805 / 0.878 / 0.7700.562 / 0.600 / 0.4410.780 / 0.877 / 0.761
SL + SCL0.819 / 0.887 / 0.7800.623 / 0.644 / 0.5160.786 / 0.882 / 0.771

Augmentation mix tab. 7, NMI / ACC / ARI. T = weak, T′ = strong:

ViewsCIFAR-10ImageNet-DogsStackOverflow
T + T0.712 / 0.805 / 0.6480.441 / 0.413 / 0.2620.546 / 0.580 / 0.420
T + T′0.790 / 0.865 / 0.7520.518 / 0.549 / 0.3810.751 / 0.860 / 0.731
T′ + T′0.678 / 0.738 / 0.6140.071 / 0.147 / 0.0270.647 / 0.715 / 0.569

Gain from mixed augmentation on CIFAR-10 tab. 6: SimCLR + k-means goes from 0.699 to 0.734 NMI (T+T to T+T′). TCL goes from 0.705 to 0.790 NMI.

Twin heads tab. 10 and decoupling tab. 8, NMI / ACC / ARI:

SettingCIFAR-10ImageNet-DogsStackOverflow
ICH + CCH0.790 / 0.865 / 0.7520.518 / 0.549 / 0.3810.751 / 0.860 / 0.731
w/o ICH0.633 / 0.658 / 0.5220.376 / 0.366 / 0.2210.735 / 0.842 / 0.706
w/o CCH0.734 / 0.821 / 0.6750.504 / 0.535 / 0.3360.666 / 0.732 / 0.516
Decoupled (Yes) tab. 80.770 / 0.854 / 0.7300.518 / 0.535 / 0.3850.747 / 0.857 / 0.725
Shared subspace (No) tab. 80.601 / 0.640 / 0.4810.343 / 0.368 / 0.2140.687 / 0.745 / 0.616

Over-clustering tab. 11, NMI / ACC / ARI. The ACC is computed by majority vote, per the paper §4.6.6:

Target clustersCIFAR-10CIFAR-100
Standard (10 / 20)0.790 / 0.865 / 0.7520.477 / 0.481 / 0.303
Doubled (20 / 40)0.759 / 0.846 / 0.7180.520 / 0.579 / 0.380
100 (CIFAR-100 only)-0.493 / 0.566 / 0.359

ImageNet-1K tab. 12, NMI / ACC / ARI. Baseline is the MoCo v2 model TCL inherits. The † row is without boosting:

MethodNMIACCARI
MoCo v2 (inherited) with k-means0.61860.30470.1428
TCL† (no boosting)0.63320.31600.1901
TCL0.67110.37890.2656

Qualitative results: t-SNE of features over training fig. 3; confident predictions and clustering quality over epochs on CIFAR-10 fig. 4; sensitivity to α, γ and batch size on STL-10 fig. 5; sensitivity to τ_I and τ_C on ImageNet fig. 6 and α and γ on ImageNet fig. 7.

Paper vs code

The official repository is Yunfan-Li/Twin-Contrastive-Learning, owned by the first author. The paper's abstract says the code "will be released on GitHub", and the repository's README links the paper README.md. No LICENSE file is present, so the code is cited but not copied. The repository has 9 Python files and no config files. It covers image data only: CIFAR-10, CIFAR-100, ImageNet-10 and ImageNet train.py data.py. Searches: the first author's GitHub repository list, and a GitHub repository search for "twin contrastive clustering", which returned this repository and third-party forks. The fork-named repositories were not used.

  • CIFAR-100 ground truth. The paper uses 20 super-classes §4.1. The data loader inherits torchvision's CIFAR-100 class and never selects coarse labels, so as far as the code shows, the targets are the 100 fine classes data.py data.py. The number of clusters also defaults to 10 train.py, and the README gives no value for CIFAR-100. Check this before comparing CIFAR-100 numbers.
  • Gaussian blur on CIFAR. The paper leaves GaussianBlur out of the weak augmentation for CIFAR-10/100 §4.2. The code's default blur probability is 1.0 and CIFAR does not override it, so blur is applied to every CIFAR weak view transforms.py transforms.py transforms.py.
  • Shear and translate ranges. tab. 2 lists ±0.3 for Shear X/Y and Translate X/Y. The code's augmentation list uses ±0.1 for all four transforms.py.
  • Cutout size. The paper says one 75×75 Cutout §4.2. The code draws the side length uniformly from 1 to 75 transforms.py.
  • Operation sampling. The paper says four randomly selected operations §4.2. The code samples with replacement, so an operation can repeat, and Identity is one of the 14 candidates transforms.py transforms.py.
  • Evaluation view. The paper resizes all images to 224×224 §4.2. The test transform resizes to 256 and centre-crops to 224 transforms.py. The strong view resizes to 224 without a crop transforms.py. The boosting pseudo-labels use this test transform engine.py.
  • Pseudo-label temperature. Eq. (10) has no temperature eq. 10. The boosting code applies softmax to c divided by the cluster temperature τ_C before computing confidence engine.py. With τ_C = 1.0 (the default and most experiments), this changes nothing. With τ_C = 0.2 (the ImageNet run), confidences are much sharper and the α = 0.99 threshold means something different.
  • Quota rounding. Eq. (11) gives n = γN/M with no rounding. The code uses ceil(γN/M), which is 13 for N = 256 and M = 10 loss.py.
  • Self-labelling normalisation. Eq. (15) divides by N_p, the number of labelled items eq. 15. The code uses PyTorch's weighted cross-entropy, which divides by the sum of the weights, not by N_p loss.py. The class weights are total/N_c rather than 1/N_c, but a constant factor cancels in that normalisation, so the difference is in the denominator.
  • Positive pairs in L_scl. Read literally, Eq. (13) removes every same-label pair from the denominator, including the positive when both views share a pseudo-label. The code keeps the other view of the same image in the denominator and removes the other same-label pairs loss.py loss.py.
  • Entropy term only in distributed mode. The entropy term is computed only inside the distributed branch of the cluster loss. The return statement adds it unconditionally, so a non-distributed call would fail with an undefined variable loss.py loss.py. Training always uses distributed=True train.py.
  • Boosting skips training on its first epoch. The first boosting epoch only generates pseudo-labels. The code continues before the gradient step engine.py. Eq. (12) and alg. 1 describe every boosting epoch as an update.
  • Boosting pseudo-labels use eval-mode BatchNorm. The model is set to eval mode for the label pass, then back to train mode for the update engine.py.
  • Frozen blocks for ImageNet. The paper freezes the first two blocks and trains only the last two §4.6.7. The freezing code in model.py is commented out, so all parameters train model.py.
  • Over-clustering accuracy. The paper uses majority vote for ACC in over-clustering §4.6.6. The code has no majority-vote logic. Its ACC always comes from Hungarian matching, and the Munkres-based alternative get_y_preds is defined but not called evaluate.py evaluate.py.
  • Projection heads. The paper says two-layer MLPs §3.2. The code adds an input BatchNorm and ReLU, and makes the hidden width equal the backbone output (512 for ResNet-34, 2048 for ResNet-50) model.py.
  • Mixed precision. The code trains and evaluates with autocast engine.py engine.py. The paper does not mention it.
  • Batch size. The paper states batch size 256. The code treats 256 as per GPU and gathers embeddings across GPUs for the contrastive loss train.py loss.py.
  • Not in the repository. The text pipeline, the STL-10 unlabelled set, ImageNet-Dogs, the text hyperparameters, and the index lists for ImageNet-10 are absent. The README says the ImageNet-10 and ImageNet-Dogs index lists are in a dataset folder, but the repository has none README.md. Ablation switches (decoupling, head removal, boost components) are not exposed as flags train.py boost.py.

Pitfalls

  • Boosting resume and epoch count. boost.py defaults --resume to checkpoint-0.pth, which is the state after the first training epoch boost.py. Loading a checkpoint sets the start epoch to the checkpoint's epoch plus one misc.py. Training saves its final checkpoint as epoch 999 train.py. A boosting run resumed from that checkpoint starts at epoch 1000. Its range then runs to --epochs, which defaults to 200, so with the defaults no epochs run boost.py. Set --resume to the final training checkpoint, and set --epochs to the training count plus the boosting count (1200 for images). I read this from the code and did not run it.
  • Pseudo-label memory is not checkpointed. The label tensor is rebuilt as all -1 at the start of every boosting run boost.py. A resumed boosting run starts with no stored labels.
  • Distributed setup is required. The losses call dist.get_world_size() and gather embeddings across processes, so a run needs an initialised process group, even on one GPU loss.py loss.py. The README launches with torch.distributed.launch on four GPUs README.md.
  • Cluster count sets the cluster head size. The cluster head outputs M dimensions, so M is a training-time parameter, not just a metric choice. Over-clustering changes the head size §4.6.6. For ImageNet the paper reports running out of GPU memory at batch 4096 even on eight RTX 3090 GPUs, which is why it inherits a MoCo v2 model and freezes blocks §4.6.7.
  • Batch size and cluster count. The paper says that with batch 256 and 100 clusters, there are about two samples per cluster on average, and that empty clusters in a batch hurt the cluster loss §4.6.6. Cluster-level learning needs a large batch.
  • Feature size. The paper's best CIFAR-10 result uses a 64-d ICH, not the default 128-d (NMI 0.798 against 0.790) tab. 9. The default is still 128.
  • Decoupled-row ambiguity in the paper. Table 8's "Yes" rows (0.770, 0.518, 0.747) match the ICH-dimension rows of Table 9 where the dimension equals the cluster count (10, 15, 20). The same default setting is reported as 0.790 NMI in Tables 5, 7 and 10 tab. 5 tab. 7 tab. 9 tab. 10. Treat the decoupling number as unclear.
  • Eq. (7) normalisation. The text gives the averaged loss as (1/2M) summed over k = 1..M eq. 7, while the final loss sums over 2M eq. 9. Use the final form. The code does the same loss.py.
  • Test-time view. Assignment is the argmax of the cluster head on one forward pass of the test-transformed image, with no augmentation §3 alg. 1 model.py engine.py.
  • Results come from a fixed cluster count. The cluster count is an input, and the paper's robustness claim covers only the tested over-clustering ratios §4.6.6.
  • Compute. Each run covers 1000 training and 200 boosting epochs on ResNet-34 at 224×224, with batch 256 §4.2. Each step passes both views through the backbone, so it runs two forward passes model.py.
  • Version. The fetched source is arXiv v1 (2022-10-21), which uses the Springer journal template. The fetched metadata lists only v1. The numbers here are from that manuscript, not from the published IJCV article.

Lineage

  • Builds on: Contrastive Clustering (Li et al., AAAI 2021). The paper calls this work "a significant extension" of that conference paper §2.3. It adds the confidence-based boosting, the mixed augmentation, and the text extension §2.3.
  • Related contrastive work: SimCLR for the weak augmentation and the InfoNCE loss §3.1 §3.2.1. MoCo and BYOL for the contrastive learning background §2.1. RandAugment and EDA for the strong augmentations §4.2.
  • Compared against: k-means, SC, AC, NMF, AE, DAE, DCGAN, DeCNN, VAE, JULE, DEC, DAC, ADC, DDC, DCCM, IIC, PICA, CC, SPICE, SCAN and PCL for images §4.3. TF, BOW, SkipVec, Para2Vec, GSDPMM, RecNN, STCC, HAC-SD, ECIC and SCCL for text §4.3.
  • Superseded by: not stated in the paper.

Source and license

Adapted from “Twin Contrastive Learning for Online Clustering” by Yunfan Li, Mouxing Yang, Dezhong Peng, Taihao Li, Jiantao Huang, Xi Peng (arXiv:2210.11680), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/Yunfan-Li/Twin-Contrastive-Learning at commit 00bc8550e4eb (none (no LICENSE file in repository)). Any quoted code is shown under that license.