Unlearnable Examples: Making Personal Data Unexploitable

In plain words

Many image models are trained on photos scraped from the web, often without the people in them agreeing. This paper asks whether a person can change their own photos before posting them, so that a model trained on them learns almost nothing useful. The change must be too small to see.

The idea is to add a faint pattern to each image. The pattern is chosen so that a helper model finds the images easy to fit, with near-zero error. A model trained on the marked images then tends to latch onto the pattern instead of the real content, so its accuracy on normal test images drops sharply. The authors make two kinds of pattern: one for each image, and one shared by all images of the same class.

The approach is only as good as its coverage. If most of the data is left clean, the model still learns from the clean part and the protection mostly disappears. Trained models that are hardened against small perturbations also recover much of their accuracy. The paper includes a face recognition case study.

TL;DR

• Each training image gets a small, bounded noise pattern that makes it "unlearnable": the error on it is pushed close to zero by a helper model during noise generation.
• Class-wise noise (one pattern per class) is the strongest form. Sample-wise noise (one pattern per image) needs more per-image work and is weaker.
• The author-reported results are on CIFAR-10, CIFAR-100, SVHN and a 100-class ImageNet subset, using four target architectures §4.
• Protection is weak if only a fraction of the data is perturbed, and adversarial training removes much of it §4.3 app. D.
• Relevant when you need to audit or reproduce data-protection claims. Do not treat the numbers as a general privacy guarantee.

Problem

Models trained on freely crawled data may use personal data without consent. The paper wants data that a person has changed before publishing to stay visually unchanged but to be unusable for training deep networks §1 §3.1. Unlike privacy-preserving training, the defender has no control over training and can only change their own data once, before upload §3.1.

The threat model: the defender has full access to the data they want to protect, cannot alter it after creating the unlearnable version, and cannot see the full training set §3.1.

Method

Objective. A classifier f is trained on the unlearnable set \mathcal{D}_u = \{(x_i + \delta_i, y_i)\}. The noise \delta is bounded in L_p norm by \epsilon. The aim is for training on \mathcal{D}_u to give a model that does poorly on clean test data §3.1 eq. 1.

Noise forms. Sample-wise noise gives each image its own \delta_i. Class-wise noise gives all images of class k the same \delta_k §3.1.

Error-minimizing noise. The noise is the inner part of a min-min bi-level problem. The inner minimization finds a noise \delta within the \epsilon-ball that lowers the source model's loss on each example; the outer minimization trains the model on the result §3.2 eq. 2. The paper stresses that both levels minimize the same loss, so the model's training must be short, or the noise gets over-fitted and the protection breaks §3.2.

Alternating procedure alg. 1 app. A:

1. Train the source model for M steps on the currently noisy data §3.2 app. A.
2. Update the noise with a few PGD steps that descend the loss §3.2 eq. 3.
3. Repeat until the error on the noisy data is at or below \lambda §3.2 app. A.

Sample-wise PGD update. The noise is moved by a signed-gradient step of size \alpha and projected back into the \epsilon-ball eq. 3. The paper uses T steps per update app. B.

Class-wise update. The same step is applied to every example of class k, and the class noise accumulates over the steps §3.2. The paper states T = 1 for class-wise noise app. B.

# Error-minimizing class-wise noise  [alg. 1] [eq. 2] [eq. 3]
# Input:  clean subset (images, labels); K classes; epsilon; alpha;
#         M (training steps per noise pass); lambda (stop error)
# Output: one noise tensor deltas[k] per class k

def class_wise_noise(images, labels, source_model, epsilon, alpha, M, lam):
    # Start every class noise at zero (the code default)  [code: perturbation.py#L462-L463]
    deltas = zeros(K, C, H, W)
    # Repeat until the source model's error is at most lambda  [§3.2] [alg. 1]
    while error(source_model, images, labels, deltas) > lam:
        # Train the source model on noisy images for M batches  [§3.2] [app. A]
        for step in range(M):
            batch_images, batch_labels = next_batch(images, labels)
            noisy_images = add_class_noise(batch_images, batch_labels, deltas)
            train_source_model(source_model, noisy_images, batch_labels)
        # One descent step of size alpha per example, then clip  [eq. 3] [app. B]
        for batch_images, batch_labels in pass_over(images, labels):
            noisy_images = add_class_noise(batch_images, batch_labels, deltas)
            grads = input_gradient(source_model, noisy_images, batch_labels)
            stepped = noisy_images - alpha * sign(grads)
            # Keep each example's noise inside the epsilon-ball  [eq. 3]
            new_noise = clip(stepped - batch_images, -epsilon, epsilon)
            # Average the per-example noise into each class  [code: perturbation.py#L212-L222]
            for class_index in classes_in(batch_labels):
                class_mean = mean(new_noise[batch_labels == class_index])
                deltas[class_index] = clip(class_mean, -epsilon, epsilon)
    return deltas

The pseudocode above follows the code's per-batch averaging. The paper's text describes accumulation per example; see Pitfalls.

Sample-wise variant. Replace the class-level update with T PGD steps per example: each example keeps its own \delta_i, which is updated over T = 20 steps app. B eq. 3.

Hyperparameters

Table 4. ImageNet to CIFAR-10 class mapping for transferred class-wise noise app. E.

Cost. Generating class-wise noise on CIFAR-10 takes about 10% of standard training time; sample-wise takes about 60% app. B.

Results

Author-reported. Setup: top-1 clean test accuracy (%) of target models trained on clean (\mathcal{D}_c) or unlearnable (\mathcal{D}_u) training sets. Lower \mathcal{D}_u means stronger protection. Noise generated on ResNet-18 §4 tab. 1.

Table 1. Clean test accuracy under full unlearnable training tab. 1.

*ImageNet subset of the first 100 classes tab. 1 note.

Table 2. Partial protection on CIFAR-10 with RN-18, clean test accuracy tab. 2. Columns give the share of unlearnable examples. \mathcal{D}_u + \mathcal{D}_c is the mixed training set; \mathcal{D}_c is trained on the clean share only.

Table 3. Clean accuracy of RN-18 trained on unlearnable CIFAR-10 with stronger augmentation tab. 3 app. D. Clean-data reference: 94.95%.

Table 5. Clean accuracy of RN-18 on unlearnable CIFAR-10 with small patches and mixed noise tab. 5 app. F. Standard training / Fast AutoAugment rows.

Key findings, with figure references:

• Sample-wise noise did not stop training when applied to the whole set with random or error-maximizing noise; class-wise random and error-maximizing noise both did, while error-minimizing noise reduced accuracy below 23% in both forms §4.1 fig. 1.
• Class-wise error-minimizing noise works on all four datasets and transfers from ResNet-18 to the other three models §4.2 tab. 1.
• Partial protection: at 80% unlearnable data, models trained on the clean 20% reach almost the same accuracy as on the mixed set; a single unlearnable class is learned as unlearnable and does not change accuracy on other classes §4.3 fig. 2.
• Data augmentation: the highest accuracy under augmentation on CIFAR-10 is 58.51% (Mixup, sample-wise) §4.3 tab. 3.
• Adversarial training with the 8/255 budget gives 85% on the unlearnable CIFAR-10; raising the noise to 24/255 lowers accuracy to 79% app. D fig. 6.
• Transfer: noise generated on ImageNet (resized to 32×32) brings CIFAR-10 class-wise accuracy to about 10% for three models §4.4 fig. 3. Sample-wise noise generated on CIFAR-10 keeps the "ship" class unlearnable inside CIFAR-100 §4.4 fig. 3.
• Face recognition (WebFace, Inception-ResNet): recognition accuracy on the 50 unlearnable identities is 16%, against 86% for clean identities §4.5 fig. 4. In the fully unlearnable setting, verification AUC on LFW falls to 0.5321 from 0.9975 on clean data §4.5 fig. 7.
• Backdoor comparison: class-wise error-maximizing noise is an effective backdoor trigger; random and error-minimizing noise are not, unless they cover at least 80% of the data app. H fig. 8.
• Other generalization methods: L-BFGS also yields error-minimizing noise, and it is slightly less effective than PGD on CIFAR-10 app. I fig. 9.

Figure 5 (appendix) shows confusion matrices for two classes made unlearnable; in words, the unlearnable classes are misclassified while other classes stay unchanged app. C.

Paper vs code

Official repository: HanxunH/Unlearnable-Examples, commit f347155ee23b002788c6594d1843076d796b2a88, MIT license. The link is in the paper's introduction §1 and the owner is the first author.

Discrepancies found:

• Learning rate. Paper: initial lr 0.025 with cosine schedule app. B. Code: the CIFAR-10 ResNet-18 config sets lr 0.1 resnet18.yaml and main.py has no learning-rate override.
• Class-wise stop error on CIFAR-10. Paper: \lambda = 0.1 for class-wise noise app. B. Code: universal_stop_error=0.01 exp_setting.sh. The CIFAR-10 class-wise run in the scripts uses the sample-wise value.
• Sample-wise stop error on ImageNet. Paper: \lambda = 0.01 for sample-wise noise app. B. Code: universal_stop_error=0.1 exp_setting.sh.
• Class-wise update. Paper describes per-example accumulation into a class noise §3.2. Code averages the per-example noise over each batch of the class and assigns it to the class noise, once per batch perturbation.py. The clip to \pm\epsilon is applied to the averaged value.
• Source-model initialization. Paper: not stated. Code builds the source model from the config and starts it at random weights unless --load_model is passed perturbation.py.
• Starting noise. Paper: not stated. Code starts at zero unless --random_start is passed perturbation.py.
• Source-model learning-rate schedule during noise generation. The scheduler is not stepped inside the noise generation loop, so the source model trains at the config's constant lr during generation (inferred from code; the paper does not say) perturbation.py.
• Image range. Code clamps perturbed images to [0, 1] after each PGD step toolbox.py. The paper's projection is only onto the \epsilon-ball eq. 3.
• Bound. Paper writes the constraint with strict inequality in §4 §4. Code clamps to \le \epsilon toolbox.py.
• Target-training augmentation. CIFAR-10 training transforms in code are random crop with 4-pixel padding and horizontal flip only dataset.py. The paper lists random shift, crop, flip and rotation as already applied app. D.
• Not reviewed. The face-recognition case-study code (configs/face, scripts/face) was not read in detail for this card, so the face numbers are paper-only.

Checked and consistent: \epsilon = 8/255 for CIFAR and SVHN, 16/255 for ImageNet exp_setting.sh; \alpha = \epsilon/10 exp_setting.sh; T = 20 sample-wise and 1 class-wise exp_setting.sh; M = 10 on CIFAR-10 and M = 20 on CIFAR-100 exp_setting.sh; M = 100 on ImageNet exp_setting.sh; training epochs resnet18.yaml.

Pitfalls

• The paper's update rule is ambiguous. §3.2 says the class noise accumulates over the steps. Algorithm 1 applies one perturbation per example and then clips alg. 1. The code averages the per-example result over each batch perturbation.py. Implement the code's batch average unless you have a reason to do otherwise, and say which one you used.
• The noise is updated many times per epoch. The code runs a full pass over the noise subset after each block of M training batches, and the noise is updated once per batch within that pass perturbation.py. A pass over 20% of CIFAR-10 is therefore many small updates, not one.
• Model and noise are optimized together. The source model is trained on the noisy data while the noise is being generated. The paper says these training steps must be limited, compared with standard training, for the noise to be effective §3.2. The exact limit is set by M (see Hyperparameters).
• Quantization. The code adds noise in 0–255 pixel units, clips, then casts to uint8, which truncates rather than rounds dataset.py. The paper does not mention this. The resulting images differ slightly from the nominal \epsilon.
• Augmentation moves the noise. Noise is baked into the stored image before the training transforms dataset.py. Random crop and flip therefore shift or mirror the noise, which is part of why augmentation has an effect.
• Imperceptibility is asserted, not measured. The paper gives no perceptual study. "Imperceptible" rests on the 8/255 bound §4. ImageNet uses 16/255 app. B.
• Partial coverage fails. Protection drops quickly below 100% of the data §4.3 tab. 2. Plan for this when you cannot control what others post.
• Adversarial training removes much of the protection. With the 8/255 budget, accuracy is still 85% on the unlearnable CIFAR-10 app. D. The paper notes this is an open problem app. D.
• Compute and memory. Sample-wise noise needs one tensor per training image. For CIFAR-10 that is 50,000 noise patches, held in memory during the search search_perturbation_noise.sh.
• Number of runs and seeds. Not stated in the paper. The code has a --seed argument with default 0 perturbation.py.
• Results are single-pipeline numbers. Author-reported figures come from one pipeline on four datasets. Do not compare them with other protection methods without matching the threat model.

Lineage

• Builds on: PGD for the inner problem §3.2; adversarial and error-maximizing noise, which the paper inverts into a min-min problem §2 §3.2.
• Compared against: Fawkes (targeted adversarial attack for face tracking) §2; error-maximizing class-wise noise and random noise as baselines §4 tab. 1; data poisoning attacks §2; backdoor attacks as a contrast app. H.
• Superseded by: not stated in the sources used for this card.