Algorithms that Remember: Model Inversion Attacks and Data Protection Law

In plain words

Machine-learning models are usually treated as company property, like a trade secret. This paper asks whether that is the whole story. A model trained on people's data can sometimes be probed to get that data back, or to learn whether a given person was in the training set. If so, the model is not just code. It may hold personal information, and privacy law would then apply to it.

The authors work through what that would mean under European data protection law. A person could ask what the model knows about them, ask for their data to be erased from it, or object to its use. The company holding the model would have to keep it secure and stop keeping old data around without reason. The paper does not settle these questions. It uses them as a test of whether the law's categories still fit modern models.

TL;DR

• Two known attacks, model inversion and membership inference, can expose training data from a model §3. The paper's claim is that this could make such models "personal data" under the GDPR §3(b).
• If so, data-subject rights (access, erasure, objection) and controller duties (security, storage limitation) would apply to the models themselves §4.
• The paper is an analysis with no experiments and no code. Attack results are cited from other work, and the authors say the real-world reach of the attacks is unclear §3, fn. 4.

Problem

Data protection law covers personal data. Models have mostly been treated as intellectual property, so the GDPR's rules were thought to govern the data around a model, not the model itself §1. The paper says this view may be out of date, because trained models can leak their training data §1, §2(a).

Current rules cover models only in limited ways. Data protection rules already apply to training data and to automated decisions. They say little about a model's own data after training or after it has been traded §2(a), §2(b).

Method

The paper's "method" is a legal argument in four parts. It has no algorithm, equations, tables or code. Figure 1 is the only figure. The sketch below only orders the paper's checks; it is not code from the paper.

# Sketch of the paper's argument, not an algorithm it states  [§3] [§3(b)]
# Input:  a trained model and the training data it was built from
# Output: whether the model counts as personal data

def model_is_personal_data(model, training_data, holder):
    # 1. Can a training record be inferred from the model?  [§3]
    inferable = inversion_or_membership_possible(model)
    # 2. If so, does a key dataset exist for pseudonymisation?  [§3(b), p. 6]
    pseudonymised = inferable and key_dataset_held(training_data, holder)
    # 3. Released model: is identification reasonably likely?  [§3(b), p. 7]
    released = reasonably_likely_identifiable(model)
    return inferable and (pseudonymised or released)

1. Describe the attacks. Model inversion starts from a model and partial knowledge of its data, and recovers some training variables for people in the training set. Membership inference does not recover values. It answers whether a given person was in the training set §3(a) (i)–(ii), p. 5–6. Both can be run as black-box attacks (query access only) or as white-box attacks (access to the model's structure and parameters) §3, p. 4. Figure 1 shows both attacks on a schematic of two datasets A and B and a model M(B) fig. 1, p. 5.

2. Argue the model is pseudonymised data. A model that is vulnerable to inversion could be treated as pseudonymised data. The dataset A would then play the role of the key that partly re-identifies it. Under Article 4(5), a controller holding both A and M(B) would have to treat both as personal data §3(b), p. 6.

3. Argue the model is personal data when released. If one party holds A and another holds the released model M(B), the paper relies on the CJEU's Breyer test. That test asks whether identification is reasonably likely, and it does not require the data to sit with one holder §3(b), p. 7. Recital 26 is quoted as the test for reasonable likelihood of identification §3(b), p. 7.

4. Argue membership information is personal data. The paper reads the CJEU's broad sense of "any information" (Nowak) to cover even trivial facts such as membership of a training set §3(b), p. 7. It notes the opposing view (Purtova), which warns that a wide definition could cover ordinary environmental data, and states that its own argument does not depend on that wide reading §3(b), p. 7–8.

From these premises the paper walks through the consequences:

• Information rights. Access (Art. 15) and information at collection (Art. 13(1)(e)) would apply, including the recipients or categories of recipients §4(a)(i), p. 8. Art. 14(1)(f), on the source of data, would apply when data is obtained from another controller. The only listed exemption is Art. 14(5)(b), which allows the duty to be replaced by public disclosure in some cases. The paper reads its safeguard as implying that the source of a trained model may have to be published §4(a)(i), p. 9.

• Erasure. Erasure (Art. 17) could be sought in two ways: retraining the model without the data, or changing the model afterwards. The paper notes that retraining has large energy and time costs, and that changing a model after training is still largely unexplored. In practice, delisting outputs is a common substitute §4(a)(ii), p. 9–10.

• Restriction and objection. Art. 18 (restriction) and Art. 21 (objection) would raise questions about whether querying a model counts as processing. The paper argues that querying a model is like querying the whole dataset, which makes a right to object over the whole model look disproportionate §4(a)(iii), p. 10. Art. 22 on automated decisions is discussed as a more practical route §4(a)(iii), p. 10.

• Security and data protection by design. Controllers must consider whether their systems leak personal data, and Art. 25(1) requires technical and organisational measures §4(b)(i), p. 10–11. The paper reviews two defences: differential privacy, which limits what can be inferred about any one person, and avoiding overfitting, which the paper says reduces but does not remove inversion risk §4(b)(i), p. 11.

• Storage limitation. Training data should not be kept longer than needed. The paper points to concept-drift methods, which gradually forget old data, as a possible approach §4(b)(ii), p. 11–12.

The paper's conclusion is that the attacks give a new reason to treat some models as personal data, but that the scope and enforceability of such rules need more work §5, p. 12–13.

Hyperparameters

Not applicable. The paper gives no training settings, model sizes or experimental parameters, because it reports no experiments §1 §5.

Results

No experiments. The paper reports no new measurements. The figures below are cited from other work and reproduced here only as the paper states them. Author-reported, cited from the source given.

The paper's own footnote says that the practical efficacy of these attacks remains unclear and that the authors know of no documented attacks "in the wild" §3(b), fn. 4, p. 6.

Paper vs code

No official code. The paper has no code link, and the arXiv metadata has no comments field pointing to code. Searched: a GitHub repository search for "model inversion data protection" returned zero repositories. The authors are at UCL, Oxford and Newcastle, and no GitHub organisation was found for them.

Pitfalls

• Not an empirical paper. The attack claims are cited from other work. The authors say the practical reach of model inversion is unclear and that they know of no documented real-world attacks §3(b), fn. 4, p. 6. Do not read the card as evidence that model inversion works at scale.
• The legal conclusions are the authors' reading. The paper presents its arguments as a "probing experiment" and does not claim settled law §1, §5. The Breyer and Nowak readings are contested by the authors' own cited critics §3(b), p. 7–8.
• Scope of "model". The paper treats a model as data only when inversion or membership inference is possible. It says models not amenable to such attacks would not be covered by this argument §5, p. 12.
• Time of writing. The paper was accepted in July 2018, and its cases and guidance come from that period §2–§4. Check any later legal development yourself. This card does not cover it.
• Not legal advice. The card summarises the paper's argument. It does not state what any particular controller must do under the GDPR.
• Figure 1 is described, not reproduced. Its layout is set out in words above. The card does not include the figure fig. 1, p. 5.
• Overlap check not run. Only paper.pdf was available, so wikillm overlap could not compare the text with the source. All prose here was written by hand.

Lineage

• Builds on: the model inversion attacks of Fredrikson et al. [20] and the membership inference attacks of Shokri et al. [28], both cited by the paper. No cards exist for either.
• Compared against: the paper's own framing against the GDPR's personal data definition, the Breyer, Nowak and YS and others CJEU decisions, and Purtova's maximalist reading §3(b), §4.
• Superseded by: not known from content in this wiki.