Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems

In plain words

Machine learning learns from examples. When there are few examples, or when a model must respect known rules such as physical laws, learning from data alone can fall short. This survey looks at the many ways people feed existing knowledge into a learning system alongside the data. Knowledge can be a formula, a logic rule, the output of a simulator, a relation between objects, a probability assumption, or a person's feedback.

The authors sort these methods along three questions: where the knowledge comes from, how it is written down, and at which step of the learning pipeline it is plugged in (the data, the model's structure, the training loss, or the final check of the answers). Those three answers give a map of the field, and the survey shows which combinations are common.

TL;DR

• Defines "informed machine learning" as learning from data plus explicit, formally represented prior knowledge that comes from an independent source §2.2.
• Proposes a taxonomy with three dimensions, eight knowledge representations and four integration points §4.
• Surveys the literature along that taxonomy and names the main paths and open challenges §3.2 §5 §7. It reports no experiments of its own.

Problem

Purely data-driven models can fail when data is scarce, when a model must satisfy natural laws or regulatory or safety rules, and when interpretability matters §1. Conventionally, knowledge is used mainly for labelling and feature engineering, and there it is intertwined with the pipeline rather than kept as a separate source §2.2.1. The paper defines the newer practice as prior knowledge that is given in formal representations and integrated explicitly §2.2 §2.2.2.

The survey notes that several names describe overlapping work, for example physics-informed deep learning, physics-guided neural networks and semantic-based regularization §1.

Method

The paper has two parts: a concept with a taxonomy §2 §4, and a literature classification built on it §3 §5.

Concept. Informed machine learning learns from a hybrid source: data, plus prior knowledge that exists independently of the learning task and is given in formal representations §2.2 §2.2.2 fig. 1. In the pipeline, the knowledge enters through one or more of four components: training data, hypothesis set, learning algorithm and final hypothesis §4.3.

Taxonomy. Three dimensions §3.2.1 fig. 2 tab. 1:

1. Knowledge source: scientific, world or expert knowledge. The list is "neither complete nor disjoint", and a source may fall into more than one category §4.1.
2. Knowledge representation: eight forms, from equations and simulator output to logical rules, graphs, symmetries, probability assumptions and human input §4.2.
3. Knowledge integration: training data, hypothesis set, learning algorithm, final hypothesis §4.3.

Survey procedure. Three literature-reading cycles. The first cycle gathered how papers answer the three questions, the second sorted papers into a first taxonomy draft, and the third re-sorted papers and extended the set §3.1.2. Analysis questions: source, representation, integration §3.1.1.

Central integration patterns, as described in the text (see Results):

• Algebraic equations as a knowledge-based loss term added to the objective §5.1 eq. 1 [Insert 1].
• Logic rules encoded in network architecture, using the KBANN procedure §5.5 [Insert 3].

# Knowledge-based objective  [eq. 1] [§5.1]
# Input:  model f; labelled pairs (x_i, y_i); knowledge loss L_k
# Output: scalar objective to minimise

def informed_objective(model, inputs, labels, weights):
    # Label-based term: usual loss over the labelled pairs  [eq. 1]
    label_term = sum(
        loss(model(x_i), y_i) for x_i, y_i in zip(inputs, labels)
    )
    # Regularisation term R(f) on the model  [eq. 1]
    regularisation = regulariser(model)
    # Knowledge term: penalises violations of the prior-knowledge equations,
    # computed from the inputs and the model only, so it needs no labels
    # [eq. 1] [§5.1]
    knowledge_term = knowledge_loss(model, inputs)
    return (
        weights.label * label_term
        + weights.reg * regularisation
        + weights.knowledge * knowledge_term
    )

# KBANN: build a network from propositional rules  [Insert 3] [§5.5]
# Input:  propositional rules
# Output: network whose weights are then refined by learning

def build_kbann(rules):
    # 1. Get the rules; rewrite them hierarchically if needed  [Insert 3]
    hierarchy = make_hierarchical(rules)
    # 2. Map each rule to units: positive weights for existing dependencies,
    #    negative weights for negated ones  [Insert 3]
    network = map_rules_to_units(hierarchy)
    # 3. Add hidden units that the rules do not define  [Insert 3]
    network = add_hidden_units(network)
    # 4. Perturb the complete set of weights  [Insert 3]
    return perturb_weights(network)

Equation 1 is the only numbered equation in the paper. Notation: f is the model, L the label loss, R the regulariser, L_k the knowledge loss and \lambda the weights eq. 1 §5.1.

f^* = \arg\min_{f} \Big( \lambda_l \sum_i L(f(x_i), y_i) + \lambda_r R(f) + \lambda_k L_k(f(x_i), x_i) \Big) eq. 1

Hyperparameters

The survey has no experiments of its own, so the table lists only the parameters that the method description leaves open.

Results

Survey outcomes, not experiments. Findings are the paper's own classification of the literature.

The main approaches are summarised in a table with taxonomy path, motivation, central idea, challenge and suggested direction tab. 4 §7.

Paper vs code

No official code. The LaTeX source has no code or project links [raw/src/1903.12394/ms.tex]. The arXiv metadata has no project link. A GitHub search for "informed machine learning taxonomy" returned only an unrelated third-party survey repository, which is not by the authors. A search for "informed-machine-learning" in the Fraunhofer-IAIS organisation returned nothing.

Pitfalls

• Survey coverage is not reproducible. The text gives no list of sources, search terms, date range or inclusion rule, and says the authors "do not claim to have covered all available research papers" §1 §3.1.2. Treat the "main paths" as the authors' sample, not as a census.
• Counts are only in figures. The path sizes in the taxonomy figure "reflect the relative count of papers" fig. 2, and the distributions in fig. 3 and fig. 4 are not given as numbers in the text. Do not invent counts.
• Eq. 1 has open choices. The label term sums over samples, but the knowledge term L_k(f(x_i), x_i) is written without a sum eq. 1. The text does not say whether it is averaged over samples, which functional measures "violation", or how the three weights are chosen §5.1 §7.
• Logic relaxation is not specified. Rules become differentiable constraints, "for example via the t-norm" §5.5. The choice of t-norm changes the gradients, and the paper does not pick one.
• A claim, not a derivation. The paper says that data augmentation with virtual examples is mathematically equivalent to a regulariser §5.4. It cites the source for this and does not derive it here.
• Simulation gap. Simulated data can mismatch real data, and the paper names adversarial domain adaptation and domain randomisation as directions §5.3 §7. It gives no measured effect of either.
• Dated and scoped. The arXiv version is v3 (2021-05-28). The title page says it is a preprint of a paper "originally published" in IEEE Transactions on Knowledge and Data Engineering p. 1. Check the journal version for citation details.
• Names overlap. Physics-informed, physics-guided, semantic-regularisation and neural-symbolic terms are used for overlapping methods §1.
• Open problems listed by the paper: how to weight knowledge against labels §7; whether KBANN-style rule encoding is practical for deep networks §7; how to learn rules when they are not given §7; how to comparably evaluate knowledge graphs, since authors often customise them §7.

Lineage

• Related surveys named by the paper: Karpatne et al. (theory-guided data science), Lauer and Bloch (knowledge in support vector machines), Battaglia et al. (relational inductive biases, arXiv:1806.01261) §1. None has a card here.
• Methods cited for the knowledge loss and for physics-guided training: physics-guided neural networks (arXiv:1710.11431) §5.1; physics-informed deep learning, part I (arXiv:1711.10561) §5.2. Neither has a card here.
• Compared against: none; the paper is a survey.
• Superseded by: not stated in the paper; the card does not know of a successor.