Evidence-based anomaly detection in clinical domains

In plain words

Hospitals record what they decided for each patient. A decision can be routine for one kind of patient and rare for a similar one. This paper asks whether a program can flag a decision as unusual for a given patient, by comparing it with decisions made for patients who look like that one.

The method first picks a group of past patients who resemble the patient being checked. It then learns a probability model from that group alone. The model estimates how likely the decision was, given the patient's other details. If that probability is very low, the decision is flagged. The authors test the method on one question, whether pneumonia patients should be admitted to hospital, and compare the flags with the judgements of three physicians.

TL;DR

• Flags a patient-management decision when a probabilistic model, trained on similar past patients, gives it low predictive probability p. 2.
• Similar patients are chosen by a weighted Mahalanobis distance over the other attributes of the case p. 3.
• Author-reported on 100 admission decisions from the PORT dataset: best AUC-ROC 0.81 tab. 2; precision 0.5 at 0.53 recall, quoted in the abstract p. 1 and the discussion p. 5.
• No official code was found. Several parameters and the training details are not stated, so an implementation needs choices the paper does not make (see Pitfalls).

Problem

For one patient case, the target attribute is a management decision A(x), and C(x) holds the other attributes of the case. The goal is to decide whether A(x) is unusual given C(x) p. 1.

Scoring a case against all past cases is biased. If the target falls in a small subpopulation, a decision that is routine for that subpopulation can look anomalous, because the model is dominated by the other patients p. 2. Restricting the model to similar cases fixes this but leaves a second problem: a small subpopulation may not support statistically sound inference p. 2.

The authors frame the output as the basis for monitoring systems that alert clinicians to unusual decisions p. 1. A stated limitation is the assumption that decisions which are anomalous against local practice are worth raising as alerts p. 5.

Method

The paper answers three design questions: how to build the probabilistic model M, how to score the anomaly, and how to choose the similar cases E p. 2. Its equations are unnumbered, so they are cited by page.

1. Probabilistic model. M is a Bayesian belief network (BBN) with structure S_M and parameters θ_M. The paper treats θ_M as random with a prior density given S_M, and predicts an event u by averaging over all parameter settings p. 2:

p(u \mid S_M) = \int_{\theta_M} p(u \mid S_M, \theta_M)\, p(\theta_M \mid S_M)\, d\theta_M

Parameters are learned from the similar cases E by Bayes' rule p. 2:

p(\theta_M \mid \mathbf{E}, S_M) = \frac{p(\mathbf{E} \mid \theta_M, S_M)\, p(\theta_M \mid S_M)}{p(\mathbf{E}, S_M)}

Two assumptions make this tractable: parameter independence, and conjugate priors, so that the posterior keeps the form of the prior and learning reduces to updating sufficient statistics p. 2.

2. Anomaly score. For the target x, the statistic is the predictive probability p(A(x) | C(x), M) of the target attribute given the other attributes p. 2. The paper uses the absolute threshold test: x is anomalous when this probability falls below a fixed threshold p_ε p. 2. The statistic depends on E only through M; it is not compared directly with the cases in E p. 2.

3. Choosing similar cases E. Cases are compared by a distance on C(x) p. 3:

r^2(\mathbf{x}_i, \mathbf{x}_j) = (\mathbf{x}_i - \mathbf{x}_j)\, \Gamma^{-1}\, (\mathbf{x}_i - \mathbf{x}_j)^T

With Γ^{-1} = I this is Euclidean. With Γ set to the population covariance Σ it is Mahalanobis p. 3. The weighted version scales each attribute difference by an importance weight w, where * is element-wise p. 3:

r_w^2(\mathbf{x}_i, \mathbf{x}_j) = \mathbf{w} * (\mathbf{x}_i - \mathbf{x}_j)\, \Gamma^{-1}\, \big(\mathbf{w} * (\mathbf{x}_i - \mathbf{x}_j)\big)^T

Weights come from univariate dependence between each attribute in C and the target A; the experiments use the Wilcoxon rank-sum statistic p. 3 p. 4. E is the k nearest cases, with k = 40 in the experiments p. 4.

The pseudocode below combines the steps. The paper gives no pseudocode; it is written from the text.

# Conditional anomaly test for one target case  [p. 1]
# Input:  target: attributes A(x) (decision) and C(x) (other attributes)
#         past_cases: database of past cases, excluding the target
#         k: size of the similar-case set E (40 in the experiments)
#         weights: one importance weight per attribute in C(x)
#         p_eps: absolute probability threshold
# Output: is_anomalous (bool), statistic (probability)

def conditional_anomaly_test(target, past_cases, k, weights, p_eps):
    # Select the k past cases closest to the target on C(x) by
    # weighted Mahalanobis distance; uniform weights give the plain
    # Mahalanobis variant  [p. 3]
    similar_cases = k_best_matches(
        target, past_cases, k, weights
    )
    # Learn M from E only: a BBN with fixed structure, or Naive Bayes
    # [p. 4]
    model = fit_model(similar_cases)
    # Predictive statistic p(A(x) | C(x), M), averaged over parameters
    # [p. 2]
    statistic = model.predictive_probability(target)
    # Absolute threshold test: a low probability means anomalous  [p. 2]
    is_anomalous = statistic < p_eps
    return is_anomalous, statistic

# Weighted Mahalanobis distance between two cases  [p. 3]
# Input:  x_i, x_j: vectors over C(x), shape (d,)
#         weights: w, shape (d,), one per attribute
#         cov: population covariance matrix, shape (d, d)
# Output: squared distance r_w^2 (the paper's form; ranking is unchanged)

def weighted_mahalanobis_sq(x_i, x_j, weights, cov):
    # Scale each attribute difference by its importance weight  [p. 3]
    scaled_diff = weights * (x_i - x_j)          # element-wise, (d,)
    # Quadratic form with the inverse covariance  [p. 3]
    return scaled_diff @ inverse(cov) @ scaled_diff

Population selection, alternatives. The paper also describes a rejection rule: compute, for each case in a candidate subpopulation, its average distance to its k best matches, and reject the subpopulation if the target's average deviates by more than 2 standard deviations from the others p. 3. The experiments do not say whether this rule was applied.

Hyperparameters

Data

PORT: 2287 patients with community-acquired pneumonia, collected from October 1991 to March 1994 at five medical institutions. The authors use a simpler version that keeps only the attributes that an earlier study identified as most relevant. The target is Hospitalization p. 3. The 18 binary attributes are listed in tab. 1.

Source: tab. 1.

Results

Author-reported. Setup: PORT subset (100 evaluated cases, leave-one-out), gold standard from a three-physician panel, 23 of 100 labelled anomalous p. 3 p. 4. The baseline is a Naive Bayes model trained on all cases p. 5.

Source: tab. 2.

• The PR curve in fig. 1 is for the BBN model with weighted Mahalanobis selection (AUC PR 0.56 in its legend).
• fig. 2 plots sensitivity and specificity against the threshold for the same model. Its legend gives AUC 0.75 for sensitivity and 0.55 for specificity. Sensitivity rises and specificity falls as the threshold increases.
• Headline precision: 0.5 at 0.53 recall, in which 1 of 2 flagged cases was a correct anomaly p. 1 p. 5. The operating threshold for this point is not stated.
• Prevalence-corrected estimate: the authors estimate 0.32 precision at 0.3 recall on the full PORT data, using a conservative prior of anomalies p. 5. The calculation is not shown.

Paper vs code

No official code.

Searched: the paper's links and footnotes (no repository link); GitHub repository search for "conditional anomaly detection" (59 hits, the first 17 checked, none owned by an author); GitHub search for PORT, pneumonia, anomaly and Bayesian (no results); and the GitHub profile matching the first author's surname, which lists zero public repositories. The remaining 42 hits were not checked.

Pitfalls

• Priors and parameter learning are unspecified. The paper says only that parameters are independent and priors conjugate. The family and its hyperparameters are not stated, so the predictive probability depends on a choice the paper does not make p. 2.
• Weights are underspecified. The paper says weights come from a Wilcoxon rank-sum statistic per attribute, but not how that statistic becomes a weight. It also does not say whether the statistic was computed on all 2287 cases or on the training data. If it used the evaluated case, the weights leak the outcome p. 3 p. 4.
• The covariance estimate is not specified. Σ is called the population covariance, but the sample it is estimated from is not stated. With k = 40 neighbours and 18 binary attributes, the estimate may be ill-conditioned. The paper does not discuss this, so any regularisation is an implementer's choice. Not stated.
• Test set is shaped by the baseline. 21 of the 100 test cases were chosen because a Naive Bayes detector flagged them; the other 79 are random. The paper does not discuss how this affects the comparison between Naive Bayes and BBN models p. 3.
• Anomaly rate is high in the test set. 23 of 100 cases are labelled anomalous. The PR figures are on this enriched set, and the authors say precision may be overstated p. 5. Use the corrected 0.32 precision figure with care, because its derivation is not shown.
• Small neighbourhoods hurt the BBN. The BBN trained on 40 best matches is worse than the BBN trained on all cases (AUC ROC 0.79 vs 0.81). The authors attribute this to the complexity of the BBN and to small samples: a smaller sample fits the target better but makes parameter estimates less accurate p. 5 tab. 2.
• Rejection rule is described but not evaluated. The 2-standard-deviation rule for rejecting subpopulations is given in the method, but the experiments do not say it was used. Do not assume it is part of the reported results p. 3.
• No variance reported. Results come from one leave-one-out run over 100 cases with 23 positives. No confidence intervals or repeated runs are reported, so differences such as 0.56 vs 0.52 in AUC PR should not be taken as significant.
• Operating point. The 0.5 precision at 0.53 recall figure has no stated threshold, and the absolute threshold p_ε is not reported for any setting p. 5.
• Minimum size. The paper asks for the minimal subpopulation size that supports sound inference, but gives no numeric rule p. 2.
• Attribute list is fixed. The 18 attributes were picked by an earlier study of PORT p. 3. Results depend on this selection.

Lineage

• Builds on: Bayesian belief networks and structure learning with a marginal likelihood score p. 2 p. 4; Naive Bayes as the baseline p. 4; Mahalanobis distance p. 3. The paper cites the original sources for these; none is on the wiki.
• Dataset: the PORT cohort and the prediction rule derived from it by an earlier study p. 3.
• Compared against: Naive Bayes on all cases, and Naive Bayes and BBN with Mahalanobis and weighted Mahalanobis selection tab. 2.
• Future work: subpopulation smoothing, to reduce the effect of small subpopulations on parameter estimates p. 5.
• Superseded by: not stated.