Discriminability index (d')

Overview

A discriminability index turns two class distributions into one number that says how separable they are. The shared idea is that a larger value means the classes overlap less. For two normals with equal covariance, the true d' is the Mahalanobis distance between the means 2012.14331 §5. With unequal covariances, the indices differ, and only some of them match the Bayes-optimal error 2012.14331 §3.1.4.

Variants

Bayes index d'_b

For equal priors, the Bayes index is defined from the optimal error p(e) as d'_b = -2\,Z(p(e)), where Z is the standard normal quantile. It can also be written with the percent correct p(c) as 2\,Z(p(c)) 2012.14331 §3.1.4. It is the index that matches the Bayes-optimal classifier. Computing p(e) requires integrating a normal over the decision region. Card: Methods to integrate multinormals and compute classification measures.

The paper also reads d'_b as the accuracy at the point on the ROC curve that maximises hit rate minus false-alarm rate 2012.14331 §3.1.5.

Simpson and Fitter d'_a

Let \mathbf{S}_{\text{rms}} = [(\bm{\Sigma}_a + \bm{\Sigma}_b)/2]^{1/2} and \bm{d} = \bm{\mu}_a - \bm{\mu}_b. Then d'_a = \lVert \mathbf{S}_{\text{rms}}^{-1} \bm{d} \rVert 2012.14331 §3.1.4. It needs no integration. In 1D, the paper says it underestimates discriminability by up to about 30% at an extreme ratio of standard deviations 2012.14331 §3.1.4 fig. 4a. Card: the same paper card.

Egan and Clarke d'_e

Let \mathbf{S}_{\text{avg}} = (\mathbf{S}_a + \mathbf{S}_b)/2, where each \mathbf{S} is a sd matrix. Then d'_e = \lVert \mathbf{S}_{\text{avg}}^{-1} \bm{d} \rVert 2012.14331 §3.1.4. It needs no integration. In 1D, the paper reports that d'_e approaches d'_b at large separation 2012.14331 §3.1.4 fig. 4c, and the card notes that d'_e is the fallback when d'_b becomes infinite in 1D 2012.14331 §5.

Two bivariate normals with correlation

For two bivariate normals with correlation \rho, the paper gives a closed form in the per-axis differences d'_x and d'_y: d'^2 = (d_x'^2 + d_y'^2 - 2\rho\, d'_x d'_y) / (1 - \rho^2) 2012.14331 §3.1.4. The card does not say which of the indices above this equals.

When to use which

Implementation sketch

Language-agnostic pseudocode of the closed-form indices. d'_b needs p(e) from a Bayes-error integral, which the card covers in its Method section 2012.14331 §2 2012.14331 §3.1.

# Closed-form discriminability indices for two normals  [arxiv:2012.14331 §3.1.4]
# Input:  mu_a, mu_b (k,); cov_a, cov_b (k, k) covariance matrices
# Output: d_a (Simpson and Fitter), d_e (Egan and Clarke)

def closed_form_indices(mu_a, cov_a, mu_b, cov_b):
    # Mean difference, the d vector  [§3.1.4]
    d = mu_a - mu_b
    # Simpson and Fitter: whiten by the root of the averaged covariance  [§3.1.4]
    s_rms = matrix_sqrt((cov_a + cov_b) / 2)
    d_a = norm(solve(s_rms, d))
    # Egan and Clarke: whiten by the average of the sd matrices  [§3.1.4]
    s_avg = (matrix_sqrt(cov_a) + matrix_sqrt(cov_b)) / 2
    d_e = norm(solve(s_avg, d))
    return d_a, d_e


# Bayes index from the optimal error, equal priors  [§3.1.4]
def bayes_index(p_error):
    # d'_b = -2 Z(p(e)), with Z the standard normal quantile
    return -2 * normal_quantile(p_error)

Pitfalls

• Which index you report. The Bayes index d'_b is not a metric. It is symmetric and positive, but it breaks the triangle inequality 2012.14331 §3.1.4.
• Ordering in 1D. The card states d'_a \le d'_e \le d'_b in 1D 2012.14331 §3.1.4.
• Infinite values. Beyond about 75, p(e) underflows and the paper's d'_b is infinite. The paper says d'_e can stand in for it in 1D only 2012.14331 §3.1.4 2012.14331 §5.
• Sample-based fits. Estimating means and covariances from samples uses the sample means and covariances, and the optimal boundary follows from them 2012.14331 §3.2. The paper leaves to the investigator whether a deviation from normality is small enough for the problem 2012.14331 §3.6 fig. 7b.

Contradictions

None known among the cited papers. Only one card covers this concept so far.

Papers

• Methods to integrate multinormals and compute classification measures: defines d'_b, d'_a and d'_e, and gives the integration methods that compute p(e) for non-quadratic domains.