Feature importance analysis for patient management decisions

In plain words

Hospitals record a lot about each patient over time: test results, drugs given, procedures done and so on. This paper asks which parts of that history a doctor looks at when deciding whether to order a particular lab test or start a particular drug in the next day.

The authors took the records of 4,486 heart-surgery patients and cut each stay into snapshots, one every 24 hours. For each snapshot they wrote down which lab tests and drugs were ordered or given in the following day. They then built many simple summaries of the past, such as how long ago a test was last run, what its last value was, and whether it was going up or down. Each summary was scored on how well it alone predicted the next-day decision.

The main finding is that the most useful signals are often simple. How long ago the same test was last done, and its most recent value, usually predict the next order better than anything else. Combining only a handful of the best signals in a basic classifier already gives useful predictions for many tests and drugs. The paper is about understanding which signals matter, not about a new model.

TL;DR

• Single-feature AUC ranking over about 9,200 temporal features, for 742 binary decisions (335 lab orders, 407 medication decisions), in 4,486 post-surgical cardiac patients p. 1 p. 3.
• The top-ranked feature is most often a "time since last" feature for labs fig. 4, and time since last procedure for medications fig. 7.
• A linear SVM on the top 1, 3 or 30 features reaches AUC 69.6–87.7% (top 1) and 80.5–89.8% (top 30) on 20 selected decisions tab. 1 tab. 2.
• No baseline other than the single-feature and top-k variants; results are author-reported on one hospital's data.

Problem

Clinical records are multivariate time series, and the number of possible temporal features is very large. The paper asks which kinds of temporal features (recent history, trends, simple time constraints, or more complex combinations) best explain what physicians order or prescribe p. 1. Without a feature-level analysis, it is hard to know which signals to build a predictor on. The authors state that a single feature can be a good predictor, while a combination may do better, and that the multivariate models they test are only a limited check p. 3 p. 5.

Method

The paper's sections are unnumbered, so citations give the page.

1. Patient-state segmentation. Each patient stay is cut at 8:00 am every 24 hours. Each cut gives one patient-state example, with features computed from data up to that point p. 1 fig. 1 p. 2. The paper reports 30,828 examples from the 4,486 patients p. 1.
2. Decision labels. For each example, a binary label says whether each lab was ordered in the next 24 hours, and whether each medication was given in the next 24 hours p. 2. The paper uses "orders" and "commissions" for these decisions fig. 3 p. 4.
3. Feature generation. Raw time series are turned into 9,223 features in five groups: labs, medications, visit/demographics, procedures and heart support devices p. 2 p. 3. Continuous labs get 40 temporal features, such as last value, value difference, percentage change, slope, nadir, baseline and 24-hour average, with the full list shown in fig. 2 p. 2. Medications get four features; procedures get four per procedure type across 36 types; devices get one feature each p. 2 p. 3.
4. Grouping. For the analysis, features are grouped into five raw categories and 40 temporal categories, such as "time since last lab" p. 3.
5. Univariate AUC. Each feature is scored by the area under the ROC curve (AUC) for predicting one decision p. 3.
6. Best-predictor counts. For each decision, the feature (or category) with the highest AUC is recorded as the best predictor. The paper plots how often each category is the best predictor over all decisions p. 3 fig. 3 fig. 4 fig. 6.
7. Top-k linear SVM. For each decision, the top 1, 3 and 30 features by AUC are used to train a linear SVM p. 3 p. 5. AUC on held-out patients is reported tab. 1 tab. 2.

# Count which feature category is the best single predictor  [p. 3]
# Input:  auc[decision][feature] for every decision and feature
# Output: counts[category] = number of decisions it wins

def count_best_predictors(auc, category_of):
    # Start every category at zero  [p. 3]
    counts = {category: 0 for category in set(category_of.values())}
    for decision in auc:
        # Pick the single feature with the highest AUC  [p. 3]
        best = argmax(auc[decision])
        counts[category_of[best]] += 1
    return counts

# Univariate ranking, then top-k linear SVM  [p. 3] [p. 5]
# Input:  features (N, F) per patient-state example, labels (N,)
# Output: held-out AUC for k = 1, 3 and 30

def evaluate_top_k(features, labels, ks=(1, 3, 30)):
    # Patients for ranking and training: 2,900  [p. 3]
    # Training set for the SVM is not stated; assumed to be the same subset
    train, test = split_by_patient(num_train_patients=2900)
    # Score each single feature by AUC on the ranking subset  [p. 3]
    ranking = rank_by_single_feature_auc(features[train], labels[train])
    results = {}
    for k in ks:
        # Keep the k best-ranked features; no joint search  [p. 3] [p. 5]
        selected = ranking[:k]
        # Fit a linear SVM; regularisation and scaling are not stated  [p. 3]
        model = fit_linear_svm(features[train][selected], labels[train])
        # Report AUC on the remaining patients  [p. 3] [p. 5]
        scores = model.decision_function(features[test][selected])
        results[k] = auc(scores, labels[test])
    return results

Hyperparameters

Results

Author-reported. Setup: UPMC post-surgical cardiac records p. 1; univariate AUC per feature p. 3; linear SVM on top 1, 3 and 30 features, tested on remaining patients p. 3 p. 5. No external baseline is reported.

Univariate findings (counts read from bar charts, so no exact values):

• For lab orders, the best single predictor is most often a lab feature, then a procedure feature fig. 3. The most frequent winning features are time since the last lab, last lab value, nadir and horizon values fig. 4 p. 3.
• For predicting the order of a different lab, the last value of that lab is the most frequent winner fig. 5 p. 4.
• For medication decisions, procedure and lab features win most often, then medication features fig. 6. The top feature for medications is time since last procedure fig. 7 p. 4.
• The paper reports that one medication's status often predicts another's, so that medications tend to be used together; it avoids combinations that could interact fig. 8 p. 4.

Paper vs code

No official code. The arXiv abstract page links no code or project page, and its comments say the submission is PDF-only with no LaTeX source. Searched GitHub repositories for the title phrase "patient management decisions" and for "Hauskrecht". The matches were unrelated projects or course work, and repositories by others implementing earlier Hauskrecht papers (charmgil/CTBN, charmgil/M-CTBN, charmgil/MLME), which are not this paper's method. A GitHub user named "hauskrecht" exists, but no repository found under it ties to this paper, so it was not used.

Pitfalls

• Headline claim vs tables. The abstract says many decisions "can be well predicted" from a few features. The multivariate tables cover only 10 labs and 10 medications, not all 335 and 407 decisions p. 1 tab. 1 tab. 2. Top-1 AUCs for some drugs (for example Iodixanol at 69.62%) are much lower than the top-30 result tab. 2.
• Data splits are under-specified. The paper says 2,900 patients were used to compute the AUC ranking, and that the models were then assessed on the remaining cases p. 3. It does not say whether the SVM was trained on those 2,900, how many patients or examples were held out, whether the split was by patient or by example, or whether any validation or repeated splits were used. An implementer should split by patient and ensure feature ranking uses only the training part.
• Greedy selection. The top-k features are taken by univariate AUC with no joint search. The authors state that with over 9,000 candidates this greedy step may not be the best choice p. 5.
• Best-predictor counts are confounded by group size. Categories differ in how many features they contain (40 temporal features per continuous lab, 4 per medication, 1 per device) p. 2 p. 3. Counts in fig. 3 to fig. 8 therefore partly reflect how many candidates each category has. This is an inference from the feature counts, not a statement in the paper.
• Terminology. The paper uses "orders" for lab decisions and "commissions" for medication decisions fig. 3 p. 4. Its definition of the medication label says "given" within 24 hours p. 2. Implement the label as stated in the definition, and check any figure's label before comparing.
• Time-since features with no event. The paper does not state the value used for "time since" when a test or procedure never occurred p. 2.
• Timing and leakage. Features are said to use data up to each 8:00 am cut p. 1 fig. 1, but the paper does not show how data entered after the cut are excluded. Check this before reuse.
• Unstated model details. Regularisation C, feature scaling, class weights, missing-value handling and random seeds are not stated p. 3.
• Non-reproducible data. The records come from one hospital's protected database. The paper does not state data availability. An implementer should expect to rebuild the pipeline from their own EHR data.
• Univariate Papaverine figure. The 95% AUC for Papaverine comes from one univariate feature, not from the multivariate models p. 4.

Lineage

• Builds on: none stated in this paper. It says its feature set was also used in earlier work by the same authors, and cites references [1]–[3] for it p. 2 p. 5. None of those are on arXiv in this wiki, so none are linked.
• Compared against: within the paper, the multivariate top-1, top-3 and top-30 linear SVMs are compared with each other; no external baseline is reported p. 3 tab. 1 tab. 2.
• Superseded by: not known.