wikillm

Paper · 2026-08-14 · cs.LG, cs.DB

Probabilistic indirect models for undrained shear strength: addressing significant data missing and variability with advanced imputation and machine learning techniques

Predicts normalised undrained shear strength su(mob)/σ'v from seven Atterberg and CPTU inputs in CLAY/10/7490. Compares three imputers, then probabilistic XGBoost and an attention network, both with normal outputs. Author-reported: MHA-PNN on normal-imputed inputs reaches MAPE 0.34 and RMSE 0.37, versus 1.22 and 0.78 for PXGB on the same inputs.

Missing data imputationGradient-boosted decision treesAttention mechanismUncertainty estimationgeotechnical-engineeringdata-imputationuncertainty-quantification

arXivPDFCodeMarkdown14 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Engineers need to know how strong a clay is when it is loaded quickly, before water can drain out of it. Lab and field tests are slow and costly, so practice often relies on formulas that take easier measurements as inputs. This paper works from a large global clay database in which most cells are blank. Only about a third of the cells are filled in, and the cone-penetration readings are missing for most rows.

The authors fill the gaps in three ways: a simple statistical filler built on a bell-curve assumption, a chained-regression filler, and a random-forest filler. They judge each filler by training the same boosted-tree model on the filled data and comparing it with the model trained on the gaps as they are. They then build two models that output a range of likely strengths rather than one number: a boosted-tree model and a small neural network with an attention layer. The neural network trained on data filled by the simple statistical method has the lowest error and the narrowest ranges of the three network variants, although its coverage is a little lower. The authors stress that filled-in values are not real measurements and that the model is no substitute for site-specific data.

TL;DR

  • Predicts su(mob)/σ'v, a normalised undrained-strength target, from seven easily measured inputs (Atterberg limits and CPTU terms) in CLAY/10/7490 §2.1.
  • Compares three imputers on the inputs (multivariate normal, MICE, miss forest), then trains probabilistic XGBoost (PXGB) and MHA-PNN, both with a normal output §3, §4.
  • Author-reported best: MHA-PNN on multivariate-normal-imputed inputs, MAPE 0.34, RMSE 0.37, coverage 0.94 tab. 6.
  • Reuse the ideas with care. The released code has tuning leakage, an attention layer that attends over one token, and several details the paper does not state (see Pitfalls).

Problem

  • Undrained shear strength (su) matters for geotechnical design. Empirical estimates carry large uncertainty: the paper cites coefficients of variation of 0.06 to 0.56 and biases of 0.40 to 1.22 across testing methods §1.
  • CLAY/10/7490 has 7490 rows from 251 studies in 30 countries or regions, with a completeness of 34.1% §1, §2.1.
  • Missing rates per variable are 45.8% (LL), 39.9% (PI), 49.3% (LI), 86.4% to 91.1% (the four CPTU terms), and 49.5% (su(mob)/σ'v) tab. 1.
  • The target is itself derived: su is "converted from su obtained from various tests using transformation models" §2.1.
  • The seven inputs were chosen because they are easy to obtain. Sensitivity and preconsolidation stress were left out because they are hard to measure §2.1.
  • Rows with three or fewer inputs give unreliable predictions, as the paper shows with the multivariate-normal baseline §2.2, fig. 4 and again in the later models §4.4.

Method

Inputs, target and transforms

Inputs are X1 = LL, X2 = PI, X3 = LI, X4 = (qt−σv)/σ'v, X5 = (qt−u2)/σ'v, X6 = (u2−u0)/σ'v, X7 = Bq; the target is X8 = su(mob)/σ'v §2.1, §2.2. Each variable is mapped to a normal-like scale before modelling eq. 4:

T_i = \sinh^{-1}(X_i) \quad (i = 1, \dots, 6), \qquad T_7 = \operatorname{sigmoid}^{-1}(X_7 / 1.2), \qquad T_8 = \ln X_8 eq. 4

Johnson-system fits per variable eq. 1–3, tab. 2 are reported as descriptions of the marginals. The transform actually used in every script that builds the models is eq. 4 (see Paper vs code).

Multivariate-normal (MN) predictor

Treat T1 to T8 as jointly normal, with means μ and standard deviations σ and correlation matrix R (Table 3). The covariance is eq. 8:

C = \operatorname{diag}(\sigma)\, R\, \operatorname{diag}(\sigma) eq. 8

Given the observed inputs O of a row, the target T8 is normal with the following mean and variance eq. 5–7:

\mu^{\text{pre}} = \mu_8 + C_{8,O}\, C_{O,O}^{-1}\,(T_O - \mu_O) \qquad [\text{eq. 6}]

(\sigma^{\text{pre}})^2 = C_{88} - C_{8,O}\, C_{O,O}^{-1}\, C_{O,8} \qquad [\text{eq. 7}]

Predicted strength is exp(T8). The code takes the median and the 2.5% and 97.5% bounds from 1000 draws of this normal, exponentiated 3_multivariate_normal_model.py. The paper states the mean and variance only.

Pseudocode
# Predictive normal for the log target T8 given observed inputs  [eq. 6, 7]
# Input:  observed (0-based indices into T1..T7), observed_values
#         mu (8,), sigma (8,), corr (8, 8)  [tab. 3]
# Output: mean and variance of T8 (log scale)

def predict_log_target(observed, observed_values, mu, sigma, corr):
    # Covariance from standard deviations and correlations  [eq. 8]
    cov = diag(sigma) @ corr @ diag(sigma)
    # Blocks of the covariance: observed inputs and target (index 7)
    # [code: 0_MN_prediction_model/3_multivariate_normal_model.py#L60-L65]
    cov_oo = cov[observed, observed]
    cov_to = cov[7, observed]
    # Gain applied to the deviation of the observed inputs  [eq. 6]
    gain = cov_to @ inverse(cov_oo)
    mean = mu[7] + gain @ (observed_values - mu[observed])
    # Conditional variance  [eq. 7]
    variance = cov[7, 7] - gain @ cov_to
    return mean, variance

MN imputation

For each missing input cell, use the conditional mean given the observed inputs of the same row eq. 9–10, §2.3. The target is never imputed §2.3. The conditioning uses only the seven input covariances 0_multionramal_model.py. Rows with no observed input are left out of the extended sets 1_pre_extended_data.py.

MICE imputation

Multiple imputation by chained equations: each variable is regressed on the others in turn, cycle after cycle §3.2. The paper runs 10 cycles on each dataset: the full set without label 0 (4,973 rows) and Subsets 1 to 3 (4,316, 3,747 and 809 rows) §3.2. Each is defined by how many inputs a row has observed, so the datasets hold rows with at least 1, 2, 3 and 4 observed inputs. Only the seven inputs are used, so the target is not imputed 1_MICE.py.

Miss Forest (MF) imputation

Start from column means. Then, for each input column in order of fewest gaps, fit a random forest on the rows where that column is observed and predict its gaps. Repeat until the largest change in an imputed value falls below 1e-4, for at most 10 passes. The forests have 80 trees and a maximum depth of 100 §3.3.

Pseudocode
# Iterative random-forest imputation  [§3.3]
# Code: 2_MICE_and_MF_datasets/2_missforest.py#L46-L104
# Input:  table (rows, 7 inputs) with gaps (NaN)
# Output: table with gaps filled

def miss_forest(table, max_iter=10, tol=1e-4):
    # Start from column means  [code: 2_MICE_and_MF_datasets/2_missforest.py#L52-L53]
    filled = fill_gaps_with_column_means(table)
    for iteration in range(max_iter):
        largest_change = 0.0
        # Columns with the fewest gaps go first  [code: ...2_missforest.py#L64]
        for column in columns_by_gap_count(table):
            # Fit on rows where this column was observed in the input
            # [code: ...2_missforest.py#L75-L89]
            model = random_forest(n_estimators=80, max_depth=100)
            model.fit(filled[observed_rows], filled[observed_rows, column])
            # Replace the gaps and track the largest change  [code: ...#L92-L98]
            new_values = model.predict(filled[gap_rows, others])
            change = max_abs(new_values - filled[gap_rows, column])
            largest_change = max(largest_change, change)
            filled[gap_rows, column] = new_values
        # Stop once the imputed values barely move  [§3.3]
        if largest_change < tol:
            break
    return filled

Probabilistic XGBoost (PXGB)

The target is modelled as normal with mean μ and scale σ, both predicted from T1 to T7 eq. 12–13. Training minimises the negative log-likelihood eq. 14:

L = -\sum_{i=1}^{N} \log p(y_i \mid x_i; \theta) eq. 14

The paper does not give the gradients. The code builds them in: each row is duplicated, the even outputs give μ and the odd outputs give log σ², and the gradient and Hessian are the ones below 1_xgb_with_best_paras.py. The boosted model runs for 500 rounds with no early stopping 1_xgb_with_best_paras.py.

Pseudocode
# Gradient and Hessian of the normal negative log-likelihood  [eq. 14]
# Code: 3_PXGB/1_xgb_with_best_paras.py#L66-L84
# Input:  raw (2 per row: mean, log variance), targets (per row)
# Output: gradient and Hessian for each of the 2 outputs per row

def nll_grad_hess(raw, targets):
    mean = raw[0::2]
    log_var = raw[1::2]
    variance = exp(log_var)
    error = mean - targets
    grad_mean = error / variance
    grad_log_var = 0.5 * (1 - error ** 2 / variance)
    hess_mean = 1 / variance
    hess_log_var = 0.5 * error ** 2 / variance
    grad = interleave(grad_mean, grad_log_var)
    hess = interleave(hess_mean, hess_log_var)
    return grad, hess

MHA-PNN

Input T1 to T7 goes through a dense layer, then a multi-head attention layer with a residual connection, then a normal output head §4.2, fig. 11. The code does this as follows 1_mha_pnn_with_best_paras.py.

Pseudocode
# Forward pass of MHA-PNN  [§4.2]
# Code: 4_MHA_PNN/1_mha_pnn_with_best_paras.py#L69-L82
# Input:  features (B, 7), the transformed inputs T1..T7
# Output: normal distribution per row (mean, scale)

def mha_pnn(features):
    # Dense hidden layer, then dropout  [code: lines L70-L71]
    hidden = dropout(activation(dense(features, units)), rate)  # (B, units)
    # One token per row: sequence length 1  [code: lines L72]
    tokens = expand_dims(hidden, axis=1)  # (B, 1, units)
    # Multi-head self-attention, dropout on attention weights  [eq. 15, 16]
    # [code: lines L73-L74]
    attended = multi_head_attention(tokens, tokens, num_heads, key_dim, rate)
    # Residual sum and layer normalisation  [eq. 17] [code: lines L75-L76]
    normed = layer_norm(tokens + attended)  # (B, 1, units)
    # Flatten, dropout, then a two-value output  [code: lines L77-L79]
    flat = dropout(flatten(normed), rate)  # (B, units)
    raw = dense(flat, 2)  # (B, 2)
    mean = raw[:, 0]
    # Softplus keeps the scale positive; 1e-6 for stability  [§4.2]
    # [code: lines L80-L82]
    scale = softplus(raw[:, 1]) + 1e-6
    return normal(mean, scale)

Attention follows eq. 16 (softmax of scaled query–key products, times values) and the output combines the heads with a residual term eq. 17. Training uses Adam at a tuned learning rate, batch size 64 and 2000 epochs, with no early stopping 1_mha_pnn_with_best_paras.py.

Evaluation protocol

  • Stratified 5-fold cross-validation, with the number of observed inputs as the stratification label, and a fixed seed of 42 §4.4 1_mha_pnn_with_best_paras.py.
  • Only rows with at least four observed inputs are scored. Rows with three or fewer are dropped from the tables §4.4 1_calc_metrics.py.
  • Coverage (CR) is the share of true values inside the predicted 95% interval eq. 18. Average interval width (wCI) is the mean of the upper minus lower bound eq. 19. The code computes RMSE, MAPE and R² on the median, on the back-transformed scale 1_calc_metrics.py.
  • Radar chart fig. 14: MAPE, RMSE and wCI are inverted so that higher is better, then all five metrics are min-max scaled.

Hyperparameters

Search ranges (Optuna, 10 trials per dataset §4.3):

  • MHA-PNN: attention heads 2 to 20, key dimension 2 to 20, dense units 20 to 200, dropout 0.2 to 0.5, learning rate 1e-4 to 1e-1, activation ReLU, sigmoid or tanh §4.3 0_auto_tune_hypars_paenn.py.
  • PXGB: learning rate 1e-4 to 1e-1, maximum depth 1 to 20, minimum child weight 1 to 20, γ 0 to 0.2, column subsampling 0.5 to 1, row subsampling 0.8 to 1 §4.3 0_auto_tune_hypars_xgb.py.

Selected values tab. 5. The learning-rate row is labelled ×10⁻² in the paper; the MHA-PNN values are converted here.

SettingPXGB Ori.PXGB MNPXGB MICEPXGB MFMHA-PNN MNMHA-PNN MICEMHA-PNN MF
attention heads––––2079
key dimension––––19213
dense units––––18484152
dropout rate––––0.310.230.41
activation––––ReLUTanhReLU
learning rate0.04290.02860.06870.08151.2e-42.1e-31.0e-3
max tree depth17191120–––
min child weight1010814–––
γ0.160.140.160.18–––
column subsampling0.970.780.790.87–––
row subsampling0.860.860.880.86–––

tab. 5. "MN" in the PXGB columns is the code's "ext" dataset, the MN-imputed one 1_xgb_with_best_paras.py.

Settings not in the paper, from code:

  • PXGB: 500 boosting rounds, fixed 1_xgb_with_best_paras.py.
  • MHA-PNN: Adam, batch size 64, 2000 epochs, no early stopping 1_mha_pnn_with_best_paras.py.
  • MICE: 10 cycles per subset §3.2. The regression model and sampling choices are left to statsmodels' MICEData defaults, which the code does not set 1_MICE.py.
  • Multivariate normal: Table 3 is fitted on the full database. Correlations are estimated by bootstrap §2.2.

Results

Author-reported. Setup: CLAY/10/7490, rows with at least four observed inputs, 5-fold stratified cross-validation, metrics as defined above §4.4. Baselines are the multivariate-normal predictor on the original incomplete data (MN-pred) and PXGB on each dataset tab. 6.

Model (dataset)MAPERMSER²wCICR
MN-pred (Ori.)0.480.59−0.060.720.80
PXGB (Ori.)1.721.17−5.62434.620.97
PXGB (MN)1.220.78−1.3124.860.97
PXGB (MICE)1.000.76−1.2928.810.93
PXGB (MF)1.190.78−1.4044.950.95
MHA-PNN (MN)0.340.370.551.030.94
MHA-PNN (MICE)0.460.450.321.580.98
MHA-PNN (MF)0.460.390.501.470.97

tab. 6. The text's relative changes check out against this table: on the MN data, MHA-PNN against PXGB cuts MAPE by about 72%, RMSE by about 53% and wCI by about 96%, and R² rises from −1.31 to 0.55 §4.4, §5. PXGB on MICE data cuts MAPE by about 42% and RMSE by about 35% against PXGB on original data §4.4.

Other results:

  • Multivariate-normal imputation did not significantly improve the MN-pred model's accuracy §2.3, fig. 6.
  • Single-input and two- or three-input cases are poorly predicted by the MN-pred model. Adding CPTU inputs improves both accuracy and interval width §2.2, fig. 4.
  • Distribution match (Kolmogorov–Smirnov D, lower is better): MICE is closest to the original data for every variable. MN imputation shifts the CPTU variables most. The paper calls the test necessary but not sufficient §3.3, tab. 4.
VariableMN DMN pMICE DMICE pMF DMF p
T10.0400.0020.0090.9860.0290.048
T20.0330.0110.0051.0000.0230.157
T30.0640.0000.0051.0000.0300.036
T40.2220.0000.1360.0000.2000.000
T50.2560.0000.0970.0000.1850.000
T60.2890.0000.1170.0000.2110.000
T70.2740.0000.0440.0700.1660.000

tab. 4

Reference numbers for the input data (not results): missing rates in tab. 1; Johnson parameters for each variable in tab. 2, with a_x, b_x, a_y and b_y as in eq. 1–3; the transformed-space normal parameters and correlations in tab. 3.

Variablea_xb_xa_yb_y
LL0.75−1.187.2933.99
PI0.70−1.104.8815.05
LI1.50−1.190.660.28
(qt−σv)/σ'v0.64−1.100.713.68
(qt−u2)/σ'v0.77−1.140.711.96
(u2−u0)/σ'v0.78−0.870.822.15
Bq2.520.260.470.63
su(mob)/σ'v0.80−1.440.060.14

tab. 2 (all eight fitted as Johnson SU).

T_iμ_iσ_iρ with T1T2T3T4T5T6T7T8
T14.710.351.000.96−0.230.000.020.100.160.07
T24.090.480.961.00−0.33−0.07−0.040.050.180.03
T30.780.31−0.23−0.331.00−0.11−0.17−0.140.01−0.13
T42.460.410.00−0.07−0.111.000.860.77−0.390.80
T52.000.430.02−0.04−0.170.861.000.49−0.610.69
T61.900.380.100.05−0.140.770.491.000.170.65
T7−0.100.450.160.180.01−0.39−0.610.171.00−0.27
T8−1.170.520.070.03−0.130.800.690.65−0.271.00

tab. 3. The MN baseline uses these full-database values for every row §2.2; the code reads them as its only parameters 3_multivariate_normal_model.py.

Paper vs code

  • MICE gives one imputed table per subset. The paper says MICE generates multiple imputed databases that are analysed and combined §3.2 step 5, and that the data are "imputed 10 times" and combined into the final database p. 27. The code runs 10 chained cycles on one chain and keeps one draw per subset 1_MICE.py.
  • Sign in the imputation variance. Equation 11 is written with a plus sign and with standard deviations eq. 11. The code subtracts the covariance term, which is the usual conditional variance 0_multionramal_model.py. The card uses the subtraction, as eq. 7 does.
  • Johnson fits are not the transform. Section 2.2 presents the Johnson system as the transformation §2.2, eq. 1–3. Every script that builds the models uses eq. 4 instead, with arcsinh, logit and log 3_multivariate_normal_model.py. The Johnson fits, which use a fixed quantile z = 0.7 1_fiting_Johnson_distributions.py, feed only Figure 2 and Table 2.
  • Attention runs over one token. Equations 15 and 16 and Figure 11 leave the attention axis open. Figure 11 draws the hidden units as separate boxes. The code gives the attention layer a single token per row 1_mha_pnn_with_best_paras.py, so the softmax has one key and returns weight 1. See Pitfalls.
  • Dropout placement. The paper places dropout in the hidden and attention layers §4.2. The code also applies dropout to the flattened output before the output head 1_mha_pnn_with_best_paras.py.
  • Repository version. The repository's README cites Computers and Geotechnics 187 (2025), article 107488, with the same title [README]. The card follows the arXiv v1 text. The relation between the two versions was not checked.
  • Consistent with the code: the search ranges §4.3, the five-fold stratified split with seed 42 §4.4, the scoring cutoff at four inputs §4.4, the softplus plus 1e-6 scale §4.2, and 10 trials per dataset §4.3.
  • Not checked: the Table 5 and Table 6 values against the repository's spreadsheets. The spreadsheet readers could not run in this session.

Pitfalls

  • Tuning leakage, MHA-PNN. The Optuna objective is the mean held-out NLL over the same five folds that the reported results use, and the best of 10 trials is kept 0_auto_tune_hypars_paenn.py. The reported scores are therefore optimistic. The paper says the search was meant to reduce the influence of hyperparameter choice §4.3.
  • Tuning leakage, PXGB. The objective scores each fold's model on all rows, including its own training rows, not on the held-out fold 0_auto_tune_hypars_xgb.py. Treat the PXGB selection as in-sample.
  • One-token attention. As coded, the attention step applies only value and output projections and mixes no features. The paper's figure suggests a different layout. The reported numbers came from the code version. Implementers should choose deliberately and say which one they used.
  • Intervals are not full uncertainty. The bounds are percentiles of 1000 draws from the predicted normal 3_multivariate_normal_model.py. The network is run once per fold 1_mha_pnn_with_best_paras.py, so the intervals carry no spread over model weights. Simulation noise also enters the bounds. The paper calls the outputs uncertainty quantification.
  • Claims of narrower intervals. The paper says MHA-PNN gives "narrower uncertainty intervals" than the baselines p. 42. Table 6 does not support this against MN-pred (Ori.): wCI 0.72 against 1.03, with coverage 0.80 against 0.94.
  • Coverage is slightly under 95%. The best MHA-PNN row has coverage 0.94 against the nominal 95% tab. 6.
  • Scored subset. Only rows with at least four observed inputs are scored §4.4. The paper does not report separate scores for rows whose inputs were partly imputed and rows with all inputs observed.
  • Metric scale and aggregation are not stated. The code scores the back-transformed values (exponential of the log scale), computes each fold separately and averages the five results 1_calc_metrics.py. R² is sensitive to this: PXGB on the original data has R² of −5.62 tab. 6.
  • Imputation sees the test folds. Imputers are fitted on all rows before cross-validation, and the multivariate-normal parameters come from the full database tab. 3. Only the seven inputs enter the imputers, so the target does not leak through them.
  • Unstated settings. Boosting rounds, epochs, batch size, optimiser, and the MICE regression model are not in the paper. They are in the code or come from statsmodels defaults (see Hyperparameters).
  • Sample counts disagree. Table 1 implies about 7490 rows, Figure 1 ends at row 6264, and Section 3.2 uses 4,973 rows with at least one input. The paper does not reconcile these. The text gives the MN D range for T4 to T7 as 0.222 to 0.290, while Table 4 gives a maximum of 0.289.
  • Row alignment unverified. The MHA and PXGB scripts index the input sheets by row position after Excel export, with index=False 1_MICE.py. I could not confirm that these positions match the rows of su_r.xlsx. Check the alignment before reuse.
  • Source data in the repository. The repository includes the source database file, 0_MN_prediction_model/CLAY_10_7490_TC304.xlsx, with no stated licence. This card reproduces no data from it.

Lineage

  • Builds on: the CLAY/10/7490 database and its transformations and correlations [Ching & Phoon 2014a, 2014b, cited in §2.1–§2.2]; the Johnson system [Slifker & Shapiro 1980, §2.2]; XGBoost, extended here to a normal output §4.1; multi-head attention §4.2; Optuna for tuning §4.3.
  • Compared against: the MN-pred baseline on original data, and PXGB on original, MN, MICE and MF data tab. 6.
  • Superseded by: not known.

Source and license

Adapted from “Probabilistic indirect models for undrained shear strength: addressing significant data missing and variability with advanced imputation and machine learning techniques” by Haibin Xiong, Shaoheng Dai, Peng Lan, Xuzhen He, Chenxi Tong, Sheng Zhang, Daichao Sheng (arXiv:2608.13934), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/haibin-xiong/Probabilistic-indirect-models-for-undrained-shear-strength at commit d5133d511245 (none (no LICENSE file in repository)). Any quoted code is shown under that license.