Feature scaling

Overview

Feature scaling changes each numeric input column before training and testing, so its values fall in a new range or spread. Most scalers are linear: they subtract a centre value and divide by a spread, with both chosen per column 2212.12343 §2, eq. 1. The others divide by a power of ten, or apply a non-linear map such as the quantile transformation, tanh or the logistic function 2506.08274 §3.1.8–§3.1.11. In both studies the scaler is fitted on the training part of a split and then applied to the test part, so the test data does not shape the scaling 2212.12343 §4.2 2506.08274 §4.3. Time-series models cannot use the statistics of a whole series, because the future would leak into the scale. The forecaster in 2505.14766 §3.1 therefore standardizes each patch with the mean and deviation of the past and current values only.

Three studies measure the effect of scaling. All three are author-reported and use different scalers, models and datasets, so their numbers are not directly comparable. A fourth study applies one scaler inside a fraud-detection pipeline but does not compare it with no scaling, so it says nothing about the size of the effect 2608.07471 §3.4:

• The choice of scaling technique matters for classification performance compares five scalers and no scaling on 82 binary KEEL datasets with 20 classifiers, using 5-fold cross-validation 2212.12343 §4.1.
• The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks compares 12 scalers and no scaling on 16 UCI datasets (9 classification, 7 regression) with 14 algorithms, on one 70/30 split 2506.08274 §4.1–§4.3.
• This Time is Different: An Observability Perspective on Time Series Foundation Models removes its causal patch scaling in one ablation. Held-out observability NLL rises by 27.3% without it, the largest of the four components tested 2505.14766 app. E. The ablation removes the step from a model that was trained with it, so it measures the step, not a choice between scalers.
• Application of Artificial Intelligence for Fraudulent Banking Operations Recognition standardizes two columns (Time and Amount) of one card-fraud dataset before seven classifiers are compared by AUC 2608.07471 §3.4 §4.2.

A fifth card, 2411.04700 §3, trains an SVM and a neural network on sensor statistics. It does not compare scalers, and the paper does not say whether its features were scaled 2411.04700 §3. It is listed here only so that readers see where scaling is unreported; it adds no evidence on the effect.

Every claim below comes from one of these five cards and is labelled with its source.

Variants

The first five entries below are the scalers in the 2212 study. The entries marked "2506 only" are used only in the 2506 study, and the entry marked "2505 only" is used only in the time-series model. Each entry gives the rule and the source.

Standard scaler (z-score)

Subtracts the mean and divides by the standard deviation 2212.12343 §2.1, eq. 3 and 2506.08274 §3.1.3, eq. 3. The 2212 paper does not say whether that deviation is the sample or population form 2212.12343 §2.1. The 2506 code uses the population form (np.std, ddof = 0) scalers.py.

Among the five 2212 scalers, this one has the highest mean F1 for Bagging, OLA and the Perceptron, and the highest mean G-Mean for SVM_RBF and the Perceptron 2212.12343 tab. 3. Its wins are concentrated in low-imbalance datasets 2212.12343 tab. 5. The 2608 fraud study standardizes only Time and Amount and describes the step in words as centring on the mean and dividing by the standard deviation 2608.07471 §3.4. It does not say whether the mean and deviation came from the training split 2608.07471 §3.4 fig. 3. It reports no unscaled run, so its results cannot show whether the step helped 2608.07471 §4.2.

Min-max scaler

Maps the column to [0, 1] using its minimum and maximum 2212.12343 §2.2, eq. 6 and 2506.08274 §3.1.1, eq. 1. The general form for any target range is given in 2212.12343 eq. 7. For GP it has the highest mean F1 and G-Mean of the five 2212 scalers 2212.12343 tab. 3. The minimum and maximum are set by single values, so the paper's results do not show how it behaves with outliers; the study does not test that. Test values outside the training range can fall outside [0, 1] 2506.08274 §3.1.1.

Maximum absolute scaler

Divides each value by the largest absolute value in the column, so values land in [-1, 1] and zeros stay at zero 2212.12343 §2.3, eq. 8. It does not shift the centre. For SVM_lin it has the lowest mean F1 and G-Mean of the five 2212 scalers, and both are below no scaling 2212.12343 tab. 3.

Robust scaler

Centres on the median and divides by the interquartile range 2212.12343 §2.4, eq. 9. The paper gives no quantile range or interpolation rule, so the library defaults apply 2212.12343 Pitfalls. The robust scaler has the highest mean G-Mean for QDA and GNB among the 2212 scalers 2212.12343 tab. 3. The study does not test datasets with extreme values 2212.12343 §2.4.

Quantile transformer

Maps each column through its empirical distribution function to a uniform value, then through the inverse normal function, which gives roughly Gaussian output 2212.12343 §2.5. Values outside the range seen in training are clipped to the bounds 2212.12343 §2.5. The number of quantiles is not stated 2212.12343 §2.5. Across all datasets this transformer has the most wins, and it also has the most wins in the medium and high imbalance strata 2212.12343 tab. 5. It is not the best choice for every model: for GP, its F1 is 0.32, against 0.52 with no scaling 2212.12343 tab. 3. The 2506 study describes it as a mapping to a uniform or normal distribution and does not say which, and its code uses the library default 2506.08274 §3.1.8 scalers.py.

Decimal scaling (2506 only)

Divides by 10 raised to a power j, the smallest integer that makes the largest absolute value below 1 2506.08274 §3.1.9, eq. 7. It has no centre term. The code computes j from a logarithm and the two agree except at exact powers of ten scalers.py. The 2506 text says it is among the lowest-memory scalers, which its own memory table does not show; see Contradictions.

Mean centering, variable stability and Pareto (2506 only)

Mean centering subtracts the mean and does not divide 2506.08274 §3.1.6, eq. 6. Variable stability (VAST) multiplies the z-score by μ/σ 2506.08274 §3.1.4, eq. 4. Pareto scaling divides the centred value by the square root of σ 2506.08274 §3.1.5, eq. 5.

Tanh and logistic (2506 only)

Both start from the z-score and then apply a non-linear map: a tanh with a 0.01 factor 2506.08274 §3.1.10, eq. 8, a logistic sigmoid 2506.08274 §3.1.11, eq. 9, and a second tanh form 2506.08274 §3.1.12, eq. 10. The 2506 card gives per-scaler results only for selected cells, so this page does not rank these variants against the others.

Causal patch standardization (2505 only)

Standardizes each patch of a series with statistics from the past and the current patch only. Padded points get weight 0, and the running mean and variance use Bessel's correction 2505.14766 §3.1 app. A.1. All time steps in a patch share the statistics of the patch's last step 2505.14766 §3.1. The variance gets a small offset before the square root, and the scale is limited to a band around a series-wide deviation 2505.14766 §3.1. The paper's wording of the offset and the band differs from the code; see Contradictions. The authors use it on 4096-point contexts with 64-point patches 2505.14766 §3.1. The paper reports no comparison with the other scalers in this list 2505.14766 app. E.

When to use which

The studies give results per model and, in 2212, per imbalance level, not a single ranking. The table reads the choice from the model family, the data and the cost of the step.

Example from the 2212 table. GP's mean F1 with no scaling is 0.52, with min-max 0.57, and with the quantile transformer 0.32 2212.12343 tab. 3. The same scaler family can help one model and hurt another, which is why the choice has to be tested per model. The 2506 Dry Bean accuracy shows the same pattern: LR rises from 0.7054 unscaled to 0.9251 with its best scaler, SVM from 0.5803 to 0.9273, and MLP from 0.2980 to 0.9334, while RF gains nothing (0.9238) 2506.08274 app. A, accuracy table.

Implementation sketch

Per-fold fitting and the variant choice, as the studies run them. The protocol is in 2212.12343 §4 and 2506.08274 §4.3.

# Fit the scaler on the training part of each fold only  [arxiv:2212.12343 §4.2]
# Input:  train, test (DataFrames of numeric features), scaler
# Output: scaled train, scaled test

def scale_fold(train, test, scaler):
    # Learn centre and spread from training rows only  [arxiv:2212.12343 §4.2]
    scaler.fit(train)
    # Apply the same centre and spread to both parts  [arxiv:2212.12343 eq. 1]
    return scaler.transform(train), scaler.transform(test)

# Compare each scaler against no scaling, with the same model  [arxiv:2212.12343 §4]
# Use the same folds for every variant so differences come from the scaler
for scaler in [None, standard, min_max, max_abs, robust, quantile]:
    for model in models:
        # Scale nothing when the variant is "no scaling"  [arxiv:2212.12343 §4.2]
        ...

Notes for implementers:

• Scale the numeric columns only. The 2212 code scales the numeric columns and leaves one-hot columns alone experiment.ipynb; the paper does not say which columns it scales 2212.12343 §4.2.
• Use the same folds and random seed for every variant, so a difference reflects the scaler 2212.12343 §4.1, code. The 2506 study uses one split with random_state=42 and no stratification train_test_split.py.
• Name the quantile output distribution. 2212 uses a normal target 2212.12343 §2.5; 2506 does not state its choice 2506.08274 §3.1.8.
• The 2608 fraud study does not state whether its standardization was fitted on the training split only 2608.07471 §3.4 fig. 3. Both benchmarks fit on the training part, so do the same; a fit on all rows would leak test data.
• Say whether features were scaled. The 2411 field study does not state it 2411.04700 §3. Its neural network reads the raw statistics dataset.py, and its SVM scales only when a flag is passed, which the README command omits svm.py README.md. Record the choice in your own results, because the paper gives no baseline.
• Missing values are not covered in the 2212 paper. The 2506 code replaces them with 0 after the split, including regression targets, and the paper does not say so train_test_split.py. Treat it as an unexamined choice.
• Min-max and decimal scaling are fitted on training data, so test values can exceed their range 2506.08274 §3.1.9 scalers.py.
• For causal standardization, update the running statistics one point at a time and skip padded points 2505.14766 §3.1 app. A.1. Clip the scale to a band around a series-wide deviation, not around the causal one. The code clips against the global deviation, while the paper compares the band with the causal value 2505.14766 §3.1 scaler.py.

Contradictions

• LDA and QDA. The 2212 text says the scaler choice has no significant effect for LDA, QDA and DT 2212.12343 §5.1. The Friedman table rejects "no difference" over all datasets for LDA (F1 p = 0.0132, G-Mean p = 0.0002) and QDA (F1 p = 0.0052, G-Mean p = 0.0006) 2212.12343 tab. 4. The page follows the table here, as the paper's own text is inconsistent with it.
• Nearest neighbours. The 2212 text lists KNN among the models whose F1 is insensitive to the scaler 2212.12343 §5.1. Table 4 rejects "no difference" for KNN F1 over all datasets (p = 0.0182) and does not reject for G-Mean (p = 0.0732) 2212.12343 tab. 4. The 2506 study finds KNN accuracy sensitive on both tests 2506.08274 app. A. Those are different metrics, datasets and folds, so the two results do not test each other.
• Naive Bayes. 2212 finds GaussianNB sensitive over all datasets for both metrics, with a small spread 2212.12343 tab. 3, tab. 4. 2506 finds no significant effect for its Naive Bayes on accuracy (Wilcoxon p = 0.3379, Friedman p = 0.9294) 2506.08274 app. A. Both use Gaussian Naive Bayes with default settings, so the disagreement is not explained by the model alone.
• Logistic regression and TabNet, text against tables. The 2506 text says LR and TabNet are significantly scaler-sensitive 2506.08274 §5.1. Its accuracy table shows LR significant across scalers (Friedman p = 0.0000) but not against unscaled (Wilcoxon p = 0.3228), and TabNet not significant on either test (Wilcoxon p = 0.0732, Friedman p = 0.1443) 2506.08274 app. A. TabNet is significant for R² 2506.08274 app. A, R² stats table. The page follows the table.
• SVM kernel, paper against code. The 2506 paper says its SVM uses an RBF kernel 2506.08274 §3.2. The classification code uses a linear kernel train_results.py, so the reported classification SVM results describe the linear model.
• Decimal scaling memory. The 2506 text says decimal scaling is among the lowest-memory scalers 2506.08274 §5.3. Its memory table shows it above min-max on Letter Recognition (3628.84 kB against 2567.93 kB) and Wine Quality (832.44 kB against 624.82 kB) 2506.08274 app. A, memory table.
• Causal statistics in training. The paper says that at inference the statistics come from the history only 2505.14766 §3.1. The training path clamps the scale with statistics over the whole input window, so training sees future points backbone.py backbone.py. The paper calls this relaxed causality 2505.14766 §3.1. An implementation that matches training is not causal; one that matches the paper's causal rule does not match training.

Papers

• The choice of scaling technique matters for classification performance: five scalers against no scaling on 82 KEEL datasets and 20 classifiers. Shows that the scaler choice changes results and that tree models are the least affected.
• The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks: 12 scalers against no scaling on 16 UCI datasets and 14 algorithms. Finds that tree ensembles are largely insensitive, while LR, SVMs, TabNet and MLPs are not, with the accuracy tests showing a different picture for LR and TabNet.
• This Time is Different: An Observability Perspective on Time Series Foundation Models: a 151M-parameter forecaster that standardizes each patch with past values only. Its ablation shows the step matters for held-out NLL, but the paper compares no other scalers.
• Application of Artificial Intelligence for Fraudulent Banking Operations Recognition: standardizes Time and Amount on one card-fraud dataset before seven classifiers are compared by AUC. Reports no unscaled comparison, so it adds a worked pipeline rather than evidence on the scaler choice.
• Field Assessment of Force Torque Sensors for Planetary Rover Navigation: SVM and neural network on sensor statistics for terrain classification. Compares no scalers and does not say whether it scaled features.