--- kind: concept slug: feature-scaling title: Feature scaling summary: >- Rescaling each input column before training. Two benchmarks compared 5 and 12 scalers with no scaling, on 82 binary and 16 UCI datasets. The choice changed results for most models that react, and a poor one can score below no scaling. Tree ensembles barely react; SVMs, perceptrons and MLPs react strongly. A time-series model rescales each patch with past values only. All are author-reported. tags: [preprocessing, classification, imbalanced-data, tabular-data, time-series] papers: ["2212.12343", "2506.08274", "2608.07471", "2505.14766", "2411.04700"] related: [random-forest, support-vector-machine, multi-layer-perceptron, ensemble-learning, time-series-forecasting] status: draft updated_at: 2026-10-25 --- ## Overview Feature scaling changes each numeric input column before training and testing, so its values fall in a new range or spread. Most scalers are linear: they subtract a centre value and divide by a spread, with both chosen per column [arxiv:2212.12343 §2, eq. 1]. The others divide by a power of ten, or apply a non-linear map such as the quantile transformation, tanh or the logistic function [arxiv:2506.08274 §3.1.8–§3.1.11]. In both studies the scaler is fitted on the training part of a split and then applied to the test part, so the test data does not shape the scaling [arxiv:2212.12343 §4.2] [arxiv:2506.08274 §4.3]. Time-series models cannot use the statistics of a whole series, because the future would leak into the scale. The forecaster in [arxiv:2505.14766 §3.1] therefore standardizes each patch with the mean and deviation of the past and current values only. Three studies measure the effect of scaling. All three are author-reported and use different scalers, models and datasets, so their numbers are not directly comparable. A fourth study applies one scaler inside a fraud-detection pipeline but does not compare it with no scaling, so it says nothing about the size of the effect [arxiv:2608.07471 §3.4]: - [The choice of scaling technique matters for classification performance](/papers/2212.12343) compares five scalers and no scaling on 82 binary KEEL datasets with 20 classifiers, using 5-fold cross-validation [arxiv:2212.12343 §4.1]. - [The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks](/papers/2506.08274) compares 12 scalers and no scaling on 16 UCI datasets (9 classification, 7 regression) with 14 algorithms, on one 70/30 split [arxiv:2506.08274 §4.1–§4.3]. - [This Time is Different: An Observability Perspective on Time Series Foundation Models](/papers/2505.14766) removes its causal patch scaling in one ablation. Held-out observability NLL rises by 27.3% without it, the largest of the four components tested [arxiv:2505.14766 app. E]. The ablation removes the step from a model that was trained with it, so it measures the step, not a choice between scalers. - [Application of Artificial Intelligence for Fraudulent Banking Operations Recognition](/papers/2608.07471) standardizes two columns (Time and Amount) of one card-fraud dataset before seven classifiers are compared by AUC [arxiv:2608.07471 §3.4] [§4.2]. A fifth card, [arxiv:2411.04700 §3], trains an SVM and a neural network on sensor statistics. It does not compare scalers, and the paper does not say whether its features were scaled [arxiv:2411.04700 §3]. It is listed here only so that readers see where scaling is unreported; it adds no evidence on the effect. Every claim below comes from one of these five cards and is labelled with its source. ## Variants The first five entries below are the scalers in the 2212 study. The entries marked "2506 only" are used only in the 2506 study, and the entry marked "2505 only" is used only in the time-series model. Each entry gives the rule and the source. ### Standard scaler (z-score) Subtracts the mean and divides by the standard deviation [arxiv:2212.12343 §2.1, eq. 3] and [arxiv:2506.08274 §3.1.3, eq. 3]. The 2212 paper does not say whether that deviation is the sample or population form [arxiv:2212.12343 §2.1]. The 2506 code uses the population form (`np.std`, ddof = 0) [code: scalers.py#L29]. Among the five 2212 scalers, this one has the highest mean F1 for Bagging, OLA and the Perceptron, and the highest mean G-Mean for SVM_RBF and the Perceptron [arxiv:2212.12343 tab. 3]. Its wins are concentrated in low-imbalance datasets [arxiv:2212.12343 tab. 5]. The 2608 fraud study standardizes only Time and Amount and describes the step in words as centring on the mean and dividing by the standard deviation [arxiv:2608.07471 §3.4]. It does not say whether the mean and deviation came from the training split [arxiv:2608.07471 §3.4] [fig. 3]. It reports no unscaled run, so its results cannot show whether the step helped [arxiv:2608.07471 §4.2]. ### Min-max scaler Maps the column to [0, 1] using its minimum and maximum [arxiv:2212.12343 §2.2, eq. 6] and [arxiv:2506.08274 §3.1.1, eq. 1]. The general form for any target range is given in [arxiv:2212.12343 eq. 7]. For GP it has the highest mean F1 and G-Mean of the five 2212 scalers [arxiv:2212.12343 tab. 3]. The minimum and maximum are set by single values, so the paper's results do not show how it behaves with outliers; the study does not test that. Test values outside the training range can fall outside [0, 1] [arxiv:2506.08274 §3.1.1]. ### Maximum absolute scaler Divides each value by the largest absolute value in the column, so values land in [-1, 1] and zeros stay at zero [arxiv:2212.12343 §2.3, eq. 8]. It does not shift the centre. For SVM_lin it has the lowest mean F1 and G-Mean of the five 2212 scalers, and both are below no scaling [arxiv:2212.12343 tab. 3]. ### Robust scaler Centres on the median and divides by the interquartile range [arxiv:2212.12343 §2.4, eq. 9]. The paper gives no quantile range or interpolation rule, so the library defaults apply [arxiv:2212.12343 Pitfalls]. The robust scaler has the highest mean G-Mean for QDA and GNB among the 2212 scalers [arxiv:2212.12343 tab. 3]. The study does not test datasets with extreme values [arxiv:2212.12343 §2.4]. ### Quantile transformer Maps each column through its empirical distribution function to a uniform value, then through the inverse normal function, which gives roughly Gaussian output [arxiv:2212.12343 §2.5]. Values outside the range seen in training are clipped to the bounds [arxiv:2212.12343 §2.5]. The number of quantiles is not stated [arxiv:2212.12343 §2.5]. Across all datasets this transformer has the most wins, and it also has the most wins in the medium and high imbalance strata [arxiv:2212.12343 tab. 5]. It is not the best choice for every model: for GP, its F1 is 0.32, against 0.52 with no scaling [arxiv:2212.12343 tab. 3]. The 2506 study describes it as a mapping to a uniform or normal distribution and does not say which, and its code uses the library default [arxiv:2506.08274 §3.1.8] [code: scalers.py#L129]. ### Decimal scaling (2506 only) Divides by 10 raised to a power j, the smallest integer that makes the largest absolute value below 1 [arxiv:2506.08274 §3.1.9, eq. 7]. It has no centre term. The code computes j from a logarithm and the two agree except at exact powers of ten [code: scalers.py#L64]. The 2506 text says it is among the lowest-memory scalers, which its own memory table does not show; see Contradictions. ### Mean centering, variable stability and Pareto (2506 only) Mean centering subtracts the mean and does not divide [arxiv:2506.08274 §3.1.6, eq. 6]. Variable stability (VAST) multiplies the z-score by μ/σ [arxiv:2506.08274 §3.1.4, eq. 4]. Pareto scaling divides the centred value by the square root of σ [arxiv:2506.08274 §3.1.5, eq. 5]. ### Tanh and logistic (2506 only) Both start from the z-score and then apply a non-linear map: a tanh with a 0.01 factor [arxiv:2506.08274 §3.1.10, eq. 8], a logistic sigmoid [arxiv:2506.08274 §3.1.11, eq. 9], and a second tanh form [arxiv:2506.08274 §3.1.12, eq. 10]. The 2506 card gives per-scaler results only for selected cells, so this page does not rank these variants against the others. ### Causal patch standardization (2505 only) Standardizes each patch of a series with statistics from the past and the current patch only. Padded points get weight 0, and the running mean and variance use Bessel's correction [arxiv:2505.14766 §3.1] [app. A.1]. All time steps in a patch share the statistics of the patch's last step [arxiv:2505.14766 §3.1]. The variance gets a small offset before the square root, and the scale is limited to a band around a series-wide deviation [arxiv:2505.14766 §3.1]. The paper's wording of the offset and the band differs from the code; see Contradictions. The authors use it on 4096-point contexts with 64-point patches [arxiv:2505.14766 §3.1]. The paper reports no comparison with the other scalers in this list [arxiv:2505.14766 app. E]. ## When to use which The studies give results per model and, in 2212, per imbalance level, not a single ranking. The table reads the choice from the model family, the data and the cost of the step. | Situation | Use | Why | |---|---|---| | Model is a decision tree or a tree ensemble (DT, RF, XGBoost, AdaBoost, CatBoost, LightGBM) | Any scaler, or none; the choice barely matters | 2212: DT, RF, XGBoost and AdaBoost are nearly insensitive, with DT and AdaBoost rejecting only on high-imbalance subsets [arxiv:2212.12343 §5.1, tab. 4]. 2506: the accuracy Friedman tests do not reject for RF, LightGBM, CatBoost, XGBoost, AdaBoost or CART [arxiv:2506.08274 app. A, accuracy stats table] | | Model is an SVM, perceptron, MLP or Gaussian process | Scale, and test the scaler choice | 2212: the mean range between best and worst variant is close to or above 0.15 for SVM_lin, SVM_RBF, GP, Perceptron and MLP [arxiv:2212.12343 §5.1, fig. 3]. 2506: the accuracy tests reject for SVM and MLP, both scaled-versus-unscaled and across scalers [arxiv:2506.08274 app. A] | | Logistic regression | Compare scalers with each other, not only scaled against unscaled | 2506: across scalers the Friedman test rejects (p = 0.0000), but scaled against unscaled the Wilcoxon test does not (p = 0.3228) [arxiv:2506.08274 app. A, accuracy stats table] | | Naive Bayes (GaussianNB) | Disputed; test on your own data | 2212 rejects "no difference" over all datasets for F1 and G-Mean (both p = 0.0000), though G-Mean spans only 0.57 to 0.62 [arxiv:2212.12343 tab. 3, tab. 4]. 2506 finds no significant effect (Friedman p = 0.9294) [arxiv:2506.08274 app. A] | | K-nearest neighbours | Scale, and test the scaler choice | 2506: accuracy is significant against unscaled (Wilcoxon p = 0.0000) and across scalers (Friedman p = 0.0040) [arxiv:2506.08274 app. A]. 2212 rejects for F1 only (p = 0.0182) [arxiv:2212.12343 tab. 4] | | Ensemble built from one base model (Bagging, LCA, KNORA-U and similar, on Perceptrons) | Pick the scaler by the base model | For the Perceptron and its six ensembles, the best scaler (standard) and the worst variant (no scaling) coincide across models [arxiv:2212.12343 §5.2, fig. 4] | | No data about the class balance, general purpose | Quantile transformer, if the 2212 result carries over | It has the most wins overall, in medium and high imbalance strata [arxiv:2212.12343 tab. 5]. In 2506's Dry Bean table, the quantile transformer is the top scaler for SVM only; the top scaler for LR and MLP is another one [arxiv:2506.08274 app. A, accuracy table] | | Low imbalance (IR ≤ 3) | Standard scaler | It has the most wins in the low-IR stratum; the quantile transformer is close [arxiv:2212.12343 tab. 5] | | Some columns are already transformed (for example PCA components) | Scale only the open columns, and say which ones | The fraud study standardized only Time and Amount and left the encrypted PCA columns alone [arxiv:2608.07471 §3.4]. It does not compare this with scaling every column, so it does not show that the split is better | | Quantile transformer on small training folds | Set the quantile count deliberately | The count is not stated in the paper, and the library warns below 1000 samples [arxiv:2212.12343 Pitfalls] | | Distribution has extreme outliers | Test the robust scaler against the others | The 2212 study does not test outlier-heavy data [arxiv:2212.12343 §2.4]; the 2506 card does not discuss outliers | | Heavy class imbalance and a model that reacts | Quantile transformer, with a Friedman test on your data | QT has the most high-IR wins [arxiv:2212.12343 tab. 5]; per-dataset ranges can be very large for some models [arxiv:2212.12343 §5.1] | | Tight latency or memory budget on large tables | Measure the scaling step on your data first | 2506: scaling overhead is small for most models, CART is unaffected, and KNN, SVM and SVR are the most sensitive [arxiv:2506.08274 §5.2]. Peak memory during scaling was higher than with no scaling on Dry Bean (1704 kB without, 2388 to 2599 kB with the scalers shown) [arxiv:2506.08274 app. A, memory table] | | Time-series forecasting with a causal model (each step may use only the past) | Causal patch standardization, with statistics from the past only | Scaling with future values leaks them into the model's input. The paper's first-patch-only alternative is causal but, the authors argue, fails when the statistics drift [arxiv:2505.14766 §3.1]. The ablation shows the step matters inside this model [arxiv:2505.14766 app. E] | | Just any scaler, with no scaling as the default | Do not skip the scaling choice | For many models no scaling ranked above some scalers, so a poor scaler can be worse than none [arxiv:2212.12343 §5.1, tab. 3] | Example from the 2212 table. GP's mean F1 with no scaling is 0.52, with min-max 0.57, and with the quantile transformer 0.32 [arxiv:2212.12343 tab. 3]. The same scaler family can help one model and hurt another, which is why the choice has to be tested per model. The 2506 Dry Bean accuracy shows the same pattern: LR rises from 0.7054 unscaled to 0.9251 with its best scaler, SVM from 0.5803 to 0.9273, and MLP from 0.2980 to 0.9334, while RF gains nothing (0.9238) [arxiv:2506.08274 app. A, accuracy table]. ## Implementation sketch Per-fold fitting and the variant choice, as the studies run them. The protocol is in [arxiv:2212.12343 §4] and [arxiv:2506.08274 §4.3]. ```pseudocode # Fit the scaler on the training part of each fold only [arxiv:2212.12343 §4.2] # Input: train, test (DataFrames of numeric features), scaler # Output: scaled train, scaled test def scale_fold(train, test, scaler): # Learn centre and spread from training rows only [arxiv:2212.12343 §4.2] scaler.fit(train) # Apply the same centre and spread to both parts [arxiv:2212.12343 eq. 1] return scaler.transform(train), scaler.transform(test) # Compare each scaler against no scaling, with the same model [arxiv:2212.12343 §4] # Use the same folds for every variant so differences come from the scaler for scaler in [None, standard, min_max, max_abs, robust, quantile]: for model in models: # Scale nothing when the variant is "no scaling" [arxiv:2212.12343 §4.2] ... ``` Notes for implementers: - Scale the numeric columns only. The 2212 code scales the numeric columns and leaves one-hot columns alone [code: Code/experiment.ipynb#L714-L723]; the paper does not say which columns it scales [arxiv:2212.12343 §4.2]. - Use the same folds and random seed for every variant, so a difference reflects the scaler [arxiv:2212.12343 §4.1, code]. The 2506 study uses one split with `random_state=42` and no stratification [code: train_test_split.py#L24-L26]. - Name the quantile output distribution. 2212 uses a normal target [arxiv:2212.12343 §2.5]; 2506 does not state its choice [arxiv:2506.08274 §3.1.8]. - The 2608 fraud study does not state whether its standardization was fitted on the training split only [arxiv:2608.07471 §3.4] [fig. 3]. Both benchmarks fit on the training part, so do the same; a fit on all rows would leak test data. - Say whether features were scaled. The 2411 field study does not state it [arxiv:2411.04700 §3]. Its neural network reads the raw statistics [code: ml/dataset.py#L96], and its SVM scales only when a flag is passed, which the README command omits [code: ml/svm.py#L93-L97] [code: README.md#L91]. Record the choice in your own results, because the paper gives no baseline. - Missing values are not covered in the 2212 paper. The 2506 code replaces them with 0 after the split, including regression targets, and the paper does not say so [code: train_test_split.py#L29-L30]. Treat it as an unexamined choice. - Min-max and decimal scaling are fitted on training data, so test values can exceed their range [arxiv:2506.08274 §3.1.9] [code: scalers.py#L57-L73]. - For causal standardization, update the running statistics one point at a time and skip padded points [arxiv:2505.14766 §3.1] [app. A.1]. Clip the scale to a band around a series-wide deviation, not around the causal one. The code clips against the global deviation, while the paper compares the band with the causal value [arxiv:2505.14766 §3.1] [code: toto/model/scaler.py#L243-L246]. ## Contradictions - **LDA and QDA.** The 2212 text says the scaler choice has no significant effect for LDA, QDA and DT [arxiv:2212.12343 §5.1]. The Friedman table rejects "no difference" over all datasets for LDA (F1 p = 0.0132, G-Mean p = 0.0002) and QDA (F1 p = 0.0052, G-Mean p = 0.0006) [arxiv:2212.12343 tab. 4]. The page follows the table here, as the paper's own text is inconsistent with it. - **Nearest neighbours.** The 2212 text lists KNN among the models whose F1 is insensitive to the scaler [arxiv:2212.12343 §5.1]. Table 4 rejects "no difference" for KNN F1 over all datasets (p = 0.0182) and does not reject for G-Mean (p = 0.0732) [arxiv:2212.12343 tab. 4]. The 2506 study finds KNN accuracy sensitive on both tests [arxiv:2506.08274 app. A]. Those are different metrics, datasets and folds, so the two results do not test each other. - **Naive Bayes.** 2212 finds GaussianNB sensitive over all datasets for both metrics, with a small spread [arxiv:2212.12343 tab. 3, tab. 4]. 2506 finds no significant effect for its Naive Bayes on accuracy (Wilcoxon p = 0.3379, Friedman p = 0.9294) [arxiv:2506.08274 app. A]. Both use Gaussian Naive Bayes with default settings, so the disagreement is not explained by the model alone. - **Logistic regression and TabNet, text against tables.** The 2506 text says LR and TabNet are significantly scaler-sensitive [arxiv:2506.08274 §5.1]. Its accuracy table shows LR significant across scalers (Friedman p = 0.0000) but not against unscaled (Wilcoxon p = 0.3228), and TabNet not significant on either test (Wilcoxon p = 0.0732, Friedman p = 0.1443) [arxiv:2506.08274 app. A]. TabNet is significant for R² [arxiv:2506.08274 app. A, R² stats table]. The page follows the table. - **SVM kernel, paper against code.** The 2506 paper says its SVM uses an RBF kernel [arxiv:2506.08274 §3.2]. The classification code uses a linear kernel [code: train_results.py#L33], so the reported classification SVM results describe the linear model. - **Decimal scaling memory.** The 2506 text says decimal scaling is among the lowest-memory scalers [arxiv:2506.08274 §5.3]. Its memory table shows it above min-max on Letter Recognition (3628.84 kB against 2567.93 kB) and Wine Quality (832.44 kB against 624.82 kB) [arxiv:2506.08274 app. A, memory table]. - **Causal statistics in training.** The paper says that at inference the statistics come from the history only [arxiv:2505.14766 §3.1]. The training path clamps the scale with statistics over the whole input window, so training sees future points [code: toto/model/backbone.py#L164] [code: toto/model/backbone.py#L176-L181]. The paper calls this relaxed causality [arxiv:2505.14766 §3.1]. An implementation that matches training is not causal; one that matches the paper's causal rule does not match training. ## Papers - [The choice of scaling technique matters for classification performance](/papers/2212.12343): five scalers against no scaling on 82 KEEL datasets and 20 classifiers. Shows that the scaler choice changes results and that tree models are the least affected. - [The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks](/papers/2506.08274): 12 scalers against no scaling on 16 UCI datasets and 14 algorithms. Finds that tree ensembles are largely insensitive, while LR, SVMs, TabNet and MLPs are not, with the accuracy tests showing a different picture for LR and TabNet. - [This Time is Different: An Observability Perspective on Time Series Foundation Models](/papers/2505.14766): a 151M-parameter forecaster that standardizes each patch with past values only. Its ablation shows the step matters for held-out NLL, but the paper compares no other scalers. - [Application of Artificial Intelligence for Fraudulent Banking Operations Recognition](/papers/2608.07471): standardizes Time and Amount on one card-fraud dataset before seven classifiers are compared by AUC. Reports no unscaled comparison, so it adds a worked pipeline rather than evidence on the scaler choice. - [Field Assessment of Force Torque Sensors for Planetary Rover Navigation](/papers/2411.04700): SVM and neural network on sensor statistics for terrain classification. Compares no scalers and does not say whether it scaled features.