The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks

In plain words

Feature scaling changes the numbers a model sees before it learns anything. Examples are squeezing every column into a fixed range, or shifting each column so its average is zero. Many people apply one scaler out of habit. This paper asks what that choice actually changes. It tries twelve scaling methods, some of them rarely used, plus no scaling at all. It runs each one with fourteen common machine learning models on sixteen public tables of data, nine for classification and seven for regression.

The main finding is that the models differ a lot. Tree-based ensembles such as random forests and gradient boosting give about the same results whatever scaler is used. Logistic regression, support vector machines, neural networks and TabNet can change a lot with the choice. The authors also timed training and prediction and measured memory. Scaling moved those numbers too, usually by small amounts. The practical lesson is that scaling is not a free default: check it for the model you actually use.

TL;DR

The paper compares 12 feature-scaling methods, plus no scaling, across 14 algorithms and 16 UCI tabular datasets. Each run uses a 70/30 split, the scaler is fitted on the training part only, and the models use the default settings the paper states. The authors report that random forests and gradient boosting are largely unaffected by the scaler, while LR, SVMs, TabNet and MLPs are strongly affected. The official code is MIT-licensed and matches most of the paper's formulas, but it differs from the text in a few places listed under Paper vs code.

Problem

Feature scaling is often applied without a clear reason, and it can be done wrongly, for example by fitting the scaler before the split, which leaks test information into training §4.2 §7. Earlier scaling studies are narrow: they cover one or two datasets, few scalers, or one or two algorithms §2 tab. 1. The paper's stated goal is a broad, reproducible comparison of scalers across models, tasks and cost measures §1.

Method

The study is a benchmark, not a new algorithm. The steps below apply to every combination of dataset, scaler and model.

1. Datasets. Sixteen UCI datasets: nine for classification (binary or multi-class) and seven for regression. All features are numeric (int64 or float64). Categorical targets are label-encoded §4.1 §4.3 etl_cleaning.py. Sizes are in the table below tab. 2 tab. 3.
2. Split. 70% training, 30% test, made before any preprocessing §4.2. The code uses test_size=0.3 and random_state=42, with no stratification train_test_split.py train_test_split.py. The repo ships the exact splits as CSV and JSON [code: dataset_split_classification.csv#L1].
3. Missing values. Replaced with 0 after the split, for features in both tasks and for targets in regression. The paper does not state this train_test_split.py train_test_split.py.
4. Scaling. Statistics come from the training features only, and the same transform is applied to the test features §4.3 alg. 1. The 12 methods are listed below. The paper does not say which quantile output distribution it uses for Quantile Transformation §3.1.8.
5. Models. Twelve classifiers and eleven regressors, 14 algorithms in total §3.2 §4.3. Default hyperparameters are used unless the code sets otherwise (see Hyperparameters) §4.3.
6. Metrics. Accuracy for classification. R², MAE and MSE for regression §3.3.1 §3.3.2. Training time, inference time and peak memory of the scaling step are also recorded §3.3.3 §5.2 §5.3.
7. Statistics. Wilcoxon signed-rank test for scaled versus unscaled, and Friedman test across scalings, both at α = 0.01 §5.1.

Scalers

Let \mu and \sigma be the mean and standard deviation of one feature on the training data. The formulas below are the paper's. Those marked "code" match the official implementation, as checked in Paper vs code.

• Min–Max, to [0, 1] §3.1.1: X_{norm} = \frac{X - X_{min}}{X_{max} - X_{min}} eq. 1
• Max-abs, divide by the largest absolute value §3.1.2: X_{norm} = \frac{X}{\max(|X|)} eq. 2
• Z-score (standardisation) §3.1.3: X_{norm} = \frac{X - \mu}{\sigma} eq. 3
• Variable stability scaling (VAST) §3.1.4: X_{norm} = \frac{X - \mu}{\sigma} \cdot \frac{\mu}{\sigma} eq. 4
• Pareto scaling §3.1.5: X_{norm} = \frac{X - \mu}{\sqrt{\sigma}} eq. 5
• Mean centering §3.1.6: X_{norm} = X - \mu eq. 6
• Robust scaler, median and IQR §3.1.7 (unnumbered in the paper): X_{norm} = (X - X_{median}) / \mathrm{IQR}
• Quantile transformation, to a uniform or normal distribution §3.1.8. No formula is given.
• Decimal scaling, j the smallest integer with \max|X_{norm}| < 1 §3.1.9: X_{norm} = \frac{X}{10^{j}} eq. 7
• Tanh transformation, with z = (X - \mu)/\sigma §3.1.10: X_{norm} = \tfrac{1}{2}\{\tanh(0.01\, z) + 1\} eq. 8
• Logistic sigmoid, with q = (X - \mu)/\sigma §3.1.11: X_{norm} = \frac{1}{1 + e^{-q}} eq. 9
• Hyperbolic tangent (as defined here), same q §3.1.12: X_{norm} = \frac{1 - e^{-q}}{1 + e^{-q}} eq. 10

The 13th setting is no scaling, which the paper calls the baseline §1.

Experiment loop

# Per-dataset experiment loop  [alg. 1]  [§4.3]
# Input:  features X and target y, a list of scalers, a list of models
# Output: one result row per (dataset, scaler, model)

def run_experiment(X, y, scalers, models):
    # Split before any preprocessing, 70/30  [§4.2]
    # Code: test_size=0.3, random_state=42, no stratification
    # [code: train_test_split.py#L24-L26]
    X_train, X_test, y_train, y_test = split(X, y, test_size=0.3, seed=42)

    # Missing values become 0 after the split; not stated in the paper
    # [code: train_test_split.py#L29-L30]
    X_train = fill_missing(X_train, value=0)
    X_test = fill_missing(X_test, value=0)

    for scaler in scalers:
        # Statistics come from the training part only  [§4.3] [alg. 1]
        scaler.fit(X_train)
        X_train_scaled = scaler.transform(X_train)
        X_test_scaled = scaler.transform(X_test)

        for model in models:
            # Only the model fit is timed; the scaler time is not included
            # [code: train_results.py#L104-L106]
            start_time = now()
            model.fit(X_train_scaled, y_train)
            train_seconds = now() - start_time

            # Inference time is the predict call alone  [§3.3.3]
            # [code: train_results.py#L110-L112]
            start_time = now()
            y_pred = model.predict(X_test_scaled)
            inference_seconds = now() - start_time

            # Peak memory is measured around the scaling step only, in kB
            # [code: train_results.py#L70-L75] [§3.3.3]
            memory_kb = scaling_peak_memory_kb  # (not model fit)
            record(scaler, model, y_pred, train_seconds,
                   inference_seconds, memory_kb)

The code wraps scaler.fit_transform and scaler.transform in tracemalloc train_results.py. The paper describes the memory figure as the increase during "dataset loading and the feature scaling step" §3.3.3. See Pitfalls.

Hyperparameters

The paper says all algorithms use default hyperparameters §4.3. The code changes several of them, shown below. Library versions are pinned in requeriments.txt requeriments.txt requeriments.txt requeriments.txt.

Pinned versions: scikit-learn 1.6.1, xgboost 3.0.2, catboost 1.2.8, lightgbm 4.6.0, pytorch-tabnet 4.1.0, torch 2.7.1 requeriments.txt requeriments.txt requeriments.txt requeriments.txt requeriments.txt requeriments.txt.

Results

Author-reported. Setup: 16 UCI datasets (9 classification, 7 regression), 14 algorithms, 12 scalers plus no scaling, a 70/30 split, default hyperparameters, one run per cell, on an AMD Ryzen 9 7900 with 64 GB RAM §4.5. The repo's raw output has 2,405 result rows (1,404 classification, 1,001 regression) results_final.csv.

Datasets, from tab. 2 and tab. 3:

Dry Bean accuracy examples, no scaling against the highest of the twelve scalers app. A, accuracy table:

Accuracy significance tests, scaled versus unscaled (Wilcoxon) and across all scalers (Friedman), α = 0.01 app. A, accuracy stats table:

The main-text summary of these tests is tab. 4. For regression, the R² tests show MLP (Wilcoxon p = 0.0083) and SVR (p = 0.0000) as significant, and KNN as not app. A, R² stats table. The authors describe KNN as not significant for regression §5.1.

Memory for three datasets, in kB, peak during the scaling step [app. A, memory table]:

Timing: the authors report that CART timings are unaffected by scaling and that KNN, SVM and SVR are the most sensitive, with overhead that is small for most models §5.2 app. A, time tables. Training-time effects follow the accuracy pattern, and tree ensembles gain little speed from scaling §5.2.

Paper vs code

The code is at commit 7b7bb3a9cf2c25e3e2e30e9055242fc1a1f4273a, linked from §4.5 §4.5. Its LICENSE file is MIT.

• SVM kernel. The paper says SVM uses an RBF kernel §3.2. The classification SVM is SVC(max_iter=1000, kernel='linear') [code: train_results.py#L33]. The SVR is kernel='rbf' train_results.py.
• Hyperparameters. The paper says default settings §4.3. The code sets max_iter=1000 for SVC and MLP, random_state for most models, and seed=42 for TabNet train_results.py.
• Seeds in the training loop. XGBoost and LightGBM are rebuilt inside the loop without their seeds train_results.py [code: train_results.py#L169]. The results from that run are not seeded for these models, even though the dictionary defines seeds.
• Memory window. The paper says memory is the increase "during dataset loading and the feature scaling step" §3.3.3. The code measures only the scaler's fit_transform and transform train_results.py train_results.py. For "None" it measures a to_numpy() copy train_results.py.
• Missing values. Not in the paper. The code fills NaN with 0 after the split train_test_split.py train_test_split.py.
• Decimal scaling. The code uses j = \lceil \log_{10}(\max\lvert x\rvert + 10^{-17}) \rceil scalers.py. The paper states the rule as "smallest integer with max < 1" §3.1.9. The two agree except at exact powers of ten, where the code gives a maximum of 1.
• Statistics pairing. The notebook compares each scaler with the unscaled rows for each dataset, merged on dataset name [code: analysis-results.ipynb#L4992-L5004]. The Friedman test uses a dataset-by-scaler table that includes "None" analysis-results.ipynb. The paper does not state the pairing. See Pitfalls.
• Formulas. Tanh scalers.py, logistic [code: scalers.py#L99-L103], hyperbolic tangent scalers.py, VAST scalers.py, Pareto scalers.py, mean centering scalers.py and Min–Max, max-abs and robust match the paper's equations or scikit-learn's defaults. Only the Decimal rule differs (above).
• Classification objective. XGBClassifier(objective="binary:logistic") is used on every classification dataset, including multi-class ones such as Dry Bean (7 classes) and Letter Recognition (26) train_results.py tab. 2. This card does not verify how the library handles that objective on multi-class data.

Pitfalls

• Scaler fit scope. Min–Max is defined with "the minimum and maximum values in the dataset" eq. 1. The code fits on the training split only train_results.py. Implement the training-only fit, as the code does and as the leakage argument requires §4.2.
• Missing values are zeroed. The paper does not say how missing values are handled. The code replaces them with 0 after the split, including regression targets train_test_split.py. The paper does not measure how this choice interacts with each scaler, so treat it as an unexamined choice.
• Test values outside the scaled range. Decimal scaling is fitted on the training data, so test values can exceed (−1, 1) eq. 7 [code: scalers.py#L57-L73]. The same holds for Min–Max.
• Memory numbers measure the scaler only. Peak memory is taken around the scaling step train_results.py, not during model fitting or dataset loading §3.3.3. The "None" column is nonzero (for example 1704.29 kB for Dry Bean) because the code copies the data with to_numpy() [app. A, memory table] train_results.py. The paper's statement that no scaling adds zero memory §5.3 does not hold for the larger datasets.
• Memory ranking in the text. The paper says Decimal Scaling is among the lowest-memory scalers §5.3. The table does not show that: for Letter Recognition DS is 3628.84 kB against 2567.93 kB for MM, and for Wine Quality it is 832.44 kB against 624.82 kB app. A, memory table.
• Training time excludes scaling. train_seconds covers only fit [code: train_results.py#L104-L106]. Scaler cost is not in the training-time figures. Inference time covers only predict train_results.py.
• Single run per cell. The code runs each (dataset, scaler, model) once train_results.py. The timings and the unseeded models have no repeat or variance estimate, so small differences are noise.
• Wilcoxon pairs are not independent. The notebook pairs each scaler's rows with the unscaled row of the same dataset [code: analysis-results.ipynb#L4992-L5004]. The same unscaled value is reused for every scaler. The reported statistics (for example 1927.5 for LR accuracy app. A, accuracy stats table) are larger than 9 paired observations could produce, so they are consistent with about 9 × 12 pairs. The independence assumption behind the p-values is therefore weak.
• Paper text against its own tables. §5.1 says LR and TabNet are significantly scaler-sensitive, and the abstract says the same for TabNet §5.1 [abstract]. The accuracy tests do not support that: LR (Wilcoxon p = 0.3228) and TabNet (p = 0.0732) are not significant [app. A, accuracy stats table]. TabNet is significant for R² (Wilcoxon p = 0.0000) [app. A, R² stats table]. The summary table omits LR and TabNet tab. 4.
• Hyperparameters are not tuned. Default settings are a stated limitation §6. Conclusions about scaler sensitivity may change with tuned models.
• No cross-validation. Only one hold-out split is used §6, so the significance tests rest on one split per dataset.
• Class balance is not controlled. There is no stratification [code: train_test_split.py#L24-L26]. Some datasets have many classes (Letter Recognition has 26 tab. 2), so a class can be rare in the test split.
• Quantile output is unstated. The paper says "uniform or normal" §3.1.8. The code uses the scikit-learn default scalers.py, so replicate that choice explicitly.

Lineage

• Builds on: not stated. The paper builds on standard scaling practice, cited in §3.1, and on the scikit-learn and pytorch-tabnet libraries §3.1 [requeriments.txt#L85].
• Compared against: nine earlier scaling studies summarised in tab. 1. The paper states each one's number of scalers, algorithms and datasets, and none has a card here.
• Superseded by: not stated.
• Version: arXiv v5, updated 2025-11-21 (raw/arxiv/2506.08274.json). The LaTeX source is an accepted IEEE Access version (DOI 10.1109/ACCESS.2025.3635541) (raw/src/2506.08274/access.tex title block).