Automated data processing and feature engineering for deep learning and big data applications: a survey

In plain words

Training a deep learning model needs data that is clean, complete, labelled, varied enough and described by useful inputs. Most of that work is still done by hand. This survey looks at the ways that work is being automated: tools that fill in missing values, fix bad records or suggest labels; methods that create new training examples by transforming existing ones or by generating them with modern generative models; and methods that build or pick the input features a model learns from.

The authors group these steps into three families (preprocessing, augmentation and feature engineering) and then show how they can be run together with AutoML, which searches over pipeline choices automatically. The survey does not propose a new method. It gives an overview of the approaches, tables that compare reported results, and a discussion of what is still hard, such as comparing methods fairly, getting results that carry over to new tasks, and the cost of search.

TL;DR

A survey, not a method: it maps automated preprocessing, augmentation and feature engineering for deep learning and big data pipelines, and their AutoML integration §1.3. It reports author-compiled comparison tables for augmentation and feature engineering. Its discussion notes that feature engineering is highly dataset-sensitive §5.5 and that fair comparison across studies is hard §3.6 §9.1.8. Use it as a map of the literature, not as a source of settings to implement.

Problem

Deep learning pipelines usually need manual collection, preprocessing and augmentation before training works well [abstract] §1.1. The authors say that data processing has been less studied than model design: few surveys cover preprocessing and feature engineering in deep learning, while many cover augmentation or general AutoML §1.2. The problem the survey addresses is the cost and fragility of hand-built data pipelines, which the authors say matters most in big data settings with heterogeneous, noisy data §1.1 §2.3.

Without automation, feature extractors are often fixed per task and must be redesigned for new conditions §2.1. In deep learning, feature extraction is learned during training, but the authors note that data preprocessing still matters a lot for performance and that augmentation and feature engineering are still needed when data is scarce or unrepresentative §2.3.

Method

This is a survey, so the "method" is its taxonomy and the status of each family.

Scope and definitions. The survey defines three data processing tasks: preprocessing (initial preparation that improves quality and fits data to the target model), augmentation (post-processing that increases quantity and diversity), and feature engineering (low-level processing that creates and selects features) fig. 3 §1.3. It treats feature extraction, feature synthesis (also called feature construction) and feature selection as separate steps, while noting that the boundaries blur and some authors merge them §5.2.

Preprocessing §3. The problem is framed as choosing a set of basic operations P_i on a dataset D so that predictive performance on a target task is maximised §3. The survey lists the approaches by degree of automation tab. 2:

1. Basic: manual, or with generic tools tab. 2.
2. Analytical: explicit formulas inside a deep learning pipeline tab. 2.
3. Deep learned: a learned module inside the network tab. 2.
4. AutoML: end-to-end processing, model selection and hyperparameter tuning tab. 2.

Subtasks covered: imputation (§3.1, generative models such as GAIN and denoising autoencoders, and AutoML tools such as HyperImpute fig. 9); categorical encoding (§3.2, where it notes that full automation is hard because no primitive operation is defined); cleaning (§3.3, learning-based methods such as Learn2Clean and BoostClean); labeling and relabeling (§3.4, including automatic labeling by label propagation and text-based annotation fig. 10); and end-to-end preprocessing tools (§3.5). tab. 1 summarises the surveyed preprocessing works by function, approach and datasets.

Augmentation §4. Automated augmentation is framed as a search over transformation operations and their magnitudes, then applying them to training data §4.1 fig. 13. The survey describes three sources of operations:

1. Analytical: fixed, hand-defined transforms such as rotation and cropping §4.1 fig. 12.
2. Learned: transforms with trainable parameters, such as spatial transformer networks, or a learned augmentation generator §4.1 fig. 14A.
3. Generative: a GAN or other generative model that makes transformed samples, learned jointly with the classifier in a bi-level scheme §4.1 fig. 14B.

Search strategies for magnitudes and policies are reinforcement learning, Bayesian optimisation, evolutionary search, gradient approximations for discrete choices, and reduced grid-like search spaces §4.1.2. Section 4.2 covers data synthesis with LLMs and diffusion models, and Section 4.3 covers dataset distillation as an alternative that keeps a smaller training set. The authors describe the data generation methods in words and show examples fig. 15 fig. 16 fig. 17, without giving training settings for them.

Feature engineering §5. The general formulation is a search over a set of transformations T applied to the original features F, producing extra features Phi, and choosing the best subset of F + Phi for the target task §5.1. Two design families are described:

1. Expansion-reduction: apply many primitive transformations at once to create a large pool, then select a small subset. Its main cost is memory §5.1 §5.3 fig. 20.
2. Iterative: grow the set in small batches, test and keep only useful candidates at each step. Its cost is a large number of evaluations §5.1 fig. 20.

Methods named in Section 5.3 for end-to-end feature engineering include SAFE (gradient-boosting-based information gain search) fig. 21, EAAFE and RTHS (evolutionary) and reinforcement-learning approaches. Section 5.4 covers neural architecture search for features, and Section 5.2 describes the extraction, synthesis and selection subtasks. Differentiable feature engineering (DIFER) maps discrete features into a continuous space and uses gradient ascent fig. 22 §5.4.

End-to-end pipelines §6. The survey treats the whole chain from data acquisition to model deployment as one search over pipeline structure and hyperparameters, with variable-length pipelines searched by evolutionary methods §6 fig. 23.

AutoML tools §7. The survey lists common functions (data cleaning, imputation, label checks, feature generation, model search, visualisation) and gives a table of open-source and proprietary tools by data type and task §7.1 §7.2 tab. 7.

# Expansion-reduction feature engineering  [§5.1] [§5.3] [fig. 20]
# Input:  features (N, F), target y, primitive transformations
# Output: selected feature subset

def expand_and_reduce(features, y, transformations):
    # 1. Expand: apply every primitive transformation to the features  [§5.3]
    new_columns = []
    for transform in transformations:
        new_columns.append(transform(features))
    pool = concatenate(features, new_columns)      # (N, F + F_new)

    # 2. Reduce: search for a small subset that scores well on the target  [§5.1]
    subset = search_subset(pool, y)  # RL, evolutionary or greedy search
    return subset

# Automated augmentation during training  [§4.1] [fig. 13]
# Input:  training batches, augmentation operations with magnitudes
# Output: trained model

def train_with_learned_augmentation(batches, operations, model):
    # 1. Draw a batch from the training set  [fig. 13]
    for batch in batches:
        # 2. Choose an operation and its magnitude from the policy  [§4.1.2]
        op, magnitude = sampler(operations)
        augmented = op(batch, magnitude)
        # 3. Train on the original batch plus the augmented samples  [fig. 13]
        loss = model_loss(model, concatenate(batch, augmented))
        # 4. Update the model, and the sampler from the training loss  [fig. 13]
        update(model, loss)
        update_policy(sampler, loss)
    return model

Hyperparameters

The survey gives no hyperparameters of its own. The settings below belong to the experiments it summarises.

Results

Author-reported. The survey compiles numbers from the original works, so setups vary by study (see Pitfalls).

Augmentation, image classification tab. 3 fig. 18. Setup: WRN-28-10 on CIFAR-10, CIFAR-100 and ImageNet, about 300 epochs §4.4. Each column compares the baseline with the best single method in the automated and traditional groups from the table:

Figure 18 averages ten methods per group fig. 18. Read from its bars: automated group 97.52 / 84.1 / 78.54 / 93.94 and traditional group 96.72 / 83.35 / 77.76 / 93.76 for CIFAR-10 / CIFAR-100 / ImageNet top-1 / top-5. The survey concludes that automated augmentation outperforms the traditional methods on these benchmarks §4.4.

Feature engineering, tabular data tab. 5 tab. 6 §5.5. Table 5 compares Raw, ME, BF, Ran, NFS, LFE, AutoFeat and DIFER on 23 UCI and OpenML datasets, with Random Forest and ten-fold cross-validation. The survey finds the automated methods clearly ahead of basic methods, but not consistently: spambase, autos and convex show little or no gain §5.5. Table 6 compares DCN-V2, FCTree, FETCH, AutoCross, SAFE, OpenFE, AutoFeat and FeatureEngineering against the raw baseline on 10 datasets with LightGBM. The authors say that the automated methods' performance drops sharply on some datasets and sometimes falls below the baseline §5.5.

Paper vs code

No official code. The paper names no code repository or project page, the arXiv record carries no code link, and a GitHub repository search (title key terms plus the author name) returned no results. As a survey, the paper has no method implementation to compare against.

Pitfalls

• Survey, not method. Nothing here can be implemented as-is. Use the taxonomy to find original papers, then compile those cards.
• Benchmark numbers come from different setups. The authors say that preprocessing has no standard algorithms or metrics, so results are hard to compare across studies §3.6 §9.1.8. Treat tab. 1, tab. 3 and tab. 5 as orientation, not a ranking.
• Baseline inconsistency in the figures. Table 3 gives a baseline ImageNet top-5 of 92.1 tab. 3. The top-5 improvement shown for the automated group in fig. 18 (0.90%) matches a baseline of 93.1 rather than 92.1. Check the source before quoting an ImageNet top-5 gain.
• Gains are dataset-dependent. For feature engineering, automated methods can drop below the raw baseline on some datasets §5.5 tab. 6.
• Search cost. Expansion-based methods create large feature pools with high memory use, and iterative methods need many evaluations. Neither cost is quantified across methods in the survey §5.1.
• Feature terms overlap. Extraction, synthesis (construction) and selection are used inconsistently in the literature. The survey fixes its own definitions §5.2; other sources may use different ones.
• Generative augmentation can change meaning. Label-changing or overly strong transforms can add wrong training signal and cause failure §4.4. Generative models can produce unrealistic or harmful data, including hallucinated content §9.1.4 §9.1.5.
• Transfer is not studied. The authors say that automatically generated data often does not transfer to new tasks or domains §9.1.5.
• Figures are not reproduced here. Several examples are described from fig. 7, fig. 13 and fig. 20 in words only, because paper figure support is not available.

Lineage

• Builds on: the authors name He et al. and Zöller and Huber as the few surveys that dedicate part of their scope to automated data processing §1.2. Full reference-level lineage is not in this wiki.
• Compared against: the paper compares automated and traditional methods within its own tables only tab. 3 tab. 5 tab. 6.
• Superseded by: not known from content in this wiki.