Can Foundation Models Wrangle Your Data?

In plain words

Large language models are trained only to predict the next word, yet they can do some classic data-cleaning work. Typical jobs are deciding whether two product listings describe the same item, filling in a missing city, or spotting a wrong value. The authors turn each job into a plain-English question, paste the table rows in as text, and ask GPT-3 for an answer, adding a few worked examples to the question. Nothing is retrained.

On several public benchmarks these answers match or beat systems that were built and trained for each job. The results are sensitive to how the question is worded and which examples are picked, so the prompt is itself something you have to engineer.

TL;DR

The paper reformulates five data tasks as prompts to a frozen 175B-parameter language model. Rows are written as "attribute: value" text. With a handful of hand-picked examples, GPT-3 175B reports state-of-the-art or tied results on several benchmarks tab. 1-tab. 3. Small changes to prompt wording or to the chosen examples shift scores by 10 or more F1 points tab. 4. Everything is author-reported and uses one OpenAI model.

Problem

Data cleaning and integration (entity matching, schema matching, error detection, data transformation, imputation) usually needs a separate architecture for each task, hand-written rules or knowledge bases, and labelled data to fine-tune on §1. The paper asks whether a large pretrained language model can handle these tasks with only a natural-language description and a few examples, and what goes wrong in that setting §1.

Method

The procedure has three parts §3:

1. Serialization. Each table row becomes text of the form attr_1: val_1 ... attr_m: val_m. A NULL value becomes the empty string. Only attributes relevant to the task are kept, chosen by experiment §3.1. The serialization follows Ditto's convention §3.1.
2. Task as text generation. Each task has a fixed question template §3.2:
   • Entity matching: the two serialized entries, each introduced as "Product A is ..." and "Product B is ...", followed by the question "Are Product A and Product B the same?". The answer is Yes or No §3.2.
   • Data imputation: the serialized known attributes, then the name of the missing attribute and a question mark. The model writes the value §3.2.
   • Error detection: the question "Is there an error in attr_j: val_j?" about one cell §3.2. The answer is Yes or No.
   • The paper gives no question templates for data transformation or schema matching §3.2. Those tasks are evaluated in tab. 3. The repository's schema-matching serializer is at data_utils.py; it has no transformation code.
3. Demonstrations. A few labelled examples are placed before the query. Two ways of choosing them are compared §3.3:
   • Random: sample k labelled training rows.
   • Manual: a person writes k examples after looking at errors on a held-out validation split (10% of the labelled data), with at most one hour of tuning per task §3.3.

Model and evaluation: GPT-3 175B (text-davinci-002) through the OpenAI API §4.1. Baselines are task-specific systems (Magellan, Ditto, HoloClean, IMP, HoloDetect, SMAT, TDE) §4.1. Metric: F1 for entity matching, error detection and schema matching; accuracy for imputation and data transformation §4.1.

A short pseudocode version of the three prompt templates and the metric follows. It is written from the paper and the released code. Where the two disagree, see "Paper vs code".

# Serialize one row as "attr_1: val_1 ... attr_m: val_m"  [§3.1]
# A NULL value is written as the empty string  [§3.1]
def serialize(row, attributes):
    # One "attribute: value" text per attribute  [§3.1]
    parts = []
    for attr in attributes:
        value = row[attr] if row[attr] is not NULL else ""
        parts.append(attr + ": " + value)
    # Released code joins with ". " (paper states no separator)
    # [code: fm_data_tasks/utils/data_utils.py#L24-L40]
    return ". ".join(parts)

# Entity matching prompt, a yes/no question  [§3.2]
def match_prompt(row_a, row_b):
    # Each entry introduced as "Product A is" / "Product B is"  [§3.2]
    text_a = "Product A is " + serialize(row_a) + "."
    text_b = "Product B is " + serialize(row_b) + "."
    # Question after the two entries  [§3.2]
    question = "Are Product A and Product B the same?"
    return text_a + " " + text_b + " " + question

# Imputation prompt: the model writes the missing value  [§3.2]
def impute_prompt(row, missing_attr):
    # Known attributes serialized, then the missing attribute name  [§3.2]
    return serialize(row, known_attributes) + " " + missing_attr + " ?"

# Error detection prompt for one cell  [§3.2]
def error_prompt(row, attr):
    # Question names the cell being checked  [§3.2]
    return "Is there an error in " + attr + ": " + row[attr] + "?"

# Few-shot prompt: k labelled demonstrations, then the query  [§3.3]
def few_shot_prompt(demos, query):
    # Each demonstration is its prompt text followed by its label  [§3.3]
    blocks = [demo.text + demo.label for demo in demos]
    # Released code: blank line between blocks, one newline before the query
    # [code: fm_data_tasks/utils/prompt_utils.py#L32-L40]
    return "\n\n".join(blocks) + "\n" + query

# Answer one query  [code: fm_data_tasks/run_inference.py#L180-L190]
def answer(model, prompt, max_tokens):
    # Temperature 0, generation stops at the first newline
    # [code: fm_data_tasks/run_inference.py#L123-L131]
    return model.generate(prompt, temperature=0.0,
                          max_tokens=max_tokens, stop="\n")

# Binary F1 over Yes/No answers  [§4.1]
def f1_score(predictions, labels):
    # Counts over the Yes/No labels  [code: fm_data_tasks/utils/utils.py#L21-L58]
    precision = tp / max(1, tp + fp)
    recall = tp / max(1, tp + fn)
    return 2 * precision * recall / max(1, precision + recall)

Lightweight fine-tuning (appendix only) is a separate experiment. It places a small trainable adapter between two passes through a frozen model: the prompt goes in, the output embeddings go through the adapter, and the adapter's output is fed back as input embeddings for a second pass that produces the answer. Only the adapter is updated app. A fig. 3. The adapter's size and layers are not stated in the text. Full fine-tuning of the whole model is the comparison app. A.

Hyperparameters

Results

Author-reported. The setup is GPT-3 175B with few-shot prompts, compared with the task-specific baselines named in §4.1. Metric: F1 for entity matching and error detection, accuracy for imputation and data transformation, F1 for schema matching.

Entity matching, F1 tab. 1

Imputation (accuracy) and error detection (F1) tab. 2

Data transformation (accuracy) and schema matching (F1) tab. 3

Headline claims in the text §4.2: the 175B model with manual demonstrations reaches state-of-the-art on 4 entity matching, 2 imputation, 2 transformation, 2 error detection and 1 schema matching dataset. Some of these are ties: Fodors-Zagats (100 against Magellan and Ditto) and Adult (99.1 against HoloDetect) are counted in the text. Zero-shot results are weak on entity matching §4.2. Zero-shot imputation beats HoloClean on both imputation datasets §4.2 tab. 2.

Prompt tuning ablation, F1, k=10 tab. 4. Random-demonstration rows report mean ± standard deviation over three seeds.

Averages computed from the table, as the text reports them §4.3: attribute selection adds 13.7 F1 points on average; manual examples beat random examples by 14.7 points; and the prompt wording change moves scores by 9.4 points on average (see Pitfalls for how that average is formed).

Small-model fine-tuning, appendix app. A fig. 5. The text reports that full and lightweight fine-tuning of GPT-J 6.7B reduce the gap to the 175B model to an average of 3.1 points across the three datasets studied. The per-dataset values are only in fig. 5, not in the text. Lightweight tuning updates about 5% of the parameters and beats full fine-tuning on two of three datasets for 6.7B app. A. For GPT-Neo 1.3B, the text reports an average gap of about 25 points to the 175B model app. A.

Slice analysis, Restaurant city names tab. 5. Accuracy by how often the city appears in the training set:

Qualitative example, zip code and city tab. 6. Given an address and phone number, GPT-3 175B returns 32533 for the first zip code query, Malibu for the second city query and San Francisco for the third. The 6.7B and 1.3B models return other values for the first two queries (35205 and 35901; Torrance and Pacific Beach), shown in red in the table. The paper reads this as evidence of encoded functional dependencies app. B.

Paper vs code

Code: fm_data_tasks at commit c954cfdac6a420f773553e19dbfda81fba25df14, Apache-2.0. The repository contains the prompting and evaluation pipeline only. It does not contain the fine-tuning code.

• Decoding. Paper: not stated §4.1. Code: temperature 0.0, top_p 1.0, n 1, stop at newline, max 3 tokens by default run_inference.py run_inference.py. The released scripts use 5 tokens for Restaurant and 10 for Buy run_results.zsh.
• Model. Paper: text-davinci-002 §4.1. Code: no model name is passed run_inference.py.
• Serialization separator. Paper: no separator between attribute pairs §3.1. Code: pairs joined by ". " data_utils.py.
• NULL token. Paper: empty string §3.1. Code default is the token "nan"; the entity matching scripts pass an empty token, but the imputation and error detection scripts do not run_inference.py run_results.zsh.
• Entity matching question. Paper: question on its own line §3.2. Code: question appended after a space, and the per-dataset question differs, e.g. "Same Song?" for iTunes-Amazon and "Are A and B the Same?" for Walmart-Amazon constants.py.
• Imputation question. Paper: missing attribute name followed by "?" §3.2. Code: natural questions such as "What is the city?" and "Who is the manufacturer?" constants.py.
• Error detection template. Paper: "Is there an error in attr_j: val_j?" §3.2. Code: Hospital uses "Is there a x spelling error in attr: val?" data_utils.py constants.py. Adult serializes the whole row first, then asks about one cell data_utils.py. The paper does not say which template produced its Hospital numbers.
• Attribute subset. Paper: relevant attributes chosen by experiment, not listed §3.1. Code: per-dataset dropped columns are listed constants.py.
• Class balancing. Paper: not mentioned. Code: the released k=10 random runs pass --class_balanced for entity matching and Hospital; the sampler uses a fixed seed of 42 and skips imputation run_results.zsh data_utils.py.
• Demonstrations across trials. Paper: three random seeds with spread reported §4.3. Code: the random demonstrations are drawn once, in trial 0, and reused for every query and every later trial run_inference.py. The ± values in tab. 4 for random demonstrations therefore cannot come from different draws in this code as written.
• Zero-shot instruction. Paper: zero-shot is the bare template §3.2. Code: the released k=0 commands add a task instruction with --add_task_instruction run_results.zsh constants.py.
• Non-Yes/No answers. Paper: an answer that is not Yes or No counts as No §3.2 footnote. Code: an off-format answer for a No label is counted as a false positive utils.py.
• Matching rule. Entity matching uses exact match after stripping and lowercasing utils.py. Error detection uses a suffix match after taking the last paragraph utils.py.
• Test size. Paper: main-table sample sizes are not stated, except the Adult sample of 1K §4.1. Code: the released scripts run the first 200 test rows for every dataset run_results.zsh run_inference.py. The released scripts have no Adult command.
• Manual examples. Paper: written by hand on the validation split §3.3. Code: the hand-written prefixes are stored in the repository constants.py and the default split is validation unless --do_test is passed run_inference.py. Use the repository copy.
• Extra sampling method. The code has a validation-cluster sampler built on sentence embeddings prompt_utils.py prompt_utils.py. The paper does not describe it, and the released scripts do not use it.
• Data transformation and schema matching. The paper evaluates on StackOverflow, Bing-QueryLogs and Synthea tab. 3. The repository has no transformation datasets; its task map covers Synthea and the other tasks constants.py.
• Metric bug. The error-detection branch of the metric code contains a breakpoint() call utils.py. Running the Adult evaluation will stop in a debugger or hang a batch job until it is removed.
• Fine-tuning. The paper describes adapter and full fine-tuning app. A. The repository has no fine-tuning code, so the appendix numbers cannot be reproduced from it.

Pitfalls

• The model is the experiment. Results are for one closed model snapshot (text-davinci-002 §4.1). A different model, a newer snapshot or a different decoding setting is a different experiment, and the paper gives no evidence for other models in this card.
• Ties count as wins. Fodors-Zagats and Adult are ties with the baselines tab. 1 tab. 2, but the text counts them among the state-of-the-art datasets §4.2.
• The "9.4 point" prompt effect is an absolute average. The three per-dataset differences are +3.7, +13.5 and -11.1 tab. 4. Their mean with sign is about 2.0 points; the 9.4 figure averages the absolute values. The direction of the effect changes between datasets.
• Few datasets, few examples. The ablations use three entity matching datasets with up to 200 samples each §4.3. The averages in the text come from these three rows, so their spread is not estimated well.
• Demonstration choice is a hidden cost. The manual examples took up to one hour of analysis per task on the validation split §3.3. The paper calls this cheaper than fine-tuning, but it is still human work that the repository contains as hard-coded text constants.py.
• Attribute selection is not listed. The paper says relevant attributes were chosen through experiments §3.1. The repository lists which columns to drop per dataset constants.py. Without these, the attribute-selection results cannot be reproduced.
• Zero-shot error detection is unusable. Zero-shot scores are 6.9 on Hospital and 0.0 on Adult tab. 2. Few-shot demonstrations are needed for this task.
• Cost. Each query is an API call to a 175B-parameter model. The paper subsampled Adult to 1K rows for budget reasons §4.1. Per-query cost is not stated.
• Slice analysis counts are not given. The subclass sizes behind tab. 5 are not reported. Two or four examples per cell would change the 25.0 and 50.0 values a lot.
• Figure-only numbers. The per-dataset fine-tuning curves are only in fig. 5. Do not infer them from the text.
• Output parsing. Answers are generated text, not class scores. The released code stops at the first newline and compares strings run_inference.py. An answer that starts with a different word fails silently.

Lineage

• Builds on: Ditto's serialization of attribute-value pairs §3.1; the frozen model with a trained adapter, cited in the paper as frozen2022 app. A; GPT-3 §1.
• Compared against: Magellan and Ditto for entity matching; HoloClean, IMP and HoloDetect for cleaning; TDE for data transformation; SMAT for schema matching §4.1.
• Related prompt-sensitivity work the paper cites: calibration of prompt formats §3.1 and ordering effects of demonstrations §3.3.