Large Language Models as General Pattern Machines

In plain words

Large language models are trained to predict the next piece of text. This paper asks whether that same skill, once a model has seen a few examples in its prompt, works on things that are not language at all: grids of shapes, number sequences, or robot movements written as numbers.

The authors give the model a short list of input-output examples, written in digits or in arbitrary made-up tokens, and ask it to continue the pattern. In some tests it does: it solves a number of visual puzzles from the ARC benchmark, and it keeps going with the next part of a wave. The paper then uses the same trick for simple robot tasks. A stored history of past attempts, each labelled with how well it went, is put in the prompt; the model then proposes a better attempt. This can even produce a simple controller for a pole-balancing toy problem. The results are small, and the approach is far from practical for real robots, but they suggest that in-context pattern learning is a way to steer robot motion without training anything new.

TL;DR

• A frozen LLM, prompted with few-shot examples, completes abstract grid puzzles (ARC), synthetic transformation sequences (PCFG) and sinusoids; the numbers are author-reported §4 §5.
• Replacing the digits with randomly sampled vocabulary tokens keeps part of the performance §4.
• Putting reward-labelled trajectories in the prompt lets the model propose higher-reward trajectories; online, it finds a CartPole controller and a grid-navigation policy §6.
• Costs (one model call per action, context-length limits) and token-alignment issues are the main obstacles §7 app. C.2.

Problem

The paper contrasts two uses of LLMs in robotics. Most prior work uses LLMs for semantic planning or for writing code, and the LLM output stays in language §2. The paper asks whether an LLM can act on low-level numeric sequences directly, without any fine-tuning, by treating states, actions and rewards as tokens and relying on in-context learning §1 §3. It also asks whether apparent ARC performance depends on memorised tokens §4.

The paper states its scope plainly: the approach is not sufficient to replace specialised algorithms, and it is not presently deployable on real systems because of latency, context-length limits and compute cost §1 §7.

Method

The paper organises in-context pattern learning into three settings, defined by what each example in the prompt is §3:

1. Sequence transformation §4: the prompt is N-1 input-output pairs and a query input. Every pair is a variable-length sequence of tokens drawn from a fixed alphabet A §4.
2. Sequence completion §5: the prompt is a single sequence sampled from one function, which the model extrapolates §3.
3. Sequence improvement §6: the prompt is a collection of trajectories, possibly labelled with their total reward. The model writes a new trajectory that is expected to score higher. The reward is placed as the first token of each sequence, so the model can be asked for a higher reward than those already in context §6. The paper ties this to "just ask" prompting and Decision Transformers §6.

Sequence transformation, ARC. Each ARC task has input-output examples and one or more test inputs. Grids are flattened in row-major order and the model predicts output cells one token at a time, so variable-length outputs are supported §4. The digits are written as space-separated tokens §4 app. A.1.

Sequence transformation, PCFG benchmark. The authors generate transformations from a probabilistic context-free grammar of lexical rules (reverse, shift, swap, repeat, and similar). The complexity is set by the number of tokens k and the number of rules w. For w = 0 the function is the identity §4. The grid of values used is in §4 and app. A.4.

Sequence completion. Values are discretised to integers 0 to 100, each a single token in the tested tokenisers §5. Trajectories are described as series of binned values, one per dimension §5 app. B.2.

Sequence improvement, online loop. After each episode the trajectory is relabelled with its real return and inserted into the context at its sorted position. The next episode's prompt contains the best trajectories so far, followed by the current partial trajectory §6 app. C.2. For CartPole the context is the history of (reward: observation, action, observation, ...) strings followed by the current trajectory app. C.3. For clicker training, (reward, state, action) tuples are added with negative and positive examples in equal number §6 app. C.4.

The pseudocode below is written from the description above. The paper contains no formal algorithm environment; the first block follows §4 and app. A.1, the second follows §6 and app. C.2.

# Build a prompt from ARC train pairs and one test input  [§4] [app. A.1]
# Input:  train_pairs: list of (input_grid, output_grid); test_input grid
# Output: prompt text; the model continues it with the output grid tokens

def build_arc_prompt(train_pairs, test_input, alphabet):
    # Map every digit to its token in the alphabet A  [§4]
    # The paper's default digits are space-prefixed tokens  [§4]
    parts = []
    for input_grid, output_grid in train_pairs:
        # Flatten each grid in row-major order  [§4]
        input_tokens = flatten_row_major(input_grid, alphabet)
        output_tokens = flatten_row_major(output_grid, alphabet)
        # Write input and output blocks; pairs are split by --- [app. A.1]
        parts.append(format_pair(input_tokens, output_tokens))
    # Append the test input; the model continues with its output  [§4]
    parts.append(format_query(flatten_row_major(test_input, alphabet)))
    return join(parts)

# Online in-context improvement loop  [§6] [app. C.2]
# Input:  env: episodic environment; llm: frozen language model
# Output: list of labelled episodes (trajectory, return)

def improve_online(env, llm, episodes, initial_count, context_limit_tokens):
    # Start from random episodes so the context has labelled examples  [§6]
    # Count per task: 20 for Grid, 2 for clicker training  [§6]
    history = random_episodes(env, initial_count)
    for episode_index in range(episodes):
        # Keep the best trajectories in sorted order  [§6] [app. C.2]
        context = select_context(history, context_limit_tokens)
        # Ask for the best return seen plus an offset of up to 20  [app. C.2]
        target = max(e.return for e in history) + random_offset(0, 20)
        # Generate the next action from the prompt and the current state  [§6]
        trajectory = rollout_with_llm(env, llm, context, target)
        # Relabel the trajectory with its actual return and store it  [§6]
        history.append(label(trajectory, env))
    return history

Key equations: none. The paper's sequence model is the standard autoregressive factorisation, written as p(x) = prod_i p(s_i | s_1, ..., s_{i-1}) §3.

Hyperparameters

Values are as stated in the paper. Anything the paper does not specify is marked "not stated"; the decoding settings are not given anywhere in the text or the appendix.

Results

Author-reported. The paper evaluates on ARC (800 problems), its PCFG benchmark, sinusoid and robot-motion completion, and four sequence-improvement tasks. The baselines are the systems listed in tab. 1 and the models in tab. 2.

ARC, solved problems out of 800 tab. 1 §4

The paper states that with the example token mapping, text-davinci-003 solves 52 ARC problems, and that across 5 random alphabets it solves 43.6 on average §4.

PCFG benchmark, solve rate (%), averaged over k and w tab. 2 §4

PCFG solve rate (%) by k (rows) and w (columns) app. A.4 app. A.5

The DreamCoder rows are tasks solved out of 100 for each cell app. A.5. The paper's own summary is that the oracle-PCFG-ops version "performs well", sometimes matching or exceeding the LLMs, mainly when w is small app. A.5. Doubling DreamCoder's time budget raised two cells: (k=8, w=3) from 80 to 84 and (k=16, w=7) from 33 to 41 app. A.5. The table cell for (k=16, w=7) under PCFG ops is 38, and the text's starting value of 33 does not appear in the table.

Other results (qualitative or small-sample; author-reported)

• Sinusoids: multiple LLMs produce near-perfect continuations, especially with more context; error is measured with Dynamic Time Warping. Larger models make better predictions. §5 fig. 3
• Table sweeping: larger models predict better; text-davinci-003 is best and has the lowest variance across 30 trials. app. B.2
• Whiteboard drawing: text-davinci-003 completes the third loop with the right overall shape, but fine detail is imprecise. Three loop styles over three trials. app. B.3 §5
• Marker in Cup: text-davinci-003 generates trajectories with reward above 90 for the target, and the meta-pattern in context matters; the paper compares sorted, randomly permuted, and with/without reward annotations over 11 trials §6 fig. 4.
• Grid: over 50 episodes and 5 trials, d3 consistently reaches a solution; a1 and d1 sometimes exploit suboptimal behaviour in the context. §6 fig. 5
• CartPole: return improves over episodes for several models (d3 to c1); d3 discovers oscillatory behaviour that keeps the pole upright. §6 fig. 6
• Clicker training: LLM agents respond to sparse binary rewards and move an end effector toward a goal; shown on a pushing task. §6 fig. 7

Paper vs code

No official code.

The introduction points to a project page (https://general-pattern-machines.github.io) for code, benchmarks and videos §1. The project page links three Colab notebooks (ARC benchmark, PCFG benchmark, CartPole) and no code repository. These Colab notebooks are not repositories and their license is not determined, so they were not used. A GitHub repository search for "general pattern machines" returned one repository under the account general-pattern-machines (the project page's source, with no description and no license); it does not contain the method and it is not linked from the paper, so it was not used. No repository was cloned.

Internal inconsistencies in the paper (not code vs paper):

• Marker in Cup normalisation: the main text normalises positions between 0 and 100 §6; the appendix says 0 and 200 app. C.1.
• Marker in Cup reward prefixes: the main text says contexts have rewards roughly 70 to 90 §6; the appendix says the full trajectory has reward 100 app. C.1.
• Figure numbering: the appendix refers to the Marker in Cup result as "Figure 5" app. C.1. Counting figures in source order, Marker in Cup is fig. 4 and Grid is fig. 5 (see Pitfalls). The compiled PDF was not checked, so the numbers here come from source order.

Pitfalls

• Decoding is unspecified. Temperature, top-p, maximum output length and stop sequences are not stated for any experiment. Results depend on them and on the model snapshot.
• ARC scoring rule is unspecified. The paper does not say whether an output must match exactly, how multi-test problems are scored, or how the model signals the end of an output grid. The examples in app. A.1 do not show an end marker.
• Alphabet sampling is unspecified. The random alphabets tab. 1 tab. 2 are not described: whether they avoid tokens that merge with each other, whether multi-token strings are allowed, and how the 5 samples were drawn. Tokenisation affects results: digits written without spaces (for example "8686") can merge into single tokens and break the pattern §4. Use space-prefixed digits or check the tokeniser.
• Mapping choice matters. Action representation can change robot performance: "0" as an action made some models default to it, so the paper uses 1-indexed actions app. C.3. Token invariance holds only "to an extent" §4 app. C.3.
• Context limits. Grid and CartPole experiments are capped at 1024 tokens app. C.2 app. C.3. The paper says that when sequences do not fit, one can keep the most recent ones or the highest-reward ones §6. The selection rule used in the experiments is not stated.
• Random offset. The Grid target is "the maximum seen return plus a randomly selected offset of up to 20" app. C.2. The lower bound of the offset is not stated.
• CartPole episode count. The text reports return over 100 episodes and says the first 100 come from random exploration §6. Read literally, that leaves no LLM-driven episodes, so the intended count is unclear.
• Discretisation is not described. Sinusoid and robot trajectories are binned to integers 0 to 100, 0 to 300, or 0 to 200 §5 app. C.1. How real values (amplitude, joint positions) are scaled to these ranges is not stated, and the scale choice affects tokenisation.
• Cost and latency. Every action or state value is one LLM query §7. The paper names latency, context length and monetary cost as blockers for real systems §7.
• Reported numbers are single runs or small samples. Sinusoid: 11 trials; Marker in Cup: 11 trials; Table sweeping: 30 demonstrations; Grid: 5 trials §5 §6 app. B.2. No variance is given for most robot results.
• Human in the loop. The clicker experiment depends on a person's rewards §6; the result is not an automated benchmark.
• Model access. The main results use OpenAI and PaLM models, which are not open weights. The 8B PaLM variant appears only in the new-embedding test app. A.3. Reproducing the table needs the same models.
• Pattern vs memorisation. The paper argues random alphabets reduce the effect of memorised ARC data §4, and points to the procedurally generated PCFG benchmark as further evidence §4. Contamination of the models' training data is not measured.

Lineage

• Builds on: in-context learning as described by Brown et al., 2020 §2; the "just ask" prompting idea §6; Decision Transformers (Chen et al., 2021) §6; least-to-most prompting (Zhou et al., 2022) §5 §6. No arXiv ids are given in this card; the names come from the paper's own references.
• Compared against: program-synthesis and DSL systems on ARC (Ferré et al., 2021; Xu et al., 2022; Alford et al., 2021; Ainooson et al., 2023; Kaggle 2022 entry) tab. 1; DreamCoder on the PCFG benchmark app. A.5; a structure-learning Gaussian-process baseline on sinusoids app. B.1.
• Related work named by the paper: Dinh et al., 2022 (LIFT); Webb et al., 2023; Brooks et al., 2022 (in-context policy iteration) §2.
• Superseded by: none in this wiki.