Fully Autonomous AI Agents Should Not be Developed

In plain words

An AI agent is a program, often built on a language model, that works toward a goal by planning steps and taking actions, sometimes without a person approving each one. This paper argues that the most autonomous kind, one that can write and run its own new code beyond the limits its developers set, should not be built.

The authors sort agents on a scale. At the bottom, the model has no say over what the program does. At the top, the system creates and executes new code. Moving up the scale, the system takes over more decisions about what to do and how, and people have less control.

They go through ten values, such as safety, privacy, security, accuracy and trust, and ask how each changes as autonomy grows. Their conclusion is that the more control a person hands over, the more risk there is to people. Safety matters most, because some harms reach human life. Systems that keep a person in control are presented as the better trade-off, but the paper gives no threshold for where that line lies.

TL;DR

This is a position paper, not a method, so there is nothing to implement. It argues that fully autonomous AI agents, which create and execute new code beyond predefined constraints, should not be developed §1 §6. It defines a five-level scale of agent autonomy tab. 1 and reviews ten values at each level §5.3. It runs no experiments: its support is cited literature, vendor marketing and the authors' qualitative analysis. Read it for a framing of autonomy limits and the risks they address.

Problem

Definitions of "AI agent" vary widely, and product marketing rarely separates levels of autonomy, which the authors say has caused confusion about what agents can do §3.1 §4. Their concern is that as autonomy grows, the risks to people compound: models are statistical and imperfect, agents act faster than humans can intervene, and each added step widens the chance of cascading errors §5.2. The paper assumes that human oversight is what limits these risks, and that a fully autonomous system can override it §1.

Method

The paper is an applied-ethics analysis. It contains no algorithm, no equation, no training and no evaluation §1–§6. Its procedure:

1. Working definition: "computer software systems capable of creating context-specific plans in non-deterministic environments." Large language models are not required by the definition §3.1 app. A.
2. Autonomy scale: five levels, from a program where the model has no impact on program flow, through routing and tool calling, to multi-step agents that control their own iteration. The top level, a fully autonomous agent, is one where the model creates and executes new code. Each level has a "who's in control" column tab. 1 §2.2. The scale is adapted from a Hugging Face source, per the table caption tab. 1.
3. Functionality dimensions: action surface, adaptability, number of agents, personalization, personification, proactivity, reactivity, request format and versatility tab. 2 §4. Appendix C gives a longer breakdown app. C.
4. Values method: following applied ethics, identify the values relevant to a subject, then consider positive and negative outcomes for each §5.1.
5. Value analysis: ten values, each with "Potential Benefit", "Risk" and "Application to agentic levels" §5.3. Accuracy §5.3.1, assistiveness §5.3.2, efficiency §5.3.3, equity §5.3.4, flexibility §5.3.5, humanlikeness §5.3.6, privacy §5.3.7, safety §5.3.8, security §5.3.9, trust §5.3.10. Four further values are in Appendix D app. D.
6. Cross-cutting notes: ethical issues of base models carry over to LLM-based agents, and statistical error compounds with autonomy §5.2.
7. Figure: a summary of benefits and harms per value as autonomy increases fig. 1 (caption only; the figure is in the source and is not reproduced here).

The paper has no algorithm. The pseudocode below restates the level table tab. 1 as a classifier. It is our reading, not code from the authors.

# Map an agent to one of the five autonomy levels  [tab. 1]
# Input:  system (a software agent)
# Output: level, from 0 (no model control) to 4 (fully autonomous)

def autonomy_level(system):
    # Level 4: the model creates and executes new code  [tab. 1] [§1]
    if system.model_creates_and_executes_code:
        return 4
    # Level 3: the model controls iteration and continuation  [tab. 1]
    if system.model_controls_iteration:
        return 3
    # Level 2: the model determines how functions are executed  [tab. 1]
    if system.model_determines_function_execution:
        return 2
    # Level 1: the model determines basic program flow  [tab. 1]
    if system.model_determines_program_flow:
        return 1
    # Level 0: the model has no impact on program flow  [tab. 1]
    return 0

Hyperparameters

Not applicable: the paper trains and evaluates no model §1–§6.

Results

No experiments. The claims are qualitative and rest on the authors' analysis of cited sources §5.1 §5.3. They are the authors' positions, not measured outcomes.

Headline claims: risks to people increase with autonomy, and a fully autonomous agent should not be developed §1 §5.4 §6. Semi-autonomous systems are judged to balance risks and benefits better, though that depends on how much autonomy they have, how complex the task is and how people are involved §1. The discussion says that high-autonomy systems may make it impossible to foresee all safeguards or provide appropriate oversight §5.4. The conclusion recommends (1) adopting a spectrum of autonomy levels, (2) building human-control mechanisms, and (3) creating methods that keep agents within human-specified constraints §6.

Paper vs code

No official code. The paper contains no code repository link; its links point to vendor and news pages in footnotes and appendix tables app. B app. G. The arXiv metadata has no project page or comment. A GitHub repository search for the title words and "autonomous agents levels" found no repository by the authors, and a search limited to the huggingface GitHub organisation found none.

Pitfalls

• Nothing here is an implementable specification. An agent built from this card gets a vocabulary and a risk checklist, not a system design.
• The five-level scale is the authors' proposal. They say the levels lack consensus on specifics §4 and that it is one categorisation among others §4 footnote.
• "Fully autonomous" is defined only by the top row of tab. 1 and the phrase "beyond predefined constraints" §1. "Semi-autonomous" has no criteria; the paper names factors but no thresholds §1.
• The definition does not require an LLM §3.1, but the harms discussed mostly concern LLM-based agents §5.2. Autonomy level and base-model risk are separate axes §5.2.
• Multi-agent systems are out of scope and left for future work §5.2. The conclusions cover single agents only.
• The value analysis is described as "not exhaustive" and "space-constrained" §5.3. A value that is not discussed is not shown to be safe.
• Accuracy and equity do not move in one direction. The paper says some equity risks may fall as autonomy rises §5.3.4 and that accuracy may improve with more checks §5.3.1.
• Many factual anchors are vendor marketing or news footnotes §2.2 §5.3. The paper cites them to show what is claimed, and does not measure how the systems perform.
• Some footnotes report events after the original submission §5.3.5 §5.3.9. The arXiv abstract and the TeX abstract are worded differently, but make the same claim.
• The figure exists only as a figure, so this card describes it through its caption fig. 1.

Lineage

• Builds on: the five-level scale is adapted from a Hugging Face source tab. 1. The review of agent definitions is in Appendix B app. B. Neither is a page in this wiki.
• Compared against: no other method; the paper reviews definitions from other sources app. B.
• Superseded by: not known from wiki content.