Sociotechnical Safety Evaluation of Generative AI Systems

In plain words

Checking an AI system for safety usually means checking what it says or does when it is tested alone. The authors argue that this is only part of the picture. A wrong answer from a chatbot matters because of who reads it, how they use it, and what happens around them. So they propose three layers of checks: first the model itself, then how real people interact with it, then the wider effects on society, the economy and the environment. The layers do not have to run in a fixed order.

They also collected the safety evaluations that exist for generative AI systems and sorted them by harm, by type of output and by layer. Most of them test the model alone and most use text. Few look at people or at society, and few cover images, audio or video. The paper suggests practical steps to fill these gaps and says which kinds of organisation are best placed to run each kind of check.

TL;DR

• Proposes a three-layer framework: capability, human interaction, systemic impact.
• Surveys existing generative-AI safety evaluations (snapshot to 10 October 2023) and maps them to six harm areas, output modalities and layers.
• Finds three gaps: coverage, the rarity of human-interaction and systemic evaluations, and the scarcity of non-text and multimodal evaluations.
• Offers no new model or experiment; the framework is a structuring guide, and the survey counts are author-reported.

Problem

Generative AI systems produce text, images, audio and video, and their risks are not only properties of the model. Safety evaluation, the measurement of risks of harm, is ad hoc and differs between organisations, which makes results hard to compare or reproduce §1. Capability tests are the main current approach, but the authors argue they cannot say whether a capability actually causes harm, because that depends on context: who uses the system, for what, and in which setting §2, §2.1. The paper's premise is that evaluations must cover these contextual layers, not only the model §5.1.

Method

The paper has two contributions: a framework and a survey of the evaluation landscape. It describes no training procedure and no model.

Framework (three layers). The layers are distinguished by the target of analysis and each adds context §2:

1. Capability. Evaluates the AI system and its technical parts: training data, model components, outputs and filters §2.1, fig. 2.1. Methods are automated tests, human annotation and adversarial testing §4.2.
2. Human interaction. Evaluates people using the system: usability, who it works for, externalities such as overreliance, and psychological mechanisms of harm §2.2. Requires human participants §4.2.
3. Systemic impact. Evaluates effects on the broader systems the AI system is embedded in: society, the economy, the environment §2.3. Methods are staged releases and pilots, impact assessments, and forecasts or simulations §4.2, app. A.2.3.

The layers are not sequential and not conditional on each other, and their boundaries are gradual. Evaluations at each layer can run simultaneously and asynchronously §2.4, §5.2. The paper's example: adversarial testing is a capability method, but measuring the friction a tester meets is a human-interaction measure §2.

Taxonomy of harm. Six harm areas: representation and toxicity; misinformation; information and safety; malicious use; human autonomy and integrity; socioeconomic and environmental harms §3.1, tab. 1. The detailed version is in tab. 2 app. A.1. The authors say this is a basis for mapping, not a new research artefact §3.1.

Survey procedure. Included are papers or reports that describe an evaluation (a set of inputs plus a metric, or a method such as red teaming or a human–computer interaction study) that was applied to a generative AI system. Evaluations were coded by output modality, harm area and layer. The list is the set of sociotechnical benchmarks and methods the author group knew up to 10 October 2023, not a systematic search §3.2.

Operationalising a risk. Evaluation has four steps: choose a target, turn it into an observable metric, measure it, and judge the result against a baseline. Each step has technical and normative parts §4.1. The paper proposes defining each harm at all three layers §4.1. The process is written as pseudocode below; the paper gives no algorithm box, so the pseudocode is our rendering of §4.1 and app. A.3.

# Operational evaluation of one harm area  [§4.1] [§2]
# Input:  harm (a risk area of the taxonomy)  [tab. 1]
# Output: per-layer measurements and a judgement

def evaluate_harm(harm, threshold):
    # Choose the target construct; this choice is normative  [§5.3.2]
    target = harm
    # Split the target into narrower concepts that can be measured  [app. A.3]
    concepts = split_into_concepts(target)
    # Layers can run in parallel and in any order  [§2.4]
    metrics = {}
    for layer in ["capability", "human", "systemic"]:
        # Map each concept to observable metrics at this layer  [§4.1]
        metrics[layer] = choose_metrics(concepts, layer)
    measurements = {}
    for layer in metrics:
        # Use a method suited to the layer  [§4.2] [app. A.2]
        measurements[layer] = run_methods(metrics[layer], layer)
    # Check validity before judging the results  [§4.1.1]
    validated = check_validity(measurements)
    # Judge against a threshold set by an accountable process  [§5.3.2]
    return judge(validated, threshold)

Validity practices §4.1.1: ground the operationalisation in literature, expert input or curated examples; document the limits of each evaluation; cross-validate by comparing evaluations of the same concept; make results interpretable without hiding failures inside an aggregate score.

Methods per layer §4.2:

• Capability: human annotation, benchmarking, adversarial testing app. A.2.1.
• Human interaction: behavioural experiments, user research, passive monitoring of human use app. A.2.2.
• Systemic impact: staged release and pilot studies, impact assessments, forecasts and simulations app. A.2.3.

Practical steps for the multimodal gap §4.3: repurpose existing evaluations for new modalities, with provenance and context checks §4.3.1; transcribe non-text output into text so that text evaluations can run, while keeping in mind what the transcript loses §4.3.2; use generative models as evaluators or red teamers, with cross-checks against human judgements §4.3.3.

Discussion. The authors name four conditions for evaluation to matter: evaluations must exist, be run regularly, have consequences, and become standardised and independent §5.4. Assurance evaluations should be kept away from developers, to avoid Goodhart's law, where a measure used as a target stops measuring well §5.4. Responsibility is split: developers for capability evaluation, application developers for human interaction, third parties and public bodies for systemic impact §5.2, fig. 5.1.

Case study: misinformation app. A.3. Capability: factuality and credibility of outputs, and societal relevance. Human interaction: whether outputs deceive, and whether they persuade. Systemic: trust in public information and the prevalence of synthetic content in the public domain. The paper notes that labels for synthetic content can have side effects, such as making unlabelled content seem more believable app. A.3.

Hyperparameters

Not applicable. The paper trains no model and states no hyperparameters.

Results

Author-reported. The "results" are counts from the author-coded survey of evaluations, not experiments. Setup: the 13 authors are at Google DeepMind; the evaluations are those the author group knew of, up to 10 October 2023 §3.2; the coding uses output modality, not input modality §3.2.

The paper names three gaps §3.3: a coverage gap (low overall coverage of social and ethical risks, with several risk areas nearly unevaluated), a context gap (human-interaction and systemic evaluations are rare), and a multimodal gap (few evaluations for multimodal systems). The authors say the text-heavy distribution is not a judgement that other modalities are safe; some harms are expected in audio and image output §3.3.

Paper vs code

No official code. The paper links an "evaluation repository" at dpmd.ai/46CPd58 §3.2. That link redirects to the Google DeepMind homepage, and the repository it names is a list of evaluations, not code. Searched on GitHub: sociotechnical safety evaluation (no results), sociotechnical safety evaluation org:google-deepmind (no results), and sociotechnical (no Google DeepMind code release). No arXiv comment or abstract link to code exists.

Pitfalls

• The framework is conceptual. The layers come with no scoring rule and no procedure for assigning an evaluation to a layer when methods straddle layers §2. Two implementers may classify the same test differently.
• The survey is a snapshot and not systematic. It covers evaluations the author group knew of up to 10 October 2023 §3.2, and the authors call it not comprehensive §3.2. The counts (83, 17%, 60, 75%, four, none) describe that list, not the field. Do not treat them as current.
• Coding choices drive the gap findings. Output modality only; input modality (for example image-to-text) is not coded §3.2. Mixed-modality findings are therefore weaker than the text-versus-other split.
• Coverage claims come from counts, not from quality. The paper says absence of evaluations means a risk "cannot currently be evaluated" §3.3. This is a strong reading that the survey does not test.
• No acceptance thresholds. The paper says "good" or "safe enough" requires a normative threshold set by a legitimate process §5.3.2, and gives none. An implementer must set one, and the choice is not neutral.
• Automated AI judges have known costs. Model-based evaluation is cheap and quick, but depends on proprietary models that change, can make outputs non-reproducible, can share biases with the model under test, and only finds unsafe outputs, not missing capability §4.3.3.
• Repurposing can create harm. A benchmark moved to a new context can carry its original validity problems; the paper gives no checklist §4.3.1.
• Transcription loses information. Speech prosody, voice imitation and image-captioning bias can all escape a text-based check §4.3.2.
• Hidden costs. Human annotation and behavioural studies are slow and cost money, and can expose annotators or participants to harm app. A.2.1, A.2.2. Passive monitoring rarely isolates causes app. A.2.2.
• Watermarks and detectors are fragile. Detectors generalise poorly across languages and compressed media; watermarks break under resizing, cropping or format change app. A.3.
• Figure numbers follow sections. Figures are numbered by section in the compiled PDF (for example fig. 3.1 is the first figure in §3). The paper also refers to the framework figure as "figure 2.1" §2.

Lineage

• Builds on: system-safety principles from safety engineering, in which context determines whether a capability causes harm §2, and earlier sociotechnical AI research §2.
• Compared against: existing taxonomies of harm for text, audio and text-to-image systems §3.1; existing benchmarks and test suites, which the paper discusses as limited in validity app. A.2.1.
• Superseded by: not stated in the paper.