The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systems

In plain words

Machine learning systems often still need people: to label data, to correct mistakes, to decide what the model should learn next, and to explain the results. This paper argues that the most useful systems will be teams of people and AI that get better together over time. It calls these "hybrid intelligence" systems.

The authors do not build such a system. They collect the design decisions a builder has to make, such as what kind of task is involved, how the human teaches the machine, how the machine gives feedback, and how much the system should explain itself. They group these decisions into a structured list (a taxonomy) so that a developer can walk through them in order.

The paper is a first map, not a tested recipe. It does not measure whether the map helps anyone build a better system.

TL;DR

A conceptual taxonomy for designing human-AI systems that learn from each other. It has four meta-dimensions (task characteristics, learning paradigm, human-AI interaction, AI-human interaction), 16 sub-dimensions and 50 categories §4 §6 fig. 3. It was built from a literature review and seven business cases §3.2 tab. 1. There are no experiments, so the taxonomy's usefulness is untested §6.

Problem

Current business applications cannot be solved by machines alone, and the machine learning pipeline still needs human input for feature engineering, parameter tuning, labeling, debugging and training §1 §2.2. The authors want structured design knowledge for systems that combine humans and machines in one socio-technical ensemble, where the two parts co-evolve §1 §2.3. They note that earlier research on decision-support and expert systems did not focus on systems that keep learning from their users, and they exclude knowledge-base systems from their review §2.3 §3.2.

Method

The paper gives no algorithm for a learner. Its method is a taxonomy development procedure, adapted from Nickerson et al. §3.1 [ref. 13]. The loop is shown in fig. 2 and summarised below.

1. Define a meta-characteristic: the generic design dimensions that all hybrid intelligence systems need. The authors start from four §3.1.
2. Set stopping conditions. Five objective conditions and six subjective conditions are listed §3.1.
3. Choose an approach for the iteration:
   • empirical-to-conceptual: pick a new subset of objects and proxy objects, find their common characteristics, and group them into dimensions to build a vector taxonomy fig. 2;
   • conceptual-to-empirical: conceptualise characteristics and dimensions from theory, examine objects against them, and build a verbal taxonomy fig. 2.
4. Repeat until the stopping conditions hold fig. 2 §3.1.

The objective conditions are: all sampled papers and empirical cases are examined; every object is classified under at least one characteristic of every dimension; the last iteration adds no new dimension or characteristic; every dimension is unique; and every characteristic is unique within its dimension §3.1. The subjective conditions are conciseness, robustness, comprehensiveness, extensibility, explanatory power and information availability §3.1.

# Taxonomy development loop  [§3.1] [fig. 2]
# Input:  meta-characteristic (initial dimensions), stopping conditions
# Output: taxonomy of dimensions, characteristics and categories

def develop_taxonomy(meta_characteristic, stopping_conditions):
    # Initial dimensions come from the meta-characteristic  [§3.1]
    dimensions = meta_characteristic
    while not stopping_conditions_met(dimensions, stopping_conditions):
        if approach == "empirical-to-conceptual":
            # Pick new objects and their proxy objects  [fig. 2]
            objects = select_new_objects(sample)
            # Find characteristics shared by these objects  [fig. 2]
            characteristics = find_common_characteristics(objects)
            # Group characteristics into dimensions  [fig. 2]
            dimensions = group_into_dimensions(characteristics)
        else:
            # Conceptualise characteristics from theory  [fig. 2] [§3.2]
            dimensions = conceptualise(literature_knowledge)
            # Classify each object against those characteristics  [fig. 2]
            classify_objects(objects, dimensions)
    return dimensions

The paper ran three iterations: conceptual-to-empirical from theory, then empirical-to-conceptual from cases, then conceptual-to-empirical again on an extended literature review §3.2.

Data sources. The literature review was conducted between April and June 2018 §3.2. The search covered AISeL, IEEE Xplore, ACM DL, AAAI DL and arXiv, and was not restricted to particular publication venues §3.2. The search string used terms such as "hybrid intelligence", "human-in-the-loop", "interactive machine learning", "machine teaching" and "human supervision" §3.2. The search returned 2505 hits. Three researchers screened titles, abstracts and keywords independently, leaving 85 articles for detailed review §3.2 tab. 1.

Empirical cases. Seven real business applications were analysed as additional evidence: Teachable Machine, Cindicator, vencortex, Cobi, Stitch Fix, AlphaGo and a Microsoft custom decision service tab. 1 §3.2.

Taxonomy structure (§4, shown in fig. 3). The four meta-dimensions and their sub-dimensions are as follows.

• Task characteristics §4.1:
  • Type: recognition, prediction, reasoning, action.
  • Goals: common, adversarial, independent.
  • Data representation: features, instances, concepts, schemas.
  • Timing in the ML pipeline: feature engineering, parameter tuning, training.
• Learning paradigm §4.2:
  • Augmentation: human, machine, hybrid.
  • Machine learning paradigm: supervised, unsupervised, semi-supervised, reinforcement.
  • Human learning paradigm: experience, explanation.
• Human-AI interaction §4.3:
  • Machine teaching: demonstrating, labeling, troubleshooting, verification.
  • Teaching interaction: implicit, explicit.
  • Expertise requirements: ML expert, domain expert, end user.
  • Amount of human input: individual, collective.
  • Aggregation: unweighted, human-dependent, human-task-dependent.
  • Incentives: monetary rewards, intrinsic rewards, customisation.
• AI-human interaction §4.4:
  • Query strategy: offline, active learning, online.
  • Machine feedback: suggestion, prediction, clustering, optimisation.
  • Interpretability: algorithm transparency, global model interpretability, local prediction interpretability.

The authors define "hybrid intelligence" as systems that achieve complex goals by combining human and artificial intelligence, collectively achieve superior results, and keep improving by learning from each other §2.3.

Hyperparameters

This paper has no training hyperparameters. The values below are the method's own settings and counts.

Results

Author-reported. There are no experiments and no baselines. The output is the taxonomy itself. Setup: literature review and seven business cases §3.2.

Paper vs code

No official code. Searches: GitHub repositories for "hybrid intelligence taxonomy" and "hybrid intelligence human-ai" returned only unrelated third-party projects. The GitHub organisation vencortex (an author's affiliation) has no public repositories. The user ddellermann (matching an author's name) has one public repository, gift-eval, which is unrelated to this paper and was not used.

Pitfalls

• Not evaluated. The taxonomy is not tested on any system, and the paper says that evaluating its usefulness is still to come §6. An implementer should treat it as a checklist of design questions, not a validated method.
• Narrow, dated evidence base. The review covers April to June 2018 §3.2. The authors also say their case selection is biased toward decision-problem contexts §6. Papers after mid-2018 are not covered.
• Company sources. Several of the seven empirical cases are cited to company websites (for example Cindicator, vencortex, Stitch Fix) rather than papers, with access dates of 2018-06-14 tab. 1 [refs. 26–32]. These pages may have changed.
• Stopping conditions are stated but not reported as met. The paper lists objective and subjective conditions §3.1 and says three iterations were done "so far" §3.2, but it does not show a final check that every condition holds. The loop in the Method section is the authors' procedure as described, not a verified stop rule.
• "Superior results" has no test. The authors define it as a result no one actor could reach alone §2.3, but they give no way to measure it. Designers must decide for themselves what counts.
• Category counts check out. The 16 sub-dimensions and 50 categories in §6 match what is shown in fig. 3 when counted by hand, which is the only check available. The figure is small, so count again before relying on exact labels.
• Two directions of interaction. The paper uses "human-AI interaction" for how the human teaches the machine and "AI-human interaction" for how the machine gives feedback and queries the human §4.3 §4.4. The two names are easy to swap, so check the direction before using a category.
• Figure 1 shows AI, ML and human-in-the-loop ML as nested sets fig. 1. It is not a method and has no data.

Lineage

• Builds on: the taxonomy development method of Nickerson et al. §3.1 [ref. 13]. The paper adopts its objective and subjective conditions §3.1.
• Compared against: none; the paper positions itself against decision-support and expert-system research §2.3 and against knowledge-base systems §3.2.
• Superseded by: not known from content in this wiki.