wikillm

Paper · 2026-10-05 · cs.AI

Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving

HEAR is a bidirectional protocol between an agent harness and an LLM inference engine: the harness sends workflow intent and requirements, the engine returns cache, queue and outcome state. Two policies instantiate it: cache-aware scheduling with a waiting guard (1.61x batch speedup on SCBench) and per-role inference-mode selection (1.23x and 2.45x end-to-end). Author-reported.

LLM servingKV-cache managementLLM agentagentsservingkv-cacheschedulinginference

arXivPDFMarkdown14 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

An LLM agent does its work in many steps: it plans, calls tools, and sends many requests to a model. Two pieces of software run this. The harness is the agent's own logic: it knows which steps depend on which, what context each step needs, and what the goal is. The inference engine is the model server: it knows which requests are waiting, what text it already holds in memory, and how busy the hardware is. Each side sees part of the picture, and today they rarely talk to each other in a structured way.

The paper proposes HEAR, a shared set of messages for the two sides to exchange in both directions. The harness says what it plans and what it requires. The engine says what it currently holds and can do, and reports what happened to each request. HEAR also says how to read a message: a hint is not a command, a preference is not a requirement, a status report is not a reservation, and a request that was accepted is not yet finished. The protocol does not decide policy. Separate policies decide what to send and what to do with it.

The authors test two policies. One reorders waiting requests so that those whose earlier text is still in cache run first, while a request that has waited too long can no longer be overtaken. The other picks a serving mode for each kind of agent role. Both gave speed-ups in their own tests, and the gains depended on the workload, so they should not be read as general wins.

TL;DR

  • HEAR is a message contract linking the agent harness with the inference engine. It fixes what each exchange means and how to handle it, and does not specify a wire format.
  • Cache-aware scheduling with a waiting guard gives a 1.61x batch speedup on the SCBench workload (author-reported).
  • Per-role inference-mode selection gives 1.23x on BrowseComp-Plus and 2.45x on DeepResearchBench end-to-end, with no observed drop in task quality (author-reported).
  • No official code could be read, so the implementation details are only what the paper states.

Problem

Agent workflows issue chains of dependent model calls with growing context §1. Engine-side decisions, such as which cached prefix to keep, need workflow knowledge the engine does not have. Harness-side decisions, such as which ready request to send first, need cache residency and queue state the harness does not see §1. The authors argue that existing systems define their own cross-layer signals, so each optimisation is tied to one harness and one engine §1. The problem is stated for memory-constrained, concurrent serving §1, §4.1.

Method

The harness and the engine keep their roles. The harness handles workflow semantics; the engine handles model execution, queues, KV caches and serving resources §3.1. HEAR governs how workflow-selected requests are served. It does not change which tasks run, which dependencies hold, or the model §3.1.

Message categories §3.2 tab. 1. There are four semantic categories, not packet formats §3.2:

  1. Harness to engine, Execution Description and Intent: roles, readiness, workload shape, context lifecycle and expected reuse §3.2.
  2. Harness to engine, Execution Requirements and Control: scheduling preferences, waiting-time constraints, cache operations and execution configurations §3.2.
  3. Engine to harness, State and Capabilities: what sits in the cache, the queue and load picture, which operations the engine can perform, and which configurations it offers §3.2.
  4. Engine to harness, Execution Outcomes: status, realised configuration, cache reuse, cost or failure §3.2.

One exchange may carry several categories. An engine may expose only the subset it supports, and capability reports let the harness find that subset §3.2.

Interaction semantics §3.3 tab. 2. Four distinctions set how a message must be treated:

  • Intent versus control: a description or prediction is context. Only an explicit control asks the receiver to act §3.3.
  • Preference versus requirement: a preference is best effort. A requirement must hold, or the request is rejected or falls back to an explicitly allowed alternative §3.3.
  • Observation versus guarantee: a state report reserves nothing §3.3.
  • Acceptance versus completion: an accepted operation may still fail or be unsupported. An outcome must distinguish accepted, completed, rejected, unsupported and failed §3.3.

Joint-information policy §3.4 eq. 1. A policy maps the harness view and the latest engine update to a decision. The engine update may be older than the decision point, because the two layers evolve independently §3.4:

d_i = \pi(h_i, s_j) eq. 1

Without engine feedback this reduces to a harness-only decision. Without workflow information, the engine only sees request-local state §3.4.

Cache-aware runtime coordination §4.2 eq. 2 eq. 3 app. A.2. At each decision point the harness gives the ready requests, their contexts and the waiting constraints. The engine gives its latest cache availability and queue state eq. 2. The policy returns priorities for the ready requests and the requirements that still apply eq. 3:

h_i = (\mathcal{R}_i, \mathcal{C}_i, \mathcal{U}_i), \qquad s_j = (K_j, Q_j) eq. 2

d_i = \pi_{\mathrm{sched}}(h_i, s_j) = (\bm{p}_i, \mathcal{U}_i) eq. 3

The priorities are preferences. The waiting guard is a requirement. The engine applies each preference within the requirements §4.2.

The three ordering policies are:

  • FCFS, the default vLLM scheduler §4.2.
  • Cache-Aware: ordering by the harness's view of reusable prefix blocks §4.2 app. A.2.
  • Cache-Aware + Guard-W: as above, but a request that has waited W seconds can no longer be bypassed by cache-hot requests §4.2 app. A.2. W is 40 s or 60 s on SCBench and 40 s on Mooncake §4.2.

The pseudocode below follows app. A.2. It is written from the paper's text, not from code.

Pseudocode
# Cache-aware priority for one ready request  [app. A.2] [§4.2]
# Hot: reusable prefix covers >= 50% of the prompt. Smaller value runs first.
def cache_priority(request):
    # Prefix matching runs from the start and stops at the first missing block
    if request.reusable_tokens < 0.5 * request.prompt_tokens:
        return DEFAULT_PRIORITY
    # Hot requests run before cold ones, longer reusable prefix first
    return -request.reusable_blocks
Pseudocode
# Waiting guard: after W seconds, a request is protected from bypass  [§4.2]
# Protected requests keep their waiting order among themselves  [app. A.2]
def effective_priority(request, now, guard_wait):
    # Assumed start of the wait: queue entry time (not stated in the paper)
    if now - request.queued_at > guard_wait:
        # Must sort before every hot value; the value itself is not stated
        return PROTECTED_PRIORITY
    return cache_priority(request)

Session-aware KV retention app. A.2. This policy acts only on eviction order, not on scheduling. The harness estimates the chance p that a session returns, from completed conversations only. After each request the engine computes a retention score from p and the idle time since the context's last request app. A.2:

r = p \cdot e^{-\mathrm{idle}/400} app. A.2

Likely returners and recently used contexts score higher, and the score decays with idle time app. A.2. The paper does not give the unit of idle time.

Pseudocode
# Session-aware retention score, computed after each request  [app. A.2]
# return_probability: harness estimate p from completed conversations
# idle_time: time since the context's last request (unit not stated)
def retention_score(return_probability, idle_time):
    # Higher score means the context is kept longer
    return return_probability * exp(-idle_time / 400)

Capability-aware execution configuration §4.3 eq. 4 eq. 5 app. A.1. This is an engine-side selector for an agent role. Its inputs are the role, the stage, the workload shape, the objective and quality requirements from the harness, plus measured latency, throughput, KV capacity and stability for each mode from the engine §4.3 app. A.1. The candidate space is two sets, one for the Main agent and one for the Reader or sub-agent eq. 4:

\mathcal{C} = \{\mathrm{vLLM}, \mathrm{OmniKV}\} \times \{\mathrm{vLLM}, \mathrm{H_2O}, \mathrm{SnapKV}\} eq. 4

The selector picks the candidate with the lowest makespan among those that pass the development quality and stability checks §4.3 eq. 5:

d_i^{\mathrm{config}} = \pi_{\mathrm{E}}(h_i, s_j), \quad d^{*}_{\mathrm{BCP}} = (\mathrm{vLLM}, \mathrm{H_2O}), \quad d^{*}_{\mathrm{DRB}} = (\mathrm{OmniKV}, \mathrm{H_2O}) eq. 5

Calibration ranks valid candidates by development makespan app. A.1 eq. 6:

\widetilde{d} = \arg\min_{d \in \mathcal{C}_{\mathrm{valid}}} T_{\mathrm{dev}}(d) eq. 6

A verifier checks the top candidate against logs and quality and stability summaries. It may pick another valid candidate only for an invalid measurement, an unsupported path, or a feasibility or reliability issue that makespan does not show app. A.1. The accepted mapping is written to a versioned file and loaded once at workflow start. The online path is a constant-size lookup per request app. A.1, app. A.3.

Pseudocode
# Offline choice of the role-to-mode mapping  [app. A.1] [eq. 4] [eq. 6]
# candidates: the (Main, Reader) mode pairs from eq. 4
def select_mapping(candidates, dev_makespan, passes_checks):
    # Drop incomplete runs and runs failing development quality or stability
    valid = [c for c in candidates if passes_checks(c)]
    # Rank by development makespan only; formal results are not used [app. A.1]
    return min(valid, key=dev_makespan)

Hyperparameters

NameValueWhereNotes
Hot thresholdreusable prefix >= 50% of promptapp. A.2counted in tokens; cold requests keep default priority
Hot priority-N_reuse (reusable KV blocks)app. A.2smaller runs first; KV block size not stated
Guard threshold W40 s and 60 s (SCBench); 40 s (Mooncake)§4.2 app. A.2start of wait not defined
Protected prioritynot statedapp. A.2value not given; inferred to sort before all hot values
Session retention scorer = p * exp(-idle/400)app. A.2idle unit not stated; p from completed conversations only
SCBench arrivalsPoisson process, rate not statedapp. A.2closed-loop rebuild; each sample is an independent session
SCBench think timeN(5, 1) sapp. A.2handling of negative samples not stated
Mooncake load levels50%, 75%, 100% of sessions with first request in a 15-minute windowapp. A.2original timing kept
SCBench model and hardwareQwen3-8B; one RTX PRO 6000; 96-GiB L2 KV cache§4.2 app. A.2"L2" not defined in the paper
Mooncake model and hardwareQwen3-8B; one H100; 96-GiB L2 KV cache§4.2 app. A.2
Role-configuration modelGLM-4.7-Flash; Main and Reader on separate H100 GPUs§4.3 app. A.4
Engine versionsvLLM 0.26.0; SparseEngine rev. 67174b1b (OmniKV, BCP), 469aa791 (sparse modes, DRB)app. A.4patched source archived with the artifact, not in the paper
BCP concurrencyat most 4 Main and 12 Reader requests in flightapp. A.4
BCP samplingtemperature 0; top-p 1; thinking disabledapp. A.4
DRB Main samplingtemperature 0.2; top-p 0.9; 4,096-token output limit; thinking onapp. A.4
DRB sub-agent samplingtemperature 0.2; top-p 0.9; 512-token output limit; thinking offapp. A.4
SnapKV budget4,672 tokens per layer; prefill not sparsifiedapp. A.4long requests hold full prompt memory until prefill ends
DRB SnapKV prefill gateFIFO; at most 456,837 prefill tokens in flight; released at first output tokenapp. A.4harness-side control, separate from SnapKV settings
Sample sizesBCP: 208 questions per arm; DRB: 100 instances per arm, 89 valid in all arms for paired tests§4.3 app. B.4
JudgesBCP: Qwen3-32B, 4,096-token output limit; DRB: RACE evaluator with gpt-5.4-miniapp. B.4
Selector objectiveminimum development makespan among valid candidatesapp. A.1 eq. 6quality and stability thresholds not stated
Repetitions per armnot statedsingle reported value per arm in the tables read

Results

Author-reported. Setup: one-GPU serving with Qwen3-8B for the cache workloads §4.2; GLM-4.7-Flash across two H100 GPUs for the role workloads §4.3. Baselines: the default vLLM scheduler (FCFS) for cache, and all-vLLM for roles §4.2, §4.3. Every arm runs on the same replayed trace or the same inputs, with each arm starting from an independent serving state, and failures and retries included §4.1.

SCBench, closed-loop multi-turn workload tab. 3. Times in seconds; reuse is the share of follow-up prompt tokens served from cache, read as percent in the text tab. 3 §4.2.

ConfigurationTTFT P50TTFT P95TTFT MAXBatch completionReuse (%)
FCFS63.174.279.048119.2
Cache-Aware4.675.893.922487.6
Cache-Aware + Guard-4028.351.165.029966.1
Cache-Aware + Guard-604.564.080.423084.7

The paper's headline numbers for Guard-40 against FCFS are a 1.61x batch speedup, and P50, P95 and MAX TTFT ratios of 2.23x, 1.45x and 1.22x tab. 3 §4.2 (these match the table).

Mooncake Bench, production-derived trace tab. 4. Caption: session is total response-waiting time per conversation; throughput in turns per minute. The TTFT unit is not stated in the table. Reuse is read as percent in the text §4.2.

LoadConfigurationTTFT P50TTFT P95TTFT MAXSessionThroughputReuse (%)
50%FCFS0.72.86.213.786.119.5
50%Cache-Aware0.62.45.913.386.020.2
50%Cache-Aware + Guard0.62.87.413.686.219.6
50%Session-Aware0.62.56.313.086.631.9
50%Session-Aware + Cache-Aware + Guard0.52.36.112.786.832.1
75%FCFS10.951.956.750.2116.48.0
75%Cache-Aware8.742.949.642.6119.210.1
75%Cache-Aware + Guard9.142.347.942.5119.38.1
75%Session-Aware10.539.045.143.2119.516.1
75%Session-Aware + Cache-Aware + Guard8.239.845.439.8121.016.9
100%FCFS163.9277.0281.5256.9122.75.2
100%Cache-Aware86.8209.9220.9156.1128.310.0
100%Cache-Aware + Guard97.1209.2217.6178.7122.88.0
100%Session-Aware93.0199.2207.8171.8124.612.7
100%Session-Aware + Cache-Aware + Guard161.7267.4277.7243.2108.912.5

Figure 3 shows KV reuse and GPU KV occupancy across these loads fig. 3 §4.2.

Role-specific configuration, BrowseComp-Plus (BCP) and DeepResearchBench (DRB) tab. 5. Wall time and makespan in hours; P95 end-to-end latency in minutes; accuracy and RACE in percent. Ratios relative to vLLM-vLLM are in the paper's table. The bold frozen mappings are vLLM-H2O for BCP and OmniKV-H2O for DRB tab. 5 eq. 5.

MainReaderBCP wall (h)BCP P95 (min)BCP acc. (%)DRB makespan (h)DRB P95 (min)DRB RACE (%)
vLLMvLLM5.6940.046.155.9339.840.6
OmniKVvLLM5.3740.347.126.1739.941.1
vLLMH2O4.6239.646.632.7615.640.5
OmniKVH2O4.9837.243.272.4215.841.1
vLLMSnapKV4.8340.444.232.8517.740.0
OmniKVSnapKV5.2940.643.752.7618.340.5

The paper reports speedups of 1.23x (BCP, vLLM-H2O) and 2.45x (DRB, OmniKV-H2O) against vLLM-vLLM, with accuracy 46.63 vs 46.15 and RACE 41.1 vs 40.6 tab. 5 §4.3.

Role-level attribution tab. 6. Latency in seconds per request, one role changed at a time tab. 6 §4.3.

BenchmarkMain output tokens per requestMain latency changeReader or sub-agent latency changeCache evidence
BCPabout 0.4KOmniKV to vLLM: 17.0 to 14.9 (1.14x)vLLM to H2O: 53.4 to 9.5 (reported 5.60x)prefill down 76.2%; reuse 6.2% to 80.3%
DRBabout 4.4KvLLM to OmniKV: 69.4 to 64.2 (1.08x)vLLM to H2O: 421.5 to 58.4 (7.22x)chain reuse 49.8%

Protocol cost tab. 7. The protocol adds control-plane work only (residency updates, prefix lookups, a constant-size mapping lookup, outcome reports). Those costs are included in the formal end-to-end times. The table lists the operations but gives no measured overhead tab. 7 app. A.3.

Paper vs code

No official code could be read. The title-page repository link (anonymised review mirror, anonymous.4open.science/r/hear-3D1E) returned an empty client-rendered page, so no file, license or commit was checked. GitHub repository searches for "HEAR harness engine", "hear-3D1E", "HEAR agentic LLM serving", "harness engine protocol agentic serving" and "HEAR harness-engine pairing" returned no matching repository. No code was cloned. This card therefore records only what the paper states; nothing in it has been compared with an implementation.

Pitfalls

  • Headline numbers belong to specific settings. The 1.61x batch speedup is for Cache-Aware + Guard-40 on SCBench. Cache-Aware without the guard reaches 2.15x batch completion, but its P95 (0.98x) and MAX TTFT (0.84x) are worse than FCFS tab. 3. The guard trades batch speed for tail latency.
  • Large unexplained P50 gap. P50 TTFT is 28.3 s at W=40 and 4.5 s at W=60 on SCBench tab. 3. The paper does not explain this step; it may depend on how the guard interacts with queue state. Test both thresholds.
  • Combined policy can hurt under overload. At 100% Mooncake load, Session-Aware + Cache-Aware + Guard has the lowest throughput of all policies there (108.9 against 122.7 for FCFS, 0.89x), while its latency gains over FCFS are small (P50 161.7 against 163.9) tab. 4. The paper says future-reuse intent must be conditioned on live queue state. No policy wins at every load §4.2.
  • Workloads are rebuilt, and some parameters are missing. SCBench has no timestamps, so the authors add Poisson session arrivals and N(5, 1) s think times app. A.2. The arrival rate and how negative think-time samples are handled are not stated. Mooncake is a production-derived trace, but it was replayed at three load levels §4.2.
  • Several cache definitions are missing. The KV block size, the block-hash scheme, and the meaning of "L2 KV cache" (96 GiB) are not stated. The harness counts reusable tokens in GPU and CPU tiers separately app. A.2, so the cut-off at 50% of the prompt has to be computed in tokens, while the priority uses block counts.
  • Waiting guard start and protected priority are undefined. The paper does not say when the wait starts or what protected priority value sorts it ahead of hot requests. The pseudocode assumes queue entry time.
  • Retention formula units are missing. The exponent divides idle time by 400, but the unit of idle time is not stated. The estimate p comes from a harness summary whose method is not specified app. A.2.
  • The protocol has no message schema. The paper defines categories and semantics, not packet formats or fields §3.2. Table 2 gives representative contents only tab. 2. Field names, versioning and serialization are left to the implementer.
  • Acceptance is not completion. A preference may be ignored, and an accepted control may fail later. Code must read the outcome message before depending on residency or prefetch §3.3. The paper's cache view is an observation, not a reservation; the engine rechecks reusable content when a request runs app. A.2.
  • Selector is not fully mechanical. The quality and stability thresholds are not stated. A verifier may override the lowest-makespan candidate for reasons the paper does not define app. A.1. The frozen mapping therefore depends on a human-applied check.
  • Accuracy gaps are about one question. BCP has 208 questions. The 0.48-point accuracy difference (46.63 vs 46.15) is one question. The paper calls the result "without observed degradation" §4.3. Some other accuracy drops in the six-arm table are larger, e.g. OmniKV-H2O at 43.27 tab. 5.
  • Judges and runs differ in environment. BCP arms were judged on two different GPUs (RTX PRO 6000 and H100), which the authors acknowledge app. B.4. DRB uses an external judge model, gpt-5.4-mini app. B.4. The number of repeated runs per arm is not stated; each arm has one reported value in the tables read.
  • SnapKV has a hidden admission cost. SnapKV does not sparsify prefill, so a long request holds memory for its full prompt until prefill ends. DRB needed a FIFO gate with a global limit of 456,837 prefill tokens in flight app. A.4. The harness must implement this gate to reproduce the reported SnapKV configuration.
  • Speedup ratios and rounded times. The role-level BCP ratio is reported as 5.60x, but 53.4 / 9.5 = 5.62. The difference is most likely rounding in the underlying measurements tab. 6.
  • Cost not measured. The paper claims protocol overhead is control-plane only app. A.3, but it gives no latency or CPU numbers for it.
  • Security and ethics. Because the protocol exchanges execution and resource metadata, the paper advises that real systems add access checks, separate tenants, and cap how long requests may wait §5, ethics statement.
  • No limitations section. The source contains a commented-out placeholder; the published conclusions have no limitations section §5.

Lineage

  • Builds on: the vLLM default scheduler (FCFS baseline) and the execution modes OmniKV, H2O and SnapKV, used as provided by the authors §4.2, §4.3.
  • Related to: KVFlow (inspiration for cache-aware ordering, §4.2); PBKV and Pythia, whose predictors and optimisation rules HEAR integrates unchanged through adapters §3.4; Conductor and Puppeteer, which orchestrate from application state alone §3.4.
  • Compared against: FCFS for cache coordination; all-vLLM for role configuration §4.2, §4.3.
  • Superseded by: not known from content in this wiki.

Source and license

Adapted from “Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving” by Jiaqi Zhao, Haodong Chen, Jitai Hao, Wei Zhao, Jinghao Pang, Qiang Huang, Jun Yu (arXiv:2610.06597), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.