Schema Ghost

Detector key: schema_ghost · Source: context · Severity: high · Suspicion: 82

When a tool returns nothing useful, a well-behaved agent should say so. Instead, LLMs often fill in the blank — the tool returns null or an error, and the next model step confidently produces specific-looking data it was never given. That’s a schema ghost: fabricated output haunting a hole in the data.

What trips it

The detector looks for this exact sequence:

1. A tool step whose output is empty or an error — null, nil, none, {}, [], undefined, "", or a failed status.
2. Followed (before any other tool call) by a model step that produces specific output — long enough and containing digits, JSON, or structured punctuation, i.e. the hallmarks of concrete data.

A subsequent tool call between the two breaks the pattern — that would mean the agent went and fetched the data properly, so nothing fires.

Score

Fixed at 82 — high confidence, because the “empty in, specific out” shape is a strong tell.

Example

trips schema_ghost

{
  "steps": [
    { "stepKind": "tool", "stepName": "get_order", "outputSummary": "null" },
    { "stepKind": "model", "outputSummary": "Order #8842 shipped on 2026-06-28 via UPS, tracking 1Z999AA10123456784." }
  ]
}

Lumni reports:

Schema ghost: model fabricated data after “get_order” returned nothing — Tool “get_order” returned an empty/null/error result (“null”), yet the next model step produced specific, detailed output. The agent is likely hallucinating data the tool never provided.

Why it matters

Hallucinated order numbers, tracking codes, balances, and confirmation IDs are worse than an error message — they look authoritative and users act on them. This detector catches the moment the model papered over missing data.