When you ask a general AI chatbot to fill a security questionnaire or draft a SOC 2 control narrative, it writes back in seconds. It sounds confident. It may be entirely fabricated.
The "Just Fill It In" Failure Mode
General AI is built for fluent conversation, not grounded trust work. Given a compliance task — a SIG question about access control, an ISO 27001 policy mapping, a PCI DSS evidence request - it will produce output that reads correctly. The problem is that fluency and accuracy are not the same thing.
A chatbot has no access to your workspace. It cannot read your policies, check your evidence artifacts, or verify whether a control was implemented. The output is patterned from training data, not from what you have actually done. Sometimes the pattern fits. Sometimes it invents.
This failure mode has produced over 1,300 documented instances of AI-hallucinated legal citations in US courts as of April 2026. In compliance, the same mechanism creates fictitious control IDs, fabricated policy clauses, and claims about evidence never collected. The output looks plausible. Only someone reviewing each line against the actual evidence can tell the difference — which is exactly the work AI was supposed to reduce.
| AI output | What actually happened |
|---|---|
| Cites a SOC 2 control by ID | The ID belongs to a different framework version |
| Describes a policy that "covers access reviews" | The policy exists but says nothing about access reviews |
| Lists three evidence artifacts supporting a claim | Two of the three were never collected |
| Provides a compliance date | The deadline was extended by regulation — AI did not know |
The problem is not that the tool makes mistakes. It is that the tool does not know it made one, and it delivers the fabricated answer with the same confidence as a correct one.
Why a Visible Gap Is More Valuable Than a Plausible Answer
In trust work, a known gap and a hidden fabrication are fundamentally different things.
A known gap is actionable. You see what is missing. You decide whether to add the evidence, adjust the answer, or accept the gap. The review conversation is about substance.
A hidden fabrication looks like complete work. The reviewer sees an answer that reads correctly and approves it — not knowing the control ID is wrong, the policy citation is fictional, or the evidence claim is false. That answer enters the record. If an auditor or customer relies on it, the problem surfaces downstream, when fixing it costs more.
The courtroom record makes this concrete. Since 2023, lawyers have been fined, suspended, and fired for submitting AI-hallucinated citations. The problem was never that they asked an AI for help. It was that the AI delivered confident-sounding output, and the lawyers — exactly like a compliance reviewer facing a stack of questionnaire answers — had no efficient way to tell which parts were real. Judge Brantley Starr of the Northern District of Texas banned unchecked AI-generated filings outright, noting that these platforms "make stuff up — even quotes and citations."
In Canada, 167 AI-hallucinated filings across 51 courts and tribunals have been documented. Chief Justice Richard Wagner observed that AI is present "in ways both promising and problematic."
Compliance faces the same structural risk. A confident fabrication that enters the audit record is worse than an incomplete answer everyone can see.
How "Not Provided / No Source Found" Changes Review
A compliance harness is built from a different starting point than a chatbot. Instead of optimizing for fluency, it optimizes for verifiability. The most visible consequence is what happens when the tool cannot support an answer.
When Compass reads a questionnaire or drafting request against the user's workspace, every claim it makes traces to a source document: a policy, an evidence artifact, a control definition, a risk register entry. If the workspace contains nothing that can support the answer, Compass marks the response "Not Provided / no source found" and presents it as a visible gap.
This changes the review workflow in two concrete ways:
-
Reviewers read gaps, not guesswork. The output shows exactly what is supported and what is not. Instead of wondering "did the AI make that up?", the reviewer asks "should we add evidence for this, or is the gap acceptable?" The decision is conscious, not accidental.
-
Gaps become the agenda. A review session starts with the list of unresolved items, not a line-by-line hunt through plausible-looking claims. Teams spend review time on decisions that matter instead of verification that should not have been necessary.
| Workflow stage | With a general chatbot | With a compliance harness |
|---|---|---|
| AI produces output | Confident answer, may be fabricated | Cited answer + visible gaps |
| Reviewer's first question | Is any of this made up? | What do we do about this gap? |
| Audit risk | Undetected fabrication enters record | Every gap is known and accounted for |
| Time spent on verification | High — must cross-check every claim | Low — focus only on flagged gaps |
The "Not Provided" behavior is not a limitation. It is the architectural feature that makes the tool safe for trust work.
Seeding Gaps to Test Any Compliance Tool
The honest-gap behavior is testable. You can evaluate any tool with a simple exercise:
- Isolate a document that contains a specific policy or control and confirm it answers a known question.
- Ask a second question that the document cannot answer — something outside its scope.
- Observe what the tool produces.
A general chatbot will typically produce a plausible answer anyway, drawing from its training data. It does not know the answer is unsupported. A compliance harness should surface the gap — marking the answer as unsupported, linked to no source, requiring a human decision.
The test reveals the tool's architecture. A tool that never admits uncertainty is optimized for fluency. A tool that visibly flags its limits is optimized for verifiability. For trust work, the second is the relevant category.
Compass Close
When a compliance tool cannot support an answer, the honest response is to say so and leave the gap visible for the reviewer. That is what Compass does. Its "Not Provided / no source found" behavior surfaces unsupported answers instead of papering them over, so every review session starts from what is actually known. Compass flags the gap; closing it remains your work. See it in action at Compass by Truvara. Watch Compass work.
FAQ
Why does Compass not just guess like other tools when it cannot find a source?
Guessing risks fabricating a claim that enters the audit record as truth. For trust work, that outcome is worse than leaving a gap visible. The "Not Provided" behavior is a deliberate design choice — one that prioritizes verifiability over fluency. Read more about how this changes the review workflow in the What One Connected Record Changes section above.
Will "Not Provided" answers make my auditor suspicious?
Auditors evaluate what is supported, not what is confidently stated. A visible gap you can explain or address is less concerning than a claim that traces to nothing. Many compliance teams use the "Not Provided" list as their review checklist — a starting point for decisions, not a red flag.
Can I turn off the "Not Provided" behavior?
The behavior is built into Compass's core architecture: it never silently fabricates. There is no "just guess" mode because the tool is designed to produce cited, reviewable work. If you prefer an approach that fills gaps aggressively, a general chatbot might suit that need — but the output requires thorough verification.
Does the "Not Provided" behavior work with any AI model?
Yes. The grounding and honest-gap behavior are part of the harness, not the model. Whether you bring your own key for OpenAI or Anthropic, or run a local model through Ollama or LM Studio, the harness enforces citations and surfaces unsupported answers — it does not depend on the model's own honesty features.
How do I close the gaps that Compass flags?
Add the missing evidence to your workspace — a policy document, a control narrative, an evidence artifact — and re-run. The gap resolves when Compass can cite a source. The review process remains the same: propose, review, accept or reject.