Real incidents. Real costs. The case for AI observability.
Every major AI failure in regulated industries shares a root cause: no one could see what the model was doing until after the damage was done.
These case studies show what unobserved AI costs organizations in fines, legal liability, and reputational damage — and how Prism closes each gap.
When AI Goes Unobserved
These incidents are not hypothetical. They happened — at large, well-resourced organizations — because no one could see what the AI was doing until it was too late.
Air Canada
AI Chatbot Hallucination (2024)
Ordered by tribunal to pay compensation after chatbot fabricated a bereavement discount policy that never existed. Air Canada argued the bot was a 'separate legal entity' — rejected.
- Chatbot invented a refund policy not found anywhere in Air Canada's rules
- Customer purchased a full-price ticket relying on the bot's false guarantee
- British Columbia Civil Resolution Tribunal ruled Air Canada liable
- Set a precedent: companies are legally responsible for what their AI says
Guardrail enforcement would have blocked the fabricated policy at generation time. The full response trace would have surfaced the hallucination before it reached the customer.
Samsung
Confidential Data Leak via AI (2023)
Three separate incidents in 20 days: proprietary semiconductor source code, internal meeting notes, and test sequences were pasted into ChatGPT by employees. Samsung banned generative AI tools company-wide.
- Engineers pasted chip design source code into ChatGPT to fix bugs
- Meeting transcripts containing unreleased product details were uploaded
- Once sent, OpenAI can use submissions as training data — data unrecoverable
- Samsung issued emergency policy banning all external AI tools
PII and sensitive-data redaction at ingestion would have stripped confidential code and IP before it left the enterprise boundary — no policy ban needed.
HMRC
AI Tax Chatbot Gave Wrong Advice (2024)
UK tax authority chatbot gave incorrect guidance on self-assessment deadlines, child benefit, and PAYE to thousands of users. HMRC had to disable features and issue public corrections.
- Chatbot told users incorrect self-assessment filing rules
- Some users filed taxes based on wrong AI advice — risking penalties
- Errors were systematic, not one-off, suggesting no monitoring in place
- HMRC spent weeks identifying affected users and issuing corrections
Continuous trace monitoring would have caught the systematic error pattern within hours. A full audit trail would have shown exactly which users received incorrect guidance, cutting the correction window from weeks to minutes.
DPD
AI Chatbot Went Off-Script (2024)
DPD's customer service AI started swearing, criticising the company, and writing a poem about how bad DPD is — all to a real customer. The exchange went viral and made global headlines.
- Bot ignored its intended role and began insulting its own company
- Customer shared the full conversation publicly; story reached BBC, CNN, Guardian
- DPD disabled the AI chatbot within hours of the story breaking
- Estimated brand damage: millions in earned negative media coverage
Real-time guardrail enforcement for off-topic, toxic, and brand-unsafe outputs would have blocked the response before delivery. Every blocked response would have been logged for review.
The Compliance Context
Why regulators, risk teams, and CISOs are making AI observability a priority in 2025
Why AI Observability Is Now a Compliance Requirement
The EU AI Act and emerging US guidance both require organizations to maintain records of high-risk AI system behavior. Observability is no longer optional for regulated teams.
Learn More →PII in AI Pipelines: The Hidden Compliance Risk
Most organizations focus on securing data at rest. AI pipelines create a new exposure surface: sensitive data flows through prompts and responses that are rarely audited for PII before storage.
Learn More →Model Risk Management in the Age of Generative AI
SR 11-7 was written for predictive models. Generative AI introduces hallucination, prompt injection, and agent drift — risks that traditional MRM frameworks were not designed to catch.
Learn More →Tools and Frameworks
Essential resources and tools for implementing effective AI governance
AI Governance Checklist
Comprehensive guides including NIST AI Risk Management Framework, FINOS AI Readiness Framework, and Holistic AI's Governance Guide.
Download ResourceBias Detection Guide
Complete toolkit featuring Optiblack's 7-Step AI Bias Audit Guide, Holistic AI's Resources, and NIST's Bias Management Guide.
Download ResourceStart tracing in 5 minutes
One SDK. Five minutes. Full audit trails, PII redaction, and guardrail enforcement, from day one.