Pillar
What an AI audit is, what it covers, and how to produce evidence regulators and external auditors actually accept.
In short
An AI audit is a structured review of an AI system's risk, fairness, robustness, and compliance posture. It is the AI-era extension of model risk management — applied to LLMs, agents, and traditional ML models alike — and it is increasingly required by regulators (SR 11-7, NAIC, NIST AI RMF, ISO 42001) and external auditors (PCAOB AS 2201 for financial-reporting AI).
Why it matters
AI auditors look for three things: did the development team document the model's intent and limitations, did they test it for fairness and robustness, and is there ongoing evidence the model still performs as expected. Without a working audit trail, the answer to all three becomes "trust us" — and that does not pass an external review.
The shift from traditional model audits to AI audits is mostly a shift in surface area. LLMs and agents introduce new failure modes (hallucination, prompt injection, drift across silent vendor model updates) that legacy MRM frameworks were not built to test. Modern AI auditing combines pre-deployment testing (model audits, red teaming) with continuous post-deployment evidence (traces, evaluations, trajectories).
What's covered
A complete AI audit typically reviews data quality, model fairness across protected classes, calibration and accuracy, robustness to adversarial inputs, security posture, and ongoing monitoring. For LLMs and agents, it also covers guardrail effectiveness, hallucination rates, agent decision traceability, and third-party model dependencies.
First-line audits are run by the model development team. Second-line audits are run by an independent model-risk function (often the requirement under SR 11-7's "effective challenge" pillar). Third-line audits are run by internal audit or external auditors. Different stakeholders need different evidence; Prism produces all three.
Pre-deployment audits run benchmarks, fairness tests, and adversarial probes before a model ships. Ongoing audits look at production behaviour: quality scores over time, drift, incidents, and governance maturity. Both are needed; either alone is insufficient.
External auditors expect dated, immutable artifacts: audit run reports, fairness metrics per protected class, red-team findings with severity, prompt and model change logs, monitoring thresholds and alerts, and incident response records. Prism exports all of these per fiscal period.
SR 11-7 model risk management has been applied to AI by the Federal Reserve since 2024. NIST AI RMF MEASURE function is built around continuous evaluation. ISO 42001's Clause 9 is a performance-evaluation cycle. EU AI Act Article 15 covers accuracy, robustness, and cybersecurity testing. NAIC's Model Bulletin expects insurers to maintain a documented AI audit program.
How Prism helps
Bias, calibration, robustness, and accuracy metrics archived per audit run, comparable across versions.
Learn moreCurated jailbreaks, prompt-injection variants, and policy-bypass tests with severity tagging.
Learn moreLLM-as-Judge scoring on every trace; week-over-week regression view.
Learn moreStep-by-step lineage of every agent decision for auditor walkthrough.
Learn moreImmutable PDF and JSON exports signed and timestamped per fiscal period.
Learn moreRelated
Model Risk Management for AI — SR 11-7 Revised Guidance
The interagency Revised Guidance on Model Risk Management supersedes SR 11-7 and SR 21-8. The three-pillar discipline carries forward, scaled to each bank's model risk profile. Prism produces the evidence at every tier.
NIST AI Risk Management Framework: Prism Compliance Mapping
Each NIST AI RMF function has subcategories that demand evidence. Prism produces it: from MEASURE-2.7 trace logs to MANAGE-2.1 adversarial test results.
ISO/IEC 42001 AI Management System
ISO 42001 is the first auditable management-system standard for AI. Prism produces the operational evidence each clause asks an internal auditor to see.
EU AI Act Compliance
The EU AI Act's high-risk category includes credit, employment, and insurance scoring. Prism is built to satisfy the logging, transparency, and oversight articles.
NAIC AI Model Governance
The NAIC Model Bulletin sets expectations for insurer use of AI. Prism produces the documentation, monitoring, and audit artifacts each pillar requires.
AI compliance and risk management for banks
From underwriting copilots to fraud-screening agents, banks need the same model risk discipline they have for traditional models. Prism is built around it.
AI governance for insurance
Insurers face NAIC's Model Bulletin and a patchwork of state-level AI rules. Prism gives carriers one platform to produce the evidence each one demands.
AI compliance for asset managers
From research copilots to trading-signal models, asset managers face SEC, FINRA, and SR 11-7 expectations on AI. Prism produces the evidence per system.
AI Red Teaming
Glossary entry
Model Drift
Glossary entry
LLM-as-Judge
Glossary entry
AI Model Audits
Model audits give you a structured review of model behavior, risk profile, and readiness for production, before deployment, not after incidents.
AI Red Teaming
Structured adversarial testing to find prompt injection vulnerabilities, guardrail bypasses, and unsafe behaviors, before they reach production.
LLM Evaluations
Define quality rubrics, score every interaction, and catch regressions before users do, with automated evaluators that run on every trace or on a schedule you control.
Agent Observability
Trajectory evaluation decomposes multi-step agent runs into ordered steps and scores each run on goal adherence, tool compliance, efficiency, and safety, automatically on ingest.
One SDK. Five minutes. Full audit trails, PII redaction, and guardrail enforcement, from day one.