Glossary
Model Drift
Also known as: concept drift, AI model drift
Definition
Model drift is the silent decline of an AI model's quality, accuracy, or behavior over time. In LLMs and agents it shows up as quality regression after a model version update, prompt refactor, or shift in input distribution — and it is rarely visible without continuous evaluation.
Why it matters
Traditional ML models drift when production data distributions shift away from training data. LLMs and agents drift in additional ways: model providers update versions silently, prompt edits introduce regressions, retrieval indexes age, and tool definitions change. The result is a model whose outputs degrade week over week, often unnoticed until a customer or auditor complains.
For regulated systems this is a documented expectation. SR 11-7 requires ongoing monitoring of model performance with documented thresholds. NIST AI RMF MEASURE expects continuous evaluation. The EU AI Act Article 72 mandates post-market monitoring. Drift detection is the operational layer that satisfies these.
In practice
Prism Evaluations score every trace automatically across five dimensions — accuracy, relevance, tone, consistency, completeness — using an independent LLM-as-Judge. Quality scores are retained as time series so week-over-week regression is visible at a glance. Alerts can fire on thresholds; trajectories make root-cause analysis a matter of minutes rather than days.
Related
More glossary terms
Start tracing in 5 minutes
One SDK. Five minutes. Full audit trails, PII redaction, and guardrail enforcement, from day one.