Evals and Model Risk Management: what makes an AI system deployable.

An LLM evaluation suite is a stratified regression-test apparatus, not an accuracy headline. It has three layers, and each layer answers a different question that your model risk function will ask in order.

Task-level evals run deterministic assertions on isolated model calls: regex matches, JSON-schema validity, exact-string ground truth against a held-out set, classification metrics on labelled fixtures. They tell you whether a single prompt + model + tool combination still does what it did last release.

System-level evals run end-to-end traces through the full agent: retrieval, tool calls, multi-turn state, intermediate decisions. Scoring is either reference-based (a known correct trace) or LLM-as-judge with a published rubric. They tell you whether the assembly still behaves when the pieces talk to each other.

Behavioural evals run adversarial probes: jailbreak attempts, refusal correctness, calibration tests, prompt-injection signatures. They tell you what the system does when the input is not the input you designed for.

Each model release, prompt change, or retriever update triggers a re-run against a golden baseline. Threshold breaches fail the CI gate. The same suite is then sampled continuously against live traffic, and the distribution of judge scores or labels is monitored with Population Stability Index. PSI under 0.10 is stable, 0.10 to 0.25 means monitor, 0.25 and above means investigate. PSI is the symmetric round-trip of KL divergence and is preferred for monitoring dashboards because it yields one signed magnitude per feature per window.

Your model risk committee is going to ask a specific question: 'show us the evidence that this system performs within tolerance, and show us how you will know when it stops.' A benchmark slide does not answer that. A three-layer eval suite with a PSI monitor does.

Concretely, for a buy-side firm running, say, an AI-assisted commentary or classification workflow, the deployable artefact is a packet: the task-level test set with version-controlled fixtures, the system-level traces with rubric scores, the behavioural red-team log, the current PSI on production samples, and a SR 11-7-aligned lifecycle document mapping the four phases to the people who own them. Without that packet the workflow stays in pilot indefinitely, because no second-line function will sign it into production without a regression mechanism they can audit.

Federal Reserve SR 11-7 (2011) defines the Model Risk Management lifecycle: development with documented conceptual soundness, independent validation by a second team, ongoing monitoring through drift and regression dashboards, and a model inventory tiered by materiality. SR 11-7 predates transformers and treats stochastic generative models as an interpretive gap that examiners are now actively filling.

EU AI Act Article 11 plus Annex IV require a Technical File before any high-risk AI system can be CE-marked: system description, development process, data governance, performance metrics, risk management, lifecycle changes, applied standards, EU Declaration of Conformity, post-market monitoring. That is nine sections, each of which assumes a real eval pipeline behind it.

Paired with DORA Articles 28 to 30 on third-party ICT risk, the eval suite has to live in a system whose operational resilience and data residency are themselves auditable. That is why we run the suite on local inference where the supervisory question allows, with the full eval trace and golden dataset retained inside the client's environment - the data that proves the model works does not leave the firm.

We instrument the three layers as separate scoring jobs, sharing a single dataset / solver / scorer schema modelled on Inspect AI's primitive split. Task-level uses promptfoo-style PR-diff tables for each release; system-level uses traced runs with rubric-graded judge calls; behavioural uses a curated red-team set kept under access control and rotated each quarter.

PSI is computed per scored dimension on a rolling window of production samples. The dashboard surfaces a single thresholded number per dimension per window, plus the histogram-over-histogram view behind it. The PSI bands trigger ticket routing: 0.10 to 0.25 routes to the engineering owner, 0.25 and above routes to the model risk committee chair.

Calibration is reported honestly. LLM-as-judge confusion matrices typically show around 75 percent accuracy on 'not relevant' and 30 to 50 percent on 'highly relevant'. We surface that asymmetry in the validation report rather than averaging it away, because the model risk function will find it later if we do not show it now.

Lifecycle docs map directly to SR 11-7's four phases and to EU AI Act Annex IV's nine sections. Independent validation is a second human team, not a second LLM - this is a point the standard is explicit on, and we keep it explicit in our deliverable.

All of this runs on infrastructure where inference, eval data, and trace logs are pinned to the client's region and tenancy. DORA Art. 28 to 30 compliance is a feature of the architecture, not a checkbox at the end.

Do not show 'AI accuracy 99.8 percent' as a hero number. It collapses three eval layers into one misleading bar, hides that benchmarks leak into training data, and ignores the LLM-as-judge verbosity and position bias that the literature has documented repeatedly.

Do not animate a single conversation bubble with a checkmark. That conveys 'I ran one prompt', not 'I am running a regression suite against a golden baseline with drift monitoring'.

Do not present LLM-as-judge as a substitute for SR 11-7 independent validation. The standard requires a second human team. Replacing it with a second model is a finding waiting to happen.

Do not put the golden dataset, eval traces, or PSI dashboards in a third-party SaaS your firm has not done a DORA Art. 28 register entry on. The evidence that proves the model is safe is itself in scope.