Today I read the OCC's new Supervisory Framework on Bank Use of Super Intelligence, comparing it against the four questions I have argued govern all model validation. My conclusions are that much of it aligns well, that its organization could be better, and that three gaps remain. This note sets out the details, and in particular the arrangement I would adopt for ownership of the evaluation set, which is the gap I think matters most.
Because what follows turns on them throughout, the four questions are these:
I set out the case for them, and for why they continue to hold when asked of agentic systems, in an earlier note in this series.
A footnote on the first page settles what the document is for. The model risk management principles in OCC Bulletin 2026-13 apply to non-generative and non-agentic models, and the framework published today is "more applicable to generative and agentic SI." When the revised interagency guidance appeared in April, I observed that it had placed generative and agentic systems outside its scope and that a separate issuance would follow. This is that issuance, and the division of labor between the two is now explicit.
It is worth being specific about the alignment, because the criticisms that follow are narrow and should not be mistaken for a general complaint.
The framework declines to treat Super Intelligence as a separate risk category, stating that "examiners should not treat SI as a separate and distinct risk category, and banks are not required to define a separate SI risk appetite." That is the correct instinct, and it is the same instinct that led several of my examiner colleagues at OCC to treat a credit scorecard as an input to a business decision, and not the only important object of validation.
The framework handles non-determinism properly, observing that "repeated testing, defined samples, distributional measures, and consistency thresholds may provide more useful evidence than a single successful run." It anticipates the collapse of champion-challenger methods, noting that "traditional champion-challenger methods may not fit open-ended generative outputs" and points toward alternative benchmarks, reference sets, expert review, and controlled comparisons. It addresses changes that occur outside a model release with specificity, identifying "upstream version changes, prompt or configuration changes, fine-tuning, data changes, or new tools and permissions that alter behavior without a traditional model release," and it asks that model, prompt, fine-tuning, data, configuration, provider, tool, and permission changes be logged and linked to testing and approval evidence. It treats a "judge model" as a model itself, listing that model's own validation, independence, and correlated failure modes among the matters a bank may consider.
Taken together, those passages are Questions 3 and 4 restated in supervisory language, and an institution that took them seriously would benefit.
Model validation has typically been organized around a one-time handoff. The first line built the model and prepared a development package setting out the business purpose, the data, the specification, the estimation, the testing performed, and the known limitations. The second line then received that package, challenged it independently against standards that policy had established well before anyone opened the model, and reported a conclusion together with any findings and limitations on use.
The arrangement depended upon two conditions that were seldom stated, because they were seldom absent. The first is that the artifact remained stationary long enough to be reviewed. The second is that materiality of change was a reasonably legible idea, so that a revalidation trigger — often based on performance — could be written into monitoring and redevelopment or recalibration plans, and relied upon thereafter.
The single handoff approach stopped being universal practice some years ago, and the reason is worth recalling, because it is the closest precedent available to what agentic systems now require.
When the large bank holding companies began building models for supervisory stress testing, they faced a submission deadline that did not move. An institution running dozens of models across its loan portfolios, its trading book, its pre-provision net revenue forecast, and its balance sheet projections could not develop each of them, document each of them, hand each of them to validation, and then absorb validation's findings in sequence, all within a single annual cycle. Something in that sequence had to give, and what gave was the one-time handoff.
What replaced it was development and review in tranches. The second line reviewed specifications before estimation was complete, commented on data and segmentation choices while those choices could still be changed, and took up components of a model rather than waiting for the assembled whole. Findings arrived during development rather than after it, which meant that the first line could act on them inside the cycle instead of carrying them forward as open items into the next one. Validation work was scheduled against the development calendar rather than positioned at the end of it.
The second line gave up something real in that arrangement, namely the clean separation between a finished artifact and an independent review of it, and nobody at the time pretended otherwise. Iterative engagement was understood to carry a risk to independence, for the plain reason that a validator who has commented upon a specification while it could still be changed has to some degree participated in it. The response was not to deny the risk but to assign it. It fell to the business line and to the second line manager to produce documented examples showing that the challenge brought to a design had in fact been genuine, independent, and effective. The obligation was not to assert that it was so. It was to exhibit instances in which it had been.
What survived, therefore, was neither a claim that the risk had been eliminated nor a signature asserting that it had been managed. It was an obligation to produce evidence of challenge actually exercised: the objection that changed a specification, the finding the first line accepted, the design altered because the challenge had landed. That is a sturdier arrangement than the one it replaced, because independence is shown rather than inferred from the shape of the process.
It is worth noticing what the obligation required in practice. Examples of that kind exist only if the exchanges that produced them were recorded while they were occurring, since they cannot be reconstructed afterwards from a process diagram or a committee minute. The evidence had to be built into the way the work was done. That is the same requirement I am pressing be adopted for agentic systems, yet it was developed more than a decade earlier and for an entirely different reason.
That precedent does not dispose of the first objection anyone will raise to what follows. It does something more useful, which is to supply the control that answers it. Continuous second line engagement is neither novel nor theoretical, the largest institutions have more than a decade of practice in it, and the mechanism by which they preserved independence while engaging early is a matter of record rather than of invention.
The precedent has a limit, however, and that limit is the subject of this note. A stress testing model, however compressed its development, still produced a reviewable artifact at the end of the cycle; the documentation was written after the fact, it described something that had stopped moving, and a validator arriving late could still read it and reach a conclusion. Compressed timelines changed when the second line engaged. They did not change what the second line was given. Agentic systems change what the second line is given, because there is no point at which the thing being validated comes to rest, and no document that can describe it after the fact.
Before turning to the gaps, it is worth being precise about what the OCC framework document actually comprises, because the answer explains a good deal about how it is arranged.
The framing is addressed to examiners. The opening sentence states that the document "provides bank examiners with a supervisory framework for evaluating banks' use of Super Intelligence," and the executive summary goes on to say what examiners may use it for, what they should not do with it, and that they will encounter varying levels of sophistication and should apply judgment. The word examiner appears seven times in the document, five of them in that summary.
The sixteen components are addressed to banks. Once past the framing, the voice changes: a bank may document its purpose, the bank may consider the judge model's validation, metrics may include accuracy and precision and recall. The construction "the bank may" occurs eighteen times, the auxiliary "may" one hundred and thirty-one times, and the words "must" and "is required" do not occur at all.
What the OCC has produced, therefore, is a hybrid of supervisory guidance and a Comptroller's Handbook booklet: the audience and authority of the latter, the content and voice of the former. I do not say that critically of the decision to publish it, which seems to me sensible given how quickly technology is moving and how little settled practice exists to codify. But the hybrid has consequences for the document's structure, and those consequences are the subject of the next two sections.
A Comptroller's Handbook is organized around examination work — who looks at what and in what order. Guidance is organized around expectations, which is to say around what must be true of a bank's arrangements. The two organize on different axes. A document attempting both will necessarily mix what must be demonstrated with who must demonstrate it and how that work is to be governed, and sixteen components distributed across six pillars is what that mixture looks like when it is written down.
This also explains something that would otherwise look like carelessness, and which I take up next: how a framework can state a requirement to define purpose, state a requirement to monitor performance, and never connect the two.
The framework opens with a pillar named Purpose, and the pillar is defined in the document's own words as one that "defines the actions the SI system or agent is permitted to execute." The component beneath it asks a bank to document, before deployment and after a material change, "what an SI system or agent is designed to do, what it is not designed to do, who or what may invoke it, the data and systems it is permitted to access, the actions it could take, and the circumstances that may require human review or suspension."
Every item on that list is a boundary. A boundary is a valuable thing, and documenting one is the beginning of answering Question 3, since an agent that can act only within a declared scope is an agent whose implementation can be checked against its specification. But none of those items is an objective, and none of them is measurable.
Metrics do appear in the framework, two pillars later, under Oversight, where performance monitoring contemplates accuracy, precision and recall, task completion, hallucination, abstention, complaint, override, exception, and remediation rates. The difficulty is that nothing joins the two. A bank may document its permitted actions under Purpose, monitor a menu of output-quality metrics under Oversight, and satisfy both without ever stating what the system was deployed to achieve in terms that those metrics could test. My Question 1 and Question 4 are each present in the document, and they are not connected to one another. That is a failure of arrangement rather than of substance, and it follows directly from the hybrid described above. Purpose, in a document that is partly a Handbook booklet, answers the examiner's question of what the bank has authorized. Performance monitoring, in a document that is partly guidance, answers the bank's question of what it should watch. Neither axis requires the two to meet, and so they do not.
The framework enumerates the change surface well, and better than most bank frameworks do. What it cannot do, for the reasons just given, is settle which function inside a bank owns each component and which of them is obliged to tell model risk management that something has moved.
In practice the system prompt and task instructions often have no inventory entry and no change log, although a prompt edit is a specification change. Tool definitions determine what an agent is able to do and are typically release-managed as software. Permission scope is frequently administered as entitlements and sits outside model risk management altogether. Planner and orchestration logic is treated as software, the retrieval corpus and its index as a matter of data quality rather than as a change in the distribution of model inputs, the vendor model version as vendor risk, and the sampling parameters as configuration.
Each of these is governed somewhere. The framework now says they should be logged and linked to testing and approval evidence, and that changes of this kind may justify more frequent monitoring or re-validation. Whether any of them reaches the second line depends on arrangements the framework does not and could not specify. Establishing which function owns each component, and whether it is obliged to notify model risk when it changes, is the first diagnostic worth undertaking in any institution that has deployed one of these systems, and it is the one most likely to produce an uncomfortable answer.
One consequence deserves to be stated plainly, because it runs against both the way guardrails are commonly sold and the way they are commonly bought. A guardrail that classifies an output and determines whether it should be blocked is itself a model, and it therefore requires that the four questions be answered of it --- including a stated objective with a metric attached, and evidence of performance against that metric. At the same time, and without any contradiction, that guardrail constitutes developmental evidence for Question 3 of the agent it constrains, since a permission boundary that is enforced and logged is evidence that the system can do only what its design permits.
The framework reaches the same conclusion from the other direction when it treats a judge model as requiring its own validation, and when it asks whether a circuit breaker or fallback agent serving as a backstop is itself independently governed, tested, and effective. An institution that treats the purchase of a guardrail as a substitute for validation has acquired an unvalidated model and described it as a control, while an institution that treats the same guardrail only as a control has overlooked evidence that its validators need.
Effective challenge has frequently meant replication: the validator rebuilds the model, or constructs a benchmark, and compares the two. Against an agentic system this is largely unavailable, because the system is non-deterministic and a single run therefore does not constitute a result, because the underlying model cannot be re-estimated, and because that model may be updated by a vendor on a schedule the institution does not set. The framework acknowledges as much in observing that champion-challenger methods may not fit open-ended generative outputs.
Three instruments substitute for replication, imperfectly. Repeated-run testing characterizes the distribution of outputs for a fixed input rather than reporting a point estimate, and variance across runs is itself a finding, since a system whose answer changes materially between identical runs has a Question 2 problem that no amount of averaging will resolve. A challenger evaluation, as distinct from a challenger model, makes the second line's independent contribution the test rather than the artifact. And ablation tests the design claim directly: where the design asserts that the agent requires a particular tool or a particular retrieval source, the assertion can be examined by removing it and observing what follows, and where the design predicts a given sensitivity to the quality of the corpus, the corpus can be degraded and the prediction checked. Ablation remains underused, precisely because it interrogates the design rather than the output.
For a credit model, outcomes arrive of their own accord. Loans default or they do not, and the validator eventually possesses outcomes data that nobody was obliged to construct. For an agent triaging alerts, reviewing documents, or servicing a customer, the correct answer must instead be built, which is to say that somebody has to determine what the system ought to have done.
The framework comes close to this point and then turns away from it. Its treatment of subjective review is careful, asking that "reviewer qualifications, procedures, sampling, response rater consistency, and the number of observations" be calibrated to the undesirable-outcome rate the bank is attempting to detect. Every one of those provisions concerns the reviewers. None of them concerns the set of cases against which the reviewers work, or who selects it. Whoever builds that set has defined acceptable performance, and it becomes the operative Question 4 threshold whatever the policy may state the threshold to be.
A set capable of carrying that weight requires four kinds of case: representative cases drawn from the production distribution; boundary cases in which the correct answer is contested and reasonable reviewers disagree; adversarial cases, including instructions injected by way of retrieved content; and cases admitting of no good answer, in which correct behavior is escalation or refusal rather than an attempt. The last of these is the category most frequently omitted, and it is the one that distinguishes a system that recognizes the limits of its own competence from one that does not.
Two further properties are material. The set goes stale, because the production distribution moves, the corpus changes, and the vendor model changes, with the consequence that a performance figure is interpretable only against a named version of the set and a declared refresh cadence. And a set that the developer is able to see is a set to which the system will be tuned, whether deliberately or through the ordinary course of iteration.
Moritz Hardt has made the stronger version of that second point about machine learning benchmarks generally, and it deserves more attention in banking than it has received. Benchmarks, on his account, do not simply measure; they “lay out the rules of the game,” and the institutions around them reinforce those rules “by rewarding those who come out ahead in the metrics.” What follows is that the scoring enters the design. Model builders put benchmark data into the development loop when searching for an architecture, and post-training, as he puts it, “is what you do with benchmark performance in mind.” The consequence for a bank is worth stating plainly. An evaluation set does not merely measure the system that was built; it selects among the systems that might have been built, because a developer who knows what will be scored will build toward it.
If that is right, then ownership of the evaluation set is not a question about testing hygiene. It is a question about who exercises design authority. A second line that leaves the set entirely to the first line has conceded a measure of Question 2 along with Question 4, and will later find itself reviewing a design that its own absence helped to select. This is the strongest argument I know of for the arrangement set out in the next section.
Hardt's companion finding bears on how thresholds are written. Model accuracies, he reports, “change erratically from one benchmark to the next,” while “relative comparisons between models turn out to be surprisingly reliable.” An absolute score is therefore the least durable thing a benchmark produces, and it is precisely the thing a policy threshold tends to record. A policy that records an acceptance threshold as a bare percentage is therefore doing less work than it appears to, unless the version of the set is named alongside it. That is a further reason to version the set and declare its refresh cadence, rather than to record a figure in a committee minute and treat the matter as settled.
No set can be shown to cover the adversarial space, which is true of every test set ever constructed and is not a reason to forgo the attempt. It is a reason to grow the set from production, so that an incident becomes a case, and so that the rate at which genuinely novel failures arrive becomes itself a measure of how well the set covers the space.
Four arrangements are available, and the choice among them determines whether effective challenge survives contact with these systems.
The first line owns it. The capacity is present, but the independence is not, and performance measured against a set that the builder controls is a statement about what the builder was aiming at rather than about the reliability of the system.
The second line owns it. The independence is present, but the capacity ordinarily is not, since the construction and maintenance of a production-representative labeled set is domain and engineering work on a scale that few validation groups carry. This arrangement also renders the second line the effective owner of the acceptance criteria, which tends toward the co-design it was intended to avoid.
A third party owns it. Two distinct arrangements are described in these terms, and only one of them fails. A published vendor benchmark measures general capability rather than this institution's use of this system on this population, and it cannot serve as a Question 4 threshold however well constructed it may be. There is a second difficulty with such benchmarks that is easily missed. Hardt observes that competitors now guard their data recipes as closely as the formula for Coca-Cola, with the result that it is rarely possible to tell how much a given model was specifically prepared for the benchmark on which it performs well. A bank relying on a published score is therefore relying on a number whose provenance it cannot inspect. A vendor engaged to build a use-specific set, to a sampling frame that the second line specifies, with custody of the result remaining at the institution, is a different proposition altogether, and a reasonable answer to the capacity problem. The conditions attaching to it are what matter: the institution specifies the frame, the institution holds the set, and the vendor's own judging models are validated as any other model upon which the institution relies. A judge drawn from the same model family as the system under test, or tuned by the party that built it, does not supply independence; a judge from a different family, specified by the second line, may do so, and still requires validation in its own right.
Split, with a withheld holdout. The first line builds and maintains the working evaluation set, both because that is where the capacity resides and because the working set ought properly to drive development. The second line independently specifies and constructs a holdout drawn from the same population, scores against it, and does not surrender it.
I would adopt the fourth. It preserves effective challenge without requiring the second line to carry the full engineering burden, since specifying and curating a holdout is a considerably smaller undertaking than building and maintaining the working set, and it leaves development where it belongs. It also yields a diagnostic that is not presently measured anywhere, which is working-set performance read against holdout performance. The signal to watch is not a gap as such, since tuning that generalizes will lift both figures and a small holdout will move on sampling noise alone; it is working-set performance improving while holdout performance stalls or declines, on a holdout large enough that the difference exceeds sampling error. That pattern is the signature of tuning to the test, which is the failure mode everyone professes to worry about and nobody currently instruments.
Three conditions are necessary. The population and the sampling frame are agreed before construction begins, so that the holdout is drawn from the same distribution rather than argued over afterwards. The refresh cadence is declared in policy. And the holdout is rebuilt rather than reused once detailed results have been reported back to the first line, because detailed feedback leaks the test. That condition preserves what Hardt calls the “no second-guessing guarantee” of a sound benchmark, which repeated exposure of a test set destroys: the leading candidate may simply have studied harder for that particular test than the others did.
There is a constructive response to benchmark-shaped design that I have long favored and will set out separately. It involves drawing training and evaluation samples at random from the same population, repeatedly rather than once, and selecting the final model structure and calibration on the basis of the drivers of performance that prove robust across those draws, rather than on the basis of which candidate scores best against a single set. What that procedure selects for is stability rather than a number, which is precisely what a set-shaped design process fails to select for. It deserves its own treatment and will get one.
The framework contemplates testing at the design and development stage, asking whether proposed controls address identified risks "before the use case is finalized," and its documentation component asks for decision traceability covering inputs, outputs, context, retrieved sources, tool calls, approvals, overrides, and actions. What it does not say, because it is addressed to examiners rather than to banks, is that the second line should state in advance what evidence it will require, and that the first line should be expected to build a system capable of producing it.
An institution writing that requirement for itself would specify at least the following. A stated objective with a metric attached, since Question 1 is not answered by a description of the task and Question 4 cannot be answered at all until the objective has been given a measure. Step-level logging, comprising the plan, each tool call together with its arguments, each result returned, and the final action, with the prompt version, the tool definition version, and the model version recorded against every entry. Replayability, so that a recorded case may be re-run against an altered configuration. Prompts and tool definitions under change control, recognizing that not every edit warrants a validation review and that insisting otherwise will fail upon contact with a development team; the requirement is that every version be recorded and attributable, and that a defined class of change covering objective, scope, and tool access should trip a review rather than a release note. And interfaces sufficiently stable that a holdout can be scored without engineering assistance from the first line.
The test for each item is the same: six months hence, will the second line be able to answer Question 3 and Question 4 concerning a decision the system takes today? If it will not, then the evidence was not designed in, and it cannot be supplied afterwards, because a log cannot be retrofitted once the decisions it ought to have recorded have already been taken.
The objection to the foregoing is that a second line which specifies evidence requirements before the build has thereby joined the design effort. I do not think that follows. Model risk policy has always established documentation and validation standards in advance, that is the ordinary form of second line work, and it is the reason the handoff functioned at all. The distinction worth defending is between specifying the evidence and choosing the design: requiring step-level logs belongs to the former, while selecting the orchestration pattern belongs to the latter.
That distinction will not always be clean in practice, and the stress testing experience suggests the right response to the residual risk. It is not to pretend that early engagement costs nothing. It is to make somebody accountable for showing that the challenge was real. A second line that specifies evidence before the build should expect to produce examples of the challenge it subsequently brought to the design, and the business line should expect to confirm them. An institution that cannot produce that documentation has a genuine independence problem; one that can has answered the objection in the only way it has ever been answered.
There is also a cost attaching to the cautious reading, and it is not a small one. A second line that engages only at the conclusion of the build receives a system it is unable to validate, and is then confronted with a choice between approving upon inadequate evidence and blocking a deployment for which the business has already paid. Late engagement does not protect independence in those circumstances. It relocates the problem to the point at which the institution has the fewest options available to it.
The framework describes itself as a living document expected to change as technology, use cases, and supervisory understanding mature. The gaps identified here are of a kind that a later revision could close, and I expect some of them will be. None of that is a reason for an institution to wait, because nothing in what follows requires an issuance that does not already exist.
In the first thirty days, map the change surface and establish which function governs each component, and whether any of them is obliged to notify model risk when it changes. In the second thirty, agree the remediations with the first and second lines: the logging specification, the scope of change control, the composition of the evaluation set, and the holdout arrangement. At ninety days, write the policy.
The OCC has now set out what these systems may be permitted to do. What remains is for institutions to state how they will know the systems are doing it well, and that is a statement about objectives, metrics, and evidence that no examiner framework can write on their behalf.
← Back to Thought Leadership