Previously in this series
Issue #001 — Layer 1 · Purpose — Coutts. NHS triage. What happens when institutions deploy systems without encoding what they actually exist to do. The gap accumulates silently. Until it doesn't.
Issue #002 — Layer 2 · Strategy — Credit Suisse. NHS England. Eight years of drift between Purpose and what Strategy was actually serving. Nobody asked whether the two were still compatible.
Issue #003 — Layer 3 · Intent — Wells Fargo. Kaiser Permanente. A mandate issued without a boundary. The mandate was real. The outcome was not intended. The gap was never closed.
Issue #004 — Layer 4 · Rules — Wirecard AG. Orpea Group. The rules were followed precisely. The outcome was catastrophic anyway. Not misconduct within the rules. Rules that had become the architecture of the harm.
This issue: Layer 5 — Judgment. The layer that was delegated. To a model. To an agent. To a system. Without anyone defining how far that delegation went.
The institutions referenced in this issue are cited on the basis of publicly documented regulatory findings, official investigations, court filings, and other published reports. All analysis is educational. Nothing here constitutes legal, regulatory, financial, or investment advice.
00 · This Issue
The model decided. Nobody owned the outcome.
Every institution delegates judgment. A credit officer to a scorecard. A clinician to a protocol. A compliance team to a screening system. Delegation is not the failure. Delegation without a defined boundary is.
Layer 5 is where judgment gets handed to something other than the person accountable for it. When that handover is to a human, the boundary is implicit in the role. A junior analyst recommends. A credit committee decides. Imperfect — but structured.
When the handover is to a model, none of that exists. The model has no role. No defined limits. A function: optimise toward an outcome. It will run that function at whatever scale, against whatever population, across whatever decisions the institution points it at. The boundary does not arrive with the model. It has to be built. Almost nobody builds it.
The Layer 5 failure is not a wrong prediction. It is an absent definition. The institution never decided what the model was permitted to decide — and what it was not. So the model decided everything it could reach. The institution can tell you what happened. It cannot tell you what it authorised.
01 · Signals
Two institutions. Two sectors. The model decided. The institution was accountable.
Apple Card launched in August 2019. Apple built the interface. Goldman Sachs built the credit infrastructure. The product was marketed on fairness. No fees. No hidden charges. A credit product for a new era.
Credit decisions were algorithmic. No human underwriter. A model processed inputs and returned a verdict. The model decided. Across hundreds of thousands of applications.
November 2019. David Heinemeier Hansson posted publicly that Apple Card had offered him twenty times the credit limit it offered his wife. Same assets. Higher credit score on her side. Steve Wozniak reported the same pattern with his spouse within hours. The New York State Department of Financial Services opened an investigation.
Goldman's response: the algorithm did not use gender as an input. This was accurate. The algorithm did not see gender. It saw variables that correlated with gender — variables that produced systematically different outcomes for women in identical financial circumstances. The model did exactly what it was trained to do. The institution never defined what it was not permitted to do.
The DFS found no existing law violation. But its findings were specific. Goldman's model had not been tested for discriminatory impact before deployment. Monitoring protocols could not detect the pattern. No human review trigger existed — no defined point at which the model paused and a human decided instead.
How it happened. The model was trained on historical consumer credit data. That data reflected decades of structural economic disparity — women with lower recorded incomes, shorter independent credit histories, more accounts held as authorised users rather than primary holders. The model learned those patterns as legitimate creditworthiness signals. It was never told they were proxies. It was never tested to see whether, in combination, they produced systematically different outcomes for women with equivalent credit scores and assets. They did.
Launch: August 2019
DFS investigation: November 2019
Consent order: January 2023
Remediation: thousands of applicants
No human review trigger defined
Goldman admission: none
What was delegated
Full credit limit determination for every Apple Card applicant, at scale, without human review. The model's output was the institution's decision.
→
What was never defined
A human review trigger for any decision where the model's output produced statistically significant disparity across demographic groups — triggered before the decision reached the applicant.
The gap
The model exercised judgment over decisions carrying disparate impact across protected classes. Nobody had defined the boundary of that authority. The DFS investigation found it. That is the wrong order.
The model did not use gender as an input. That is accurate and entirely beside the point. The question was never what the model used. It was what the model was permitted to decide without a human in the loop.
UnitedHealth is the largest health insurer in the United States. Its subsidiary naviHealth built nH Predict — a tool used by Medicare Advantage plans to determine post-acute care coverage. Hip replacements. Strokes. Cardiac events. The tool decided how long elderly patients could remain in a skilled nursing facility before coverage stopped.
The mechanism was simple. The algorithm predicted a length of stay. When the actual stay exceeded that prediction, coverage was denied. The treating physician's judgment was irrelevant. They could document medical necessity. The algorithm had already decided. The insurer followed the algorithm.
In 2023, a federal class action alleged UnitedHealth had used nH Predict to systematically override physician judgment and deny medically necessary care. Specific patients were named. Elderly individuals discharged prematurely. Readmitted to hospital. Some died. Internal documents cited in the suit alleged that UnitedHealth employees knew the algorithm's denial rate far exceeded clinical norms. The deviation was not investigated. It was operationalised.
UnitedHealth denied the allegations. The case has since moved. In February 2025, a federal judge denied UnitedHealth's attempt to dismiss — allowing breach of contract and good faith claims to proceed toward trial. Nine of ten appealed denials were ultimately reversed, a figure cited in court filings. In March 2026, the court ordered UnitedHealth to disclose the nH Predict algorithm itself in discovery. UnitedHealth continues to dispute the allegations. The case is active.
Tool: nH Predict — predicted length of stay
Denial trigger: stay exceeds prediction
Physician override: none
90% of appealed denials reversed
Feb 2025: case survives dismissal
Mar 2026: algorithm disclosure ordered
What the algorithm could legitimately do
Inform clinical review. Generate a predicted length of stay as one input into a coverage determination that a human clinical reviewer would then make.
→
What it was permitted to do instead
Issue coverage denials automatically when a patient's stay exceeded the predicted duration — overriding documented physician assessments of medical necessity without mandatory human clinical review.
The gap
No defined clinical override point. When a treating physician documented medical necessity contradicting the algorithm's prediction, no mechanism required human clinical sign-off before the denial was issued.
The question is not whether the algorithm was accurate. It is whether the algorithm was authorised to make the decision it was making. A model can inform clinical judgment. A model that automatically overrides a physician's documented assessment is not informing judgment. It is replacing it.
02 · Diagnosis
Delegated judgment without a defined boundary is ungoverned judgment.
Both failures share the same structural signature. Same country. Different sectors. Different scales. The same underlying absence.
A model decided. The institution called it an objective process. When the outcomes produced harm — disparate credit access, denied medical care — the defence was identical. The model did not use a protected characteristic as an input. Both institutions said the same thing. Both were accurate. Neither was relevant.
The Layer 5 failure is not whether the model used the right inputs. It is whether the institution defined the boundary of the model's authority. What it was permitted to decide. Under what conditions. At what point a human was required before a decision was issued.
Goldman had not defined a human review trigger for credit decisions producing statistically significant demographic disparity. UnitedHealth had not defined a clinical override point — a threshold at which a physician's documented medical necessity halted an algorithm-generated denial.
In both cases, the judgment was fully delegated. The accountability was not.
A model without an authority boundary does not have bounded judgment. It has unlimited judgment operating inside whatever constraints its training data happened to encode. That is not governance. That is inference at scale.
The DIC™ is precise here. Judgment is not binary. Not delegated or retained. It exists on a spectrum — consequence on one axis, uncertainty on the other. The institution's job is to mark, at each point on that spectrum, what requires a human before an outcome is issued. Not after. Before.
When that marking is absent, the institution has not delegated judgment. It has abandoned it. The model decides. The institution is accountable. The gap between those two facts is where Layer 5 failures live.
03 · Engineering Note
What Layer 5 governance actually requires.
Three mechanisms every institution needs at Layer 5.
// L5 Governance Mechanisms
Before deployment, the institution defines in writing what the model is authorised to decide, what it is authorised only to inform, and what conditions require a human before a decision is issued. Not a model card. An authority mandate — the document that would govern a human professional with equivalent discretion. If it does not exist, the deployment has no defined boundary.
Apple Card / Goldman Sachs
A defined human review trigger for any credit decision where the model's output produces a statistically significant disparity across demographic groups — triggered automatically and before the decision is communicated to the applicant.
UnitedHealth / nH Predict
A defined clinical override point: when a treating physician documents medical necessity that contradicts an algorithm-generated predicted length of stay, coverage denial requires human clinical review and sign-off before it is issued. The algorithm informs. The human decides.
Institutions monitor accuracy. Almost none monitor outcome distribution across populations the model was not designed to discriminate against. These are not the same thing. A model can be accurate on average and produce systematically disparate outcomes across demographic or clinical subgroups. Accuracy monitoring will not surface it. Outcome distribution monitoring will — if it exists. This is not a post-hoc audit. It runs in parallel with deployment, with defined thresholds that trigger review when the distribution shifts.
Goldman Sachs
The DFS investigation found the disparity pattern that Goldman's own monitoring had not surfaced. That is the wrong order. The distribution anomaly should trigger internal review before it triggers a regulatory investigation.
UnitedHealth
A denial rate significantly exceeding clinical norms should trigger automatic governance review. The allegation is that it was instead operationalised. Outcome monitoring, not accuracy monitoring, is what catches this.
Every consequential model deployment maintains a log. Decisions where a human overrode the model. Decisions where review was triggered and the model was upheld. Decisions where no review occurred and the model's output was final. The absence of override records is not evidence the model is performing well. It is evidence nobody is watching the boundary.
Apple Card / Goldman Sachs
No documented human review trigger existed. No override log therefore existed. The institution had no systematic record of where the model's judgment had operated without human governance.
UnitedHealth / nH Predict
The gap between physician medical necessity documentation and coverage denial decisions should appear in an override log as an anomaly. That log is governance functioning. Its absence is governance absent.
04 · The Board Question
A model is making decisions inside your institution right now. Affecting customers. Patients. Counterparties.
For each one: what is the defined boundary of its authority? What can it decide without human review? What triggers mandatory human sign-off before a decision is issued? When did someone last check — not the accuracy, but the distribution of outcomes?
If your governance committee cannot answer those questions in writing, for each deployed model — the boundary has not been defined. The judgment is ungoverned. The accountability is yours.
Decision Integrity Chain™ · Layer 5 of 8
Judgment
The exercise of discretion where rules run out — the act of interpreting a situation and choosing a course of action in conditions the rulebook did not fully anticipate. Judgment becomes a governance failure at Layer 5 when it is delegated to a model without a defined authority boundary: what the model is permitted to decide, what it is permitted only to inform, and what conditions require a human to decide before an outcome is issued. Ungoverned judgment is not a model problem. It is an institution problem.
Issue #006 — Layer 6 · Decision. Where the act of deciding is recorded, or isn't. Where replayability is either possible or permanently foreclosed. Knight Capital. Theranos.
Read Issue #006 →