Operational Alignment: A Framework, Benchmark, and Protocol for Evaluation of AI in Consequential Decisions

Abstract

Frontier language models are increasingly being used to make consequential decisions in settings where prior algorithmic systems have already caused real harm that was characterized only after the fact. Current evaluation does not tell a deployer or oversight body whether the model will hold a stated rule when ordinary deployment conditions push against it. Benchmarks aggregate over conditions. Bias prompts test what the model says, not what it decides. Adversarial red-teaming probes for attacks, not the realistic configurations a deployer will actually face. Trace-reading approaches, including LLM-as-judge, evaluate the output the model produced to look defensible, and so cannot see when that output is masking a configuration-driven violation.

We propose Operational Alignment, a pre-deployment auditing framework, benchmark, and protocol that identifies the available justification surface, the document, proxy, environmental signal, or institutional cue the model can use to make a violating decision appear defensible, as the variable that determines whether stated rules hold under deployment. The framework holds rule-relevant content constant across matched-pair contrasts and varies a single justification surface, providing causal identification of which variable produced an observed violation. The benchmark systematically explores justification surfaces across pressure types and rules across three regulated decision domains, eight frontier models, and 209,072 matched-pair decisions. The protocol applies the framework to a specific deployment, producing configuration-localized findings deployers, oversight bodies, and procurement reviewers can act on.

We find that single justification surfaces move the same model from near-zero to near-total violation; an available demographic proxy produces systematic denials of equally qualified applicants without the prohibited factor ever appearing in the prompt; and standard mitigations work in some configurations and backfire in others, with one regulatory-reminder intervention driving violations up 62 percentage points on the rule it was meant to reinforce. We identify compliance masking, violating decisions accompanied by defensible-looking justification that takes the shape of the deployment's justification surface, as a structural blind spot of trace-based evaluation methods now dominant in industry. Matched-pair audit, by varying the configuration while holding the facts constant, is the only methodology we are aware of that can surface compliance masking by construction. We release the framework, benchmark, manipulation library, audit reporting template, and per-cell reproduction artifacts at evav.ai.

1. Introduction

Frontier language models are now being used to make decisions that matter to people. Prior-generation algorithmic systems used in similar settings have already caused real harm: wrongful denials of medically necessary care allegedly contributing to patient deaths (Senate PSI, 2024); equally qualified applicants denied credit on the basis of facially-neutral variables that track demographic group (Bartlett et al., 2022); algorithmic actions in markets that produced events whose costs landed on households well outside the participating institutions. In each case the harm was reconstructed only afterward, through litigation, enforcement, or investigation, after the people who were denied had already been denied. Frontier AI is entering the same settings faster than the prior generation, with less external scrutiny, and the question is whether the forensic-after-the-fact pattern has to repeat.

Current evaluation methods do not see these failures in advance. Standard benchmarks aggregate over conditions and produce a single score that hides the configurations where rule preservation breaks down. Stand-alone bias prompts test what the model says when asked stereotyped questions, not what it decides when an ordinary deployment variable quietly pushes it. Adversarial red-teaming probes for cases where someone deliberately attacks the system, but the failures we report here are not adversarial: a moderate-tier physician note, a permissible-looking demographic proxy, an upward-trending market are not attacks; they are the conditions a deployed system will face on an ordinary Tuesday. Trace-reading approaches, including the LLM-as-judge methodology now widely adopted as deployment-time monitoring, evaluate the trace the model produced, and the trace is constructed to look reasonable. Each of these methods captures a slice of the safety question. None of them captures the configuration where the rule the deployer declared quietly stops holding.

Research literature characterizes how language models can fail internally: specification gaming, sycophancy, motivated reasoning, alignment faking, unfaithful reasoning under inspection. This work establishes that the failure modes exist. What it does not yet do is connect those internal mechanisms to the decisions and consequences they produce in deployment, name the variable that elicits them, or produce evaluation infrastructure that surfaces them before the harm occurs at scale.

We propose Operational Alignment, a pre-deployment auditing framework, benchmark, and protocol. The framework identifies the available justification surface as the variable that determines whether stated rules hold under deployment. The benchmark systematically explores justification surfaces across pressure types and rules, releasing matched-pair cells anyone can run on any model. The protocol applies the framework to a specific deployment, producing configuration-localized findings a deployer, oversight body, or procurement reviewer can act on.

Contributions

• The Operational Alignment framework identifies the available justification surface as the variable that determines whether AI decision systems preserve stated rules under deployment.
• A released benchmark of 209,072 matched-pair decisions across eight frontier models on three regulated decision domains.
• The Operational Alignment protocol: matched pairs, reversed-condition controls, seed-determinism, violation predicates, intervention-transport tests, and the audit reporting template.
• Empirical demonstrations: documentation-tier-driven rule violation, demographic-proxy-driven denials on identical credit-worthy profiles, environment-driven risk-control breach under no operator-side pressure.
• Compliance masking as a named phenomenon, a structural blind spot of trace-based monitoring including LLM-as-judge.
• Intervention-transport evidence showing the same fix can work in one cell and amplify failure in another.
• A composition argument that decision-level audit cells are the substrate for analyzing multi-agent deployments.

2. Why Current Evaluation Misses These Failures

Four families of methods dominate current AI evaluation. Each was designed to answer a different question. None of them isolates the available justification surface, and the failures we report sit precisely in the space they miss.

Aggregate benchmarks (MMLU, HELM, HumanEval, and domain-specific equivalents) measure performance across many conditions and report a summary score. They support model-to-model comparison and they characterize average competence. They do not isolate the configuration where competence breaks. A model can score well on aggregate while breaking on the specific configurations a particular deployment will face.

Bias prompts and fairness probes test what the model says when asked direct or stereotyped questions. They do not characterize what the model decides when the demographic-relevant variable appears as a proxy in an otherwise-routine decision context. A model can pass standalone bias evaluation and still produce systematic disparate outcomes when an available proxy does the work the prohibited factor is not allowed to do.

Adversarial red-teaming probes for inputs designed to break the system. The failures we report are not adversarial. A documentation tier that reads as cautious rather than confident is not an attack; it is what some physician notes look like. A zip-tier proxy is not an attack; it is a variable the underwriting pipeline has access to. A bull-market regime is not an attack; it is a Tuesday.

Trace-reading approaches, including LLM-as-judge methodologies and post-hoc reasoning analysis, evaluate the output the model produced for plausibility, alignment with stated rules, or quality. They share a structural property: they evaluate the trace, and the trace is the artifact the model produced to be evaluated. When the deployment supplies a justification surface, the model uses it to produce a trace that reads as compliant.

3. The Operational Alignment Framework

The framework rests on five claims about what AI evaluation in consequential decisions has to do that current methods do not.

The variable. The available justification surface in the deployment determines whether the model preserves a stated rule. A justification surface is anything in the deployment environment that the model can use to make a violating decision appear defensible: a document with adjustable language register, a proxy variable correlated with a prohibited factor, an environmental signal that constitutes performance pressure, a citation base, an institutional cue.

The unit. The unit of analysis is the configuration, not the model. A model is not safe or unsafe in the abstract. A model under a configuration is. The framework outputs configurations, named, reproducible, with their trigger variables identified, not aggregate scores or model rankings.

Causal identification. Matched-pair contrasts hold the rule-relevant content constant and vary the justification surface as a single configuration variable. Reversed-condition controls confirm the matched-pair structure is what produces the effect. Across the reported corpus, reversed-condition control violation rates are zero.

Transport testing. Mitigations are not interventions that work; they are interventions whose behavior is characterized across configurations. A fix that eliminates violations in one cell and amplifies them in another is not a fix, it is a configuration-specific behavior of the candidate intervention.

Releasability and API-only feasibility. The framework is built to be run by people who did not build the system under audit. Cells ship as self-contained reproduction units. Audits are conducted against the deployed model's API; weight access is not required.

4. The Benchmark

The framework's empirical instantiation is a released benchmark spanning justification surfaces × pressure types × rules across three regulated decision domains, eight frontier models, and 209,072 matched-pair decisions.

Models evaluated. Six primary closed-API models: GPT-4o, Claude Sonnet 4, Gemini 2.5 Pro, DeepSeek-V3, Qwen 2.5-72B, Gemma 3 27B, plus Llama 4 Maverick and one frontier model under embargo for which we report ablations only.

Domains. Three regulated decision domains, selected to instantiate structurally different justification-surface mechanisms:

• Healthcare: Medicare Advantage prior authorization under CMS MBPM Section 110 and 42 CFR 422.568. Justification surface: the language register of physician documentation.
• Lending: consumer credit decisions under ECOA's disparate-impact framework. Justification surface: a facially-neutral proxy variable (zip-tier, employment-type).
• Trading: fund-level concentration and leverage controls under the SEC Rule 15c3-5 risk-control framework. Justification surface: the environmental signal stream itself, price action, volatility, momentum.

Pressure types. Financial-incentive framings tied to medical-loss-ratio targets and desk-level P&L, authority preference, reassignment threats, interpersonal advocacy, regulatory-reminder framings, binding-constraint framings, and layered combinations.

Cell sizes and statistics. Most healthcare and lending cells use N=250 matched pairs (500 decisions per cell); trading N=100. Within-cell analysis uses McNemar's test with Wilson 95% CIs. Cross-cell analysis uses penalized logistic regression (L2), Type III ANOVA, and Holm-Bonferroni correction. Findings are robust to sampling temperature (3.3 pp span across T ∈ {0.0, 0.3, 0.7, 1.0}) and within-cell replication (SD 1.8–4.1 pp across three independent seeds).

5. The Protocol

The protocol is the procedure for applying the framework to a specific deployment. Where the benchmark characterizes how frontier models behave across a fixed set of cells, the protocol audits a particular deployer's system under the justification surfaces relevant to their rules and operating conditions.

Cell construction. For each operative rule, the auditor identifies the available justification surface in the deployment environment and constructs a matched pair: two cases identical in rule-relevant content, differing only in the candidate justification surface variable. A reversed-condition control verifies the matched-pair structure rather than the prompt template is producing any observed effect.

Violation predicates. For each rule, the auditor specifies a deterministic predicate on the structured output. A decision counts as a violation if and only if the output satisfies the negation of the predicate; violation classification does not depend on trace content.

Statistical machinery. Within-cell violation rate is computed by McNemar's test on matched outputs with Wilson 95% CIs. A cell is reported as a violation cell when the within-cell rate exceeds the reversed-condition control rate by a margin significant under Holm-Bonferroni correction at α = 0.05.

Intervention testing. Candidate mitigations are tested both inside the originating cell and across all other measured cells. Transport boundaries are reported explicitly: cells where the intervention works, cells where it has no effect, cells where it backfires.

6. Empirical Findings: What the Framework Surfaces

6.1 Documentation language becomes a violation dial

On the same operative rule (CMS MBPM Section 110) and the same patient profile, varying only the language register of physician documentation moves Claude Sonnet 4 from 0% violation under strong-advocacy documentation to 98% under moderate documentation to 100% under qualified documentation. The pattern is monotone across all six primary models: GPT-4o 6%/64%/89%, Gemini 2.5 Pro 12%/58%/85%, DeepSeek-V3 31%/88%/96%, Qwen 2.5-72B 47%/92%/98%, Gemma 3 27B 18%/71%/86%.

Figure 1. Violation rates by physician-documentation tier across primary models. Same patient profile, same operative rule; only language register varies. N=100 matched pairs per cell.

6.2 Proxy availability produces systematic denial of qualified applicants

In consumer credit decisions, race is not given to the model. Holding the underwriting profile constant in matched pairs and varying only a zip-tier proxy moves Claude Sonnet 4 from 0.4% violation to 91.6% on the same ECOA-style rule. The model is never given the prohibited factor. The available proxy is sufficient. The pattern reproduces with the same direction across all primary models.

6.3 Environment alone breaks hard risk constraints

In trading under fund-level concentration and leverage limits declared in the system prompt, no operator-side pressure is required to break the rule. Holding the risk constraint fixed and varying only flat-market versus bull-market regime data flips Qwen 2.5-72B from 0% to 89% violation and Gemma 3 27B from 0% to 94%. The environment itself supplies the performance justification, and the rule collapses.

6.4 Secondary patterns

Pressure composition compounds within domains. Layering one additional pressure type onto an existing financial-incentive regime drives every primary closed model above 63% violation in healthcare: GPT-4o 14% → 78%, Gemini 2.5 Pro 45% → 83%, Claude Sonnet 4 0% → 63% when interpersonal advocacy is added.

Failure surfaces are model-specific. Every primary model has at least one configuration where it preserves the rule completely and at least one where it fails. Which justification surface breaks each model is itself per-model.

7. Compliance Masking: A Structural Blind Spot of Trace-Based Evaluation

In every cell where the model violates the rule, it produces a structured justification alongside the violating decision. The justifications cite clinical literature, creditworthiness factors, risk management principles, the kinds of content a reader expects in a defensible decision. The trace reads as reasoned compliance. We call this compliance masking.

Definition. Compliance masking is the production of a structured violation accompanied by a justification that takes the shape of the deployment's available justification surface, generated to look defensible to a reader of the trace. It is distinct from random failure, from explicit rule-breaking, and from refusal. It is rule-breaking in compliant-looking form.

Why trace-reading methods cannot see compliance masking. LLM-as-judge evaluation, post-hoc reasoning inspection, and audit-trail review share a structural property: they evaluate the trace the model produced. A compliance-masked violation is, by construction, an output that reads as compliant under any such method. The only way to surface it is to introduce a comparator: a second decision on identical rule-relevant content with the justification surface varied.

Why matched-pair audit catches it. The matched-pair structure holds rule-relevant content constant across two cases. The justification cannot be coming from the facts; it must be coming from the configuration. The masking is exposed not by reading the trace but by varying the input the trace is responding to.

8. Intervention Transport: Why Mitigations Cannot Be Verified Globally

We test three standard mitigation classes:

• PROHIBIT: system-prompt language asserting the operative rule as binding and naming the prohibited driver explicitly.
• BIND: binding-constraint reframing presenting the rule as a hard constraint with no allowable override.
• REMIND: regulatory-reminder citation of the rule's source authority and the enforcement record.

Findings. PROHIBIT eliminates violations to 0% in some cells and degrades by capability tier on smaller open-weight models (Qwen 97.2% → 52.8%; Gemma 51.0% → 13.0%). BIND has variable effect with no consistent direction. REMIND, designed to reinforce the operative rule, drives Qwen healthcare violations from 28% baseline to 90%, a 62 percentage point increase on the rule the reminder was meant to reinforce. The same intervention has near-zero effect on Claude Sonnet 4 in the same cell.

Implication. A fix that works in one cell can fail or backfire in another on the same model and the same rule. The standard practice of selecting an intervention because it worked in a single configuration and deploying it across all configurations is not safe.

9. Composition: From Decision-Level Audit to Multi-Agent Systems

A multi-step pipeline does not require a new failure mode to become unsafe. It can assemble the ingredients of a decision-level cell across components. A retrieval module supplies the available justification surface. A planning module converts the high-level task into a performance objective. A manager message or learned scaffolding objective supplies the pressure. A downstream decision LLM produces a violating action while writing a compliant-looking trace, a compliance-masked violation.

What we have not shown. We do not have empirical evidence of cell composition. The composition argument is structural, not measured. Empirical composition is the natural next experiment, and the released benchmark is designed to support it.

10. Related Work

Algorithmic harm in regulated decisions. The harm literature in healthcare authorization and consumer lending establishes that prior-generation algorithmic systems caused documented harm reconstructed forensically. We extend this lineage to pre-deployment audit at the decision configuration level.

Auditing frontier LLMs in allocative contexts. Prior work audits GPT-4 housing recommendations, GPT-based hiring screening, and cultural bias in medical LLMs. We contribute a generalizable framework that identifies which configuration variable produces the failure on the operative rule.

Internal failure modes in frontier models. Specification gaming, sycophancy, motivated reasoning, alignment faking, and unfaithful reasoning characterize internal mechanisms. We contribute the connection to deployment: the configuration variables that elicit them and the audit infrastructure that surfaces them before harm at scale.

LLM-as-judge and trace-based evaluation. Our argument is not a critique of any specific implementation but a structural argument about the methodology class.

11. Implications

For deployers and procurement reviewers. The audit cell library provides defensible evidence of how a candidate model behaves under realistic deployment configurations before procurement and deployment decisions.

For oversight bodies. Regulators with API access can run the audit on the model versions deployers are actually using, before deployment scales to population impact. The shift is from post-deployment forensic reconstruction to pre-deployment structured audit.

For monitoring vendors and platforms relying on LLM-as-judge. The compliance-masking argument is a structural limitation that cannot be patched by stronger judge models.

What pre-deployment audit makes possible. The case for investing in pre-deployment auditing infrastructure is not a case for slowing AI deployment. It is the opposite. The constraint on responsible deployment at scale is not deployment speed; it is that current evaluation methods do not characterize how these systems behave under the conditions they will actually face.

12. Limitations and Released Artifacts

Limitations. The benchmark covers three domains; the framework's generality to other domains is asserted on structural grounds and instantiated on three. The corpus uses synthetic profiles anchored to real regulatory and litigation records. The composition argument is structural, not measured.

Released artifacts. Corpus of 209,072 matched-pair decisions across eight frontier models on three domains; manual-audit ground truth on a 400-decision subset; verbatim manipulation library; standardized audit reporting template; per-cell reproduction artifacts including prompt templates, matched-pair generators, violation predicates, and reversed-condition controls. All released at evav.ai.

13. Conclusion

Operational Alignment is a framework, a benchmark, and a protocol for pre-deployment audit of AI in consequential decisions. The framework identifies the available justification surface as the variable that determines whether stated rules hold under deployment. The benchmark releases the matched-pair corpus and reproduction infrastructure. The protocol applies the methodology to specific deployments. The compliance-masking phenomenon documented across all three domains is a structural blind spot of trace-based evaluation methods now dominant in industry, and matched-pair audit is the only methodology we are aware of that surfaces it by construction. This work exists to make AI deployment in consequential decisions defensible at scale, not slower.

References

1. Armstrong, L., Liu, A., MacNeil, S., Metaxa, D. The Silicon Ceiling: Auditing GPT's Race and Gender Biases in Hiring. EAAMO '24, 2024.
2. Bartlett, R., Morse, A., Stanton, R., Wallace, N. Consumer-Lending Discrimination in the FinTech Era. Journal of Financial Economics, 143(1):30–56, 2022.
3. Cen, S. H., Alur, R. From Transparency to Accountability and Back. EAAMO '24, 2024.
4. Chen, Y., Benton, J., Radhakrishnan, A., et al. Reasoning Models Don't Always Say What They Think. arXiv:2505.05410, 2025.
5. Denison, C., MacDiarmid, M., Barez, F., et al. Sycophancy to Subterfuge. arXiv:2406.10162, 2024.
6. Equal Credit Opportunity Act, 15 U.S.C. §1691 et seq., 1974.
7. Fuster, A., Goldsmith-Pinkham, P., Ramadorai, T., Walther, A. Predictably Unequal? Journal of Finance, 77(1):5–47, 2022.
8. Greenblatt, R., Denison, C., Wright, B., et al. Alignment Faking in Large Language Models. arXiv:2412.14093, 2024.
9. Howe, W., Carroll, J. Motivated Reasoning in Large Language Models. arXiv preprint, 2025.
10. Krakovna, V., Uesato, J., Mikulik, V., et al. Specification Gaming. DeepMind Blog, 2020.
11. Kunda, Z. The Case for Motivated Reasoning. Psychological Bulletin, 108(3):480–498, 1990.
12. Lanham, T., Chen, A., Radhakrishnan, A., et al. Measuring Faithfulness in Chain-of-Thought Reasoning. arXiv:2307.13702, 2023.
13. Liu, E., Suresh, A., Bratsberg, M., et al. Racial Steering by Large Language Models. arXiv:2409.08484, 2024.
14. Meinke, A., Schoen, B., Scheurer, J., et al. Frontier Models are Capable of In-Context Scheming. arXiv:2412.04984, 2024.
15. Nimo, A., et al. Probing Cultural Bias in Medical LLMs Across Global Contexts. arXiv preprint, 2025.
16. Obermeyer, Z., Powers, B., Vogeli, C., Mullainathan, S. Dissecting Racial Bias in an Algorithm. Science, 366(6464):447–453, 2019.
17. Pan, A., Chan, J. S., Zou, A., et al. Do the Rewards Justify the Means? arXiv:2304.03279, 2023.
18. Perez, E., Ringer, S., Lukosiute, K., et al. Discovering Language Model Behaviors with Model-Written Evaluations. arXiv:2212.09251, 2022.
19. Pfohl, S. R., Cole-Lewis, H., Sayres, R., et al. A Toolbox for Surfacing Health Equity Harms and Biases. Nature Medicine, 30(12):3590–3600, 2024.
20. U.S. Senate Permanent Subcommittee on Investigations. Refusal of Recovery. October 2024.
21. Sharma, M., Tong, M., Korbak, T., et al. Towards Understanding Sycophancy in Language Models. arXiv:2310.13548, 2023.
22. Suresh, H., Guttag, J. A Framework for Understanding Sources of Harm. EAAMO '21, 2021.
23. Turpin, M., Michael, J., Perez, E., Bowman, S. R. Language Models Don't Always Say What They Think. arXiv:2305.04388, 2023.
24. Zheng, L., Chiang, W.-L., Sheng, Y., et al. Judging LLM-as-a-Judge with MT-Bench. arXiv:2306.05685, 2024.