What It Takes to Trust an AI Deployment Gate: Measuring Four Models Against a Dumb Baseline

A Green Build is not a Risk Assessment

Your test suite answers one question: is this code correct according to the checks we wrote? That is a useful question, but it is not the one that matters most at deployment.

Take two changes that both pass every check. The first is a one-word fix to a README. The second is a 1,057-line change to the payment service, pushed at 18:40 on a Friday, to a service with three alarms currently firing that was last deployed thirty minutes ago. Your pipeline treats these two changes as the same event. Both are green, so both ship.

An engineer would not treat them the same. If you asked the engineer why they hesitated on the second, they would tell you something about the payments module, the time, and the state of the service. They are combining the change with the environment it is going into, and that combination is what turns a correctness check into a risk assessment.

Every signal that an engineer would use already exists in your infrastructure. CloudWatch knows the error rate. Your scanner knows about the unscored finding in that dependency bump. Your deployment history knows this is the fourth push today. The information is all there, but it never reaches the point of decision, because no component is responsible for making one. The pipeline ships whatever passes the checks.

So I built something to sit in that position. It collects signals about the change and about the target environment, asks an AI model to assess the risk, and acts on the answer by deploying in full, rolling out a canary, or halting and paging a human. It runs end to end on AWS, and it works. It is also the least interesting part of this article.

The interesting part started when I tried to see whether the model earned its place. As a control, I wrote a scoring function (fifteen lines that count attributes like diff size and alarm state, with no AI model involved at all). I expected the model to beat it comfortably. However, it did not. I ran four models from three vendors against 23 labelled scenarios: Haiku, Sonnet, and the scoring function all waved through dangerous changes at the same rate. The two Claude models failed on the same scenarios and gave the same reasoning for failing, while the scoring function failed on different ones

That result took the project somewhere I hadn't planned to go, and that's why this article exists.

By the end of this article, you will know how to:

• Build a deployment gate that uses an AI model to decide release risk, and why the component that decides must never be the component that acts
• Design an evaluation that can actually tell you whether a model is helping, including the two error rates you must never average together
• Tell the difference between a model that is not capable enough and a job you never specified properly
• Constrain a model’s output with a deterministic floor without making it uselessly cautious
• Avoid the measurement mistakes that make most published agent evaluations meaningless

Every number here comes from a working system. The code, evaluation harness, raw results and decision log are public, so that you can reproduce any of it yourself. You can access them on this repo.

How the AI Deployment Gate Works

The gate sits between a passing build and production traffic. When the pipeline reaches the deploy stage, it calls a decision service instead of deploying straight away.

That service gathers three kinds of signals:

The change itself. How many lines moved, which paths they touched, how many times this service has been deployed today, how long since the last deploy, and what the commit message says.

Security findings. Amazon Inspector results for the artefact, counted by severity, including findings that have not been scored yet.

The target environment. The service's error rate and latency over the last few hours, the same figures for the equivalent window a week ago, and which alarms are currently firing.

Those signals are assembled into a single document and sent to the model, which returns a structured verdict: a risk level (low, medium or high), a confidence score, its main concerns and the reasoning behind them. The verdict returns through forced tool use rather than asking the model to reply with JSON, so the API rejects responses that don't fit the schema instead of parsing them.

The verdict is then validated, written to DynamoDB, and acted on. Low-risk deploys in full. Medium risk goes out as a canary. High risk halts the pipeline and sends a human an email with an override command they can run if they disagree.

The decider holds no deploy permissions

One important design decision is that the component that produces the verdict cannot deploy anything. It has no permission to change traffic, no permission to invoke the executor, and no write access to anything except its own verdict table. A separate executor holds the deployment permissions and acts only on a verdict record that has already passed validation.

This is important because the decision service processes input that an attacker can manipulate. A commit message is written by anyone who wants to initiate the deployment. If someone includes a sentence in a commit message instructing the model to approve the change, the model will read it.

Splitting the roles makes that attack boring. The worst-case scenario is when someone convinces a component that cannot deploy to issue a verdict in favour of deployment. However, this verdict must still pass validation before any authorised actions can take place. The security boundary is maintained by IAM (Identity and Access Management) rather than relying on the model to disregard instructions embedded in its input.

Running it safely

The gate runs in one of three modes.

Mode	Produces a verdict	Reports into the pipeline	Can block a deploy
Shadow	Yes	No	No
Advisory	Yes	Yes	No
Enforcing	Yes	Yes	Yes

Shadow mode is how you find out what a configuration would have decided without letting it decide anything. It is also how you compare a new configuration against the one currently running: both see the same real pipeline executions, and you watch where they disagree before giving the new one authority. Since changing the model, the prompt or the signals all change the verdicts, every such change gets run in shadow first.

Every failing path resolves to halt and escalate: a validation error, a timeout, a missing metric, a model that will not return a usable verdict. The gate never defaults to deploying when it does not know what it is looking at.

In practice, this AI deployment gate has issued verdicts on 26 real pipeline executions, with a median response time of 3.6 seconds, and halted six deploys

How I Measured the Gate

You cannot tell whether a model is helping by watching it work. The verdicts look reasonable, and the reasoning reads well. Every example you check by hand seems about right, none of which tells you whether a far simpler system would have produced the same answers for a fraction of the cost.

To find out, you need scenarios where you already know what the answer should be, and something to compare against.

The scenarios

I built 23 scenarios, each representing a complete signal bundle that the gate receives in production: change metadata, security findings, and environment health. They are not real deploys. They are constructed to cover specific shapes of failure.

Kind	Count	What it tests
Risky	9	Changes that the gate should catch
Adversarial	1	A commit message written to steer the gate
Fail-closed	1	Malformed or missing signals
Benign	8	Changes that should sail through the gate
Absent signal	2	A scanner that never ran, a service with no traffic
Inverted	1	A revert of a bad deploy
Contested	1	Two competent engineers would disagree

Twenty-two of the entries are scored. The contested entry is recorded but deliberately left out of the scoring because I could not defend any answer as correct, and grading based on my own preference would have been dishonest.

Scored against a set of acceptable answers

Each scenario has a set of acceptable verdicts rather than a single right one. A 1,057-line payments change on a Friday evening clearly deserves caution, and reasonable engineers would disagree about whether that means canary or halt. Grading against one answer would penalise the model for a judgement call a human would also make, and the resulting number would measure agreement with me rather than safety.

Two rates, kept apart

The scoring generates two distinct numbers, which should never be combined.

Missed dangerous changes. Out of the 11 scenarios that should have been caught, how many were waved through? This is the rate that hurts you in production.

False alarms. Out of the 11 scenarios that should have sailed through, how many were flagged? This is the rate that makes your team stop trusting the gate.

Averaging them into one score destroys the information. Imagine a lazy gate that returns high risk for every change it sees. It never misses a dangerous change, so its first rate is a perfect zero. It also halts every deploy you attempt, so it is completely useless. A single averaged score would still rate it respectably, which is how you know the average isn't measuring anything worth knowing.

Three repeats per model

Every model ran each scenario three times. Models are not deterministic, and a wrong answer given consistently is a different problem from a correct answer given unreliably. The first needs a better specification, while the second needs you to decide whether you can live with the variance.

This distinction turned out to matter. Claude Haiku failed one scenario that Claude Sonnet passed, and on inspection it was an instability failure rather than a wrong answer; Haiku produced different verdicts for the same input across repeats.

The control

The comparison is against a scoring function I wrote in fifteen lines. It counts attributes: how many lines changed, how many critical findings exist, whether alarms are firing, how recently the service was deployed. It adds up weights and emits a verdict. It reads no prose at all — not the commit message, not the finding descriptions, nothing a model would consider context.

I did not tune it to win. Its job is to establish what you get without a model, so that the model’s numbers mean something. Without a control, “77% acceptable verdicts” sounds good or bad depending on your mood.

What this measurement cannot tell you

Three repeats across 22 scored scenarios is a small sample, and I want to be precise about what that costs before showing any results.

One scenario is worth 4.5 percentage points. More importantly, I ran the same configuration twice with no changes and got missed-dangerous rates of 18.2% and 9.1%. That is the noise floor of this harness.

So any difference inside roughly the 9 to 18 percent band is not a result, and I will not conclude one. The gaps that matter in this article are much larger than that.

What the Numbers Showed

Here is what four models and the scoring function produced across 23 scenarios and three repeats each.

Decider	Missed dangerous changes	False alarms	Acceptable verdicts
Scoring function (15 lines)	27.3% (3 of 11)	9.1% (1 of 11)	81.8% (18 of 22)
Claude Haiku 4.5	27.3% (3 of 11)	9.1% (1 of 11)	77.3% (17 of 22)
Claude Sonnet 4.5	27.3% (3 of 11)	9.1% (1 of 11)	81.8% (18 of 22)
Gemini 3.5 Flash-Lite	9.1% (1 of 11)	9.1% (1 of 11)	90.9% (20 of 22)
Amazon Nova Pro	see note	9.1% (1 of 11)	see note

Raw results for every run are in evals/results/, and the scenarios are in evals/fixtures/

Nova Pro ran during the window when I was building the deterministic floor described in the next section, and its stored result does not record whether the floor was active. That affects both its missed-dangerous rate and its acceptable-verdict count, so I report them as unknown rather than guess. Its false alarm rate is unaffected, since that figure was identical across every model in every configuration.

Three main takeaways from the table:

Fifteen lines of arithmetic matched both Claude models. The scoring function, Haiku and Sonnet each missed exactly three of the eleven dangerous scenarios. A function that cannot read, working only from counts and thresholds, performed as well as a model costing orders of magnitude more per call.

The two Claude models failed on the same scenarios. Every scenario Sonnet got wrong, Haiku got wrong too. All three were security scenarios where a finding existed, yet the model let the change through anyway.

The scoring function failed somewhere else. Only one of its three misses overlapped with the models’. The other two were a low-traffic service where more than a third of requests were failing, but no alarm had gone off, and the first deploy to a service in over a month. Those cases need the numbers read in context, and a function that only adds up points cannot do that.

The models explained themselves

Because the verdicts include reasoning, I could read why each model let those three through. The pattern was consistent: the model noticed the security finding, judged that it predated this change, and concluded it was therefore not this deploy’s problem.

That is a defensible position. An engineer might say the same thing. If a critical vulnerability has been sitting in your dependency tree for three weeks, blocking today’s unrelated README fix does seem unhelpful.

The trouble is that I never told the system which position to take. I had never decided whether a pre-existing finding should block a deploy, and I had not recorded it anywhere. So each model picked its own answer, and both Claude models picked the same one.

A capability gap and a specification gap look different

A capability gap looks like a weaker model failing more often than a stronger one, failing on different scenarios, and failing for reasons that read as confused. You fix this gap by upgrading, or by giving the model better context, or by writing a clearer prompt.

A specification gap looks like a stronger model repeating a weaker model’s mistakes exactly, with the same coherent justification. Nothing is confused. Both models are answering a question you never settled, and answering it the same way.

I had the second gap. I spent time rewriting prompts before I understood that, and the rewrites did nothing, because the prompt was not the problem.

The scoring function is what made the picture clear. On overall score, it tied both Claude models, but it failed in completely different places. The models missed the rule. The scoring function missed the judgement. Without the control, I would have seen two models at 27.3% and reached for a bigger one. With it, I could see that the models were capable of the judgment and short of a rule, while the function had the opposite problem.

The Fix: A Deterministic Floor

The rules to fix the specification gap are just two short ones.

1. If the change carries a critical security finding, the deploy is at least medium risk.
2. If the change carries a finding whose severity has not been scored yet, the deploy is also at least medium risk, because an untriaged finding is an unknown rather than a safe one.

# Simplified from required_floor() in services/decision_service/verdict/floor.py
def required_floor(bundle):
    security = bundle.security.data
    if security is None:
        return None
    if security.critical_count:
        return Floor(MEDIUM, "critical finding")
    if security.unknown_count:
        return Floor(MEDIUM, "unknown severity")
    return None

The floor doesn't produce a verdict; it just sets a minimum. The model may raise the risk above it for any reason it deems fit, but can't lower it beneath. If the model says low and the floor says medium, the verdict is medium, and the override is recorded so you can see it happened.

Both sides get the floor

The floor belongs to the gate rather than to the model. If I had switched it on for the model and left the scoring function without it, I would have been measuring the floor and reporting it as a model improvement. So I ran everything twice (once with the floor off, once with it on) and compared each decider against itself.

Decider	Missed dangerous changes, floor off	Missed dangerous changes, floor on
Scoring function	27.3% (3 of 11)	18.2% (2 of 11)
Claude Sonnet 4.5	27.3% (3 of 11)	0.0% (0 of 11)
Gemini 3.5 Flash-Lite	9.1% (1 of 11)	0.0% (0 of 11)

False alarms stayed at 9.1% for every decider in both configurations. Nothing became more cautious across the board. The improvement is confined to the specific scenarios the two rules were written for.

The scoring function caught one more dangerous scenario than before and then stopped. Both models caught everything.

The most important result is what happened after the floor. All three deciders got the same two rules, and those rules fixed every scenario the models had been missing, because they were security questions. The scoring function still missed two, and neither was about security. In one, a service was getting its first deploy in over a month, along with a runtime upgrade. An engineer would be uneasy about that, but the scoring function was not, because it adds up points for things like firing alarms and change size, and neither scenario scores high on those. Spotting the risk meant reading the numbers in context, which is the part a model is good at.

The models were always capable of the judgement I needed. The problem was that I had also left them a policy decision I had never made myself. Once the two rules made that decision for them, the judgement was all that was left.

That is a principle that should be applied to any system with a model in a decision path: give the model only the judgement that cannot be computed, and constrain it with the part that can.

What the floor refuses to look at

The two rules read security findings and nothing else. Not the diff size, not the paths touched, not the deploy frequency and not the commit message. Here is what that looks like.

Scenario	Floor applied
6,512-line refactor on a healthy service	none
1,057-line payments change	none
Commit message instructing the gate to approve the deploy	none
2-line dependency bump with one unscored finding	medium

A two-line change to requirements.txt gets floored, and a 6,512-line refactor does not. That looks wrong until you consider where each signal comes from.

Diff statistics are produced by a script running inside the repository being judged. The commit message is written by whoever wants the deploy to go out. Both are under the control of the change itself, which means a floor built on either is a floor the change can lower(and that defeats the purpose of the floor). Security findings come from a scanner the change does not control, so they are the only input suitable for a rule that cannot be overridden.

The model still sees all of it, and it should. Diff size and commit messages are useful context for weighing risk. They are simply not safe foundations for the part of the decision that nothing is allowed to argue with.

What It Still Gets Wrong

One scenario in the set defeats every decider I have tested, and I have not solved it.

Picture the worst morning you have had at work. A deploy went out half an hour ago and took the payments service down. Alarms are firing, customers cannot pay, and someone has just pushed the revert change that puts the working code back. That change is the thing standing between you and a resolved incident, and it needs to ship immediately.

Here is what the gate sees: 1,057 lines changed, payment paths touched, two alarms firing and a 7.4% error rate on the target service, and a deploy that went out thirty-six minutes ago. Every signal it has says this is dangerous. Measured against the rules, this is one of the riskiest changes in the entire scenario set.

So the gate halts it, and the outage continues for as long as the halt holds. In practice, there is a way out. Every halt sends a human an email containing an override command, so an engineer can run the command and ship the revert change.

What the gate cannot see is that this change is the fix rather than the cause. A broken deploy and the repair for a broken deploy produce almost identical signals, because both touch production code on an unhealthy service in a hurry. The only thing separating them is why the change was made, and the gate has no access to why.

Claude Haiku, Claude Sonnet, Amazon Nova Pro and Gemini Flash-Lite all fail it. So does the scoring function. It is the single false alarm behind the 9.1% that appears in every row of both result tables.

Catching it would require reading intent. A human knows this is a repair because they know what happened an hour ago, and because the commit message says so. The commit message is exactly the input the floor refuses to read, for the reasons described above, and the models that do read it still do not weigh it heavily enough to overcome four signals pointing the other way.

So the design decision that makes the floor trustworthy is the same one that leaves it powerless here, and for now a human covers the gap. The fix I want is to give the gate access to intent, most likely by passing recent deploy and rollback history as an explicit signal so a repair looks structurally different from a cause. I have not built it yet. If you want to attempt it, the scenario is in the repository as revert_of_a_bad_deploy.

Five Things Worth Stealing

Most of what I learned here is portable to any system with a model making decisions, whether or not deployment is involved.

Build the dumb control before you tune anything. Write the stupidest possible version of your decision-maker, something that counts attributes and applies thresholds, and measure it alongside the model. Without it, you have no way to know whether your model is contributing, and you will spend weeks improving prompts that were never the constraint.

Keep your error rates separate. Work out the two ways your system can be wrong, measure each independently, and never average them. For a gate, the two are waving through something dangerous and blocking something harmless. For a classifier, they are false negatives and false positives. For an agent, they might be acting when it should have asked and asking when it should have acted. A single combined number can be gamed by a system that always answers the same way, which means it is not telling you anything.

When a bigger model repeats a smaller one’s mistakes, stop upgrading. Identical failures across different capability levels point to your specification rather than the model. Read the reasoning behind the failures. If the explanations are coherent and consistent, you are looking at a question you never decided, being answered by something that had to decide it for you.

Write the countable rules down, and let the model work above them. Anything you can express as a threshold on a number should be a threshold on a number, enforced deterministically. Leave the model the part that needs weighing rather than counting, which is also the part you cannot write down.

Make the thing that decides unable to act. Whatever your agent reads, assume some of it is controlled by someone who wants a particular outcome. Enforce the boundary with permissions rather than with instructions in the prompt.

Conclusion

I set out to build an AI deployment gate that would decide whether a change was safe to deploy, and I assumed the hard part would be getting a model to reason well about risk. The hard part was working out what I was actually asking it.

A function that counts attributes, a small model and a large one all let dangerous changes through at exactly 27.3%. The two models failed on the same scenarios, and when I read their reasoning, both had quietly resolved a question I had never answered: whether a security finding that predates a change should block that change from shipping. Neither was confused. They were filling a gap I had left.

Two rules closed the gap: a critical finding means at least medium risk, and an untriaged finding means the same. A few lines of code, enforced deterministically, took one specific decision away from the model entirely. The models went to zero misses. The scoring function improved by one and still missed two(both cases that needed weighing rather than counting).

The gate still gets one thing wrong. It cannot tell a repair from the thing that caused the damage. This is still open and is in the repository with the evidence attached.

If you are putting a model somewhere it makes decisions, the thing I would most want you to take from this is to write the stupid version first and measure against it. It is an afternoon of work that tells you whether your model is contributing anything, and it is the only way to tell the difference between a model that needs replacing and a question you never finished asking.

The code, the scenarios, the labels, the raw results, 84 architectural decisions and 31 documented failures are all public at github.com/poly4concept/ai-pre-traffic-gate. Every number in this article is reproducible from the harness. If you run it against your own setup, or you think one of my scenarios is wrong, I would genuinely like to hear about it.

Thank You for Reading

You can follow me on LinkedIn and subscribe to my YouTube Channel, where I share more valuable content. Also, let me know your thoughts in the comments section.