Short answer: An AI agent's success report is a claim, not a fact. I run a meaningful part of my company's operations on AI agents: nightly content jobs, cost tracking, code review, research for sales material. For the past month I have kept score of what independent verification catches, and the score is uncomfortable: every single time a genuinely independent check ran, it found something real. Not sometimes. Every time.
This post is the evidence, incident by incident, and the operating rules that came out of it. If you are an owner deciding how far to trust AI in your own operations, my answer is: further than you think, and with far more verification than you have.
The job that reported retries while shipping nothing
I run a nightly job system with a health log — the kind of thing that tells you a job "failed 3 times, escalate." One of those jobs was a nightly content generator. Its health entry read six recent fails. Six retries over a stretch of days looks routine. Flaky network, a rate limit, whatever. Nothing about it says dead.
I ran an adversarial audit of the whole 16-agent system: eight finder agents, each paired with an independent verifier agent that re-checked a different evidence path. One verifier ignored the health log entirely and went to the downstream repository — the place the content was supposed to actually land — and checked the pull request history. The last PR that had actually merged was 13 days old.
The job had shipped zero output for 13 days. Every night it produced a draft PR that nobody merged, logged the run, and moved on. "6 consecutive fails" and "13 days of zero output" described the exact same broken job — one framing from the system's own logs, one from ground truth. If I had only read the logs, I would have filed it under routine flakiness.
That is the whole thesis in one incident. The log is what the agent says about itself. The artifact — the merged PR, the delivered email, the written database row — is what actually happened. They are allowed to disagree, and when they do, the log loses.
Tests that could never fail
The most expensive category of false "done" is the AI-written test that passes because it cannot do anything else.
In an AI-heavy codebase I operate, this happened three separate times in the same period. Once, a test compared a string to itself. Once, a golden-image test compared the reference file against the reference file, instead of against the rendered output it existed to check. Once, a test's failure condition was mathematically unreachable. All three shipped green. All three produced confidence worth precisely nothing.
The related failure is older code rotting underneath green tests. During a pre-deploy sweep on LinkStorm — a client codebase we have worked in since 2023 — a reviewing agent refused to accept a passing suite at face value and traced back through git history. An earlier PR had split one method, isExcluded(), into two new ones and moved the production call sites over. Three tests still stubbed the old method name. PHPUnit does not error on an unstubbed method; it silently returns false. So those tests had been asserting against a no-op — exclusion logic could have stopped firing in production and no test would have flagged it. Two years of fixtures had drifted while every run stayed green: a passing suite guarding untested behavior.
The fix that stuck is what I now call the mutation probe, and it is a gate, not a spot-check: before any test-adding change is trusted, deliberately break the exact behavior the test claims to guard — hard-code the literal, remove the guard, flip the flag — rerun, and watch the test go red. When we ran that probe honestly on one suite (removing a single .lowercased() call), it produced five real failures. A test has proven nothing until you have seen it fail on a known-bad input.
The sharpest version of this arrived the same week. A coding agent opened a PR whose body claimed a specific test had run and passed, "1/1." The reviewing agent's mechanical analysis showed that test could not deterministically terminate at all — no run of it could ever have printed a pass. It was the third non-reproducing test claim in about 24 hours, and all three sat in a lane the CI pipeline did not independently execute. The process fix we adopted: in any lane CI does not cover, a PR must paste the raw terminal tail of the test run, not a prose summary. A sentence saying "tests passed" is cheap to fabricate, even unintentionally; a raw terminal tail from a run that never finished is not.
The fallback that became the only path
I recently dug into a nightly job that scraped per-project Vercel billing through a browser. It had a sensible fallback chain: try the browser integration, then try a debug port, then fall back to a cruder API-only total. For 25 straight days it had served the crude fallback — no error, no alert, report delivered nightly.
The bug was not in the fallback logic. The fallback logic worked perfectly every night. The bug was that the job's primary method depended on a browser integration that only exists in interactive sessions — a headless, unattended run never gets it, whether Chrome is running or not. I proved this by running the identical command interactively and headless and diffing the available tools. The "primary" path was structurally impossible in the context the job actually ran in. Degraded mode was not a failure state. It was the only path the job ever had, and nothing anywhere said so.
The lesson generalizes past AI: when an unattended job has a fallback, ask what percentage of real runs take it. If you cannot answer, assume 100% until proven otherwise. A capability you tested by hand at your desk can be categorically absent at 3 a.m.
Two characters that deleted my monitoring
The pettiest incident is my favorite, because the mechanism fits in a shell one-liner. I found a job whose command was structured roughly as (step1 && step2 && step3) || echo ok. That trailing || echo ok means the compound command exits 0 no matter what any real step did. Every failure, converted to success, forever.
The job's escalation path fires after 3 consecutive failures. It could never fire. The run log showed 154 consecutive "ok" entries — while the underlying step was actually failing. A status wrapper that reports success on error is strictly worse than no monitoring: it manufactures the confidence that suppresses the escalation it exists to trigger. The fix was deletion — let the real exit code propagate — plus a rule that health checks read raw exit codes, never a self-reported status line.
A smaller cousin: a price-parsing regex in a cost report silently dropped any row it could not match. A trimmed decimal — "$2.1" instead of "$2.10" — was enough. The dropped row happened to be the single most expensive project in the report. The report still "ran clean" every night.
A month of keeping score
Those are the dramatic ones. The statistic that changed how I operate came from reviewing one month of my agent pipeline's own history. Every content and analysis lane in that pipeline runs an independent check pass before anything ships. Here is what those checks caught:
| Day | What independent verification caught |
|---|---|
| Day 1 | A fabricated stat in our sales deck: "1.37M+ organic clicks" attributed to one client — the research agent had conflated it with a different product's install count that happened to also be 1.37M |
| Day 8 | 14 issues in one content draft |
| Day 15 | 4 errors in another draft, plus a research CLI fabricating a CORS claim outright |
| Day 16 | 2 AI-writing tells in copy that read fine to its author agent |
| Day 17 | A subagent reporting wrong character counts |
| Day 18 | Two subagents independently agreeing on the same wrong root-cause diagnosis for a search-ranking decay |
That last row deserves a beat. Two agents, no shared context, same wrong answer — because they were fed the same misleading data slice. Agreement between AI agents is not evidence when they share correlated blind spots. More agents voting is not verification. A check with genuinely different inputs and a different method is.
And the pattern across the whole table: the verification layer did not occasionally earn its cost. It found something real on every date it ran. Single-source AI output in my pipeline has an error rate high enough that shipping it unchecked is not a shortcut, it is a liability.
The rules we run now
Four rules, each paid for by an incident above.
1. Verify against the artifact, never the log. "Done" means the merged PR exists, the email arrived, the row was written — confirmed by looking at the PR, the inbox, the table. An agent's transcript saying it did the thing is testimony, not evidence. The 13-day content outage lived entirely in the gap between those two.
2. Review agents get no shared context. A verifier that reads the worker's reasoning inherits the worker's blind spots. Ours start from the task and the artifact only. And after the twin wrong diagnoses: agreement raises confidence, it never closes the question. Disagreement is the useful signal — it comes to me.
3. Every check must be shown capable of failing. A test proves nothing until you have watched it go red on a known-bad input. A monitor proves nothing until you have fed it a failure and seen the alert. 154 straight "ok" entries were not a healthy job; they were a check with no power to fail. Negative controls are not paranoia, they are calibration.
4. "The AI wrote the fix" and "the problem is solved" are different milestones. Track them separately, celebrate them separately. The gap between them is where every incident in this post lived.
None of this is an argument against delegation. The agents stay; the output is worth it, and the verification layer costs a fraction of what it catches. But if you are wiring AI into your own operations, budget for that layer from day one — an independent path to ground truth for every lane that matters. The alternative is a dashboard full of green lights wired to nothing, and you will find out which ones during the worst week to find out.
Shyam Verma founded Ready Bytes in 2009 and has been building software since 2005. He writes about legacy modernization, migrations and applied AI at readybytes.in/blog.
Shyam Verma
Full Stack Developer & Founder
Shyam Verma is a seasoned full stack developer and the founder of Ready Bytes Software Labs. With over 13 years of experience in software development, he specializes in building scalable web applications using modern technologies like React, Next.js, Node.js, and cloud platforms. His passion for technology extends beyond coding—he's committed to sharing knowledge through blog posts, mentoring junior developers, and contributing to open-source projects.
