Testing AI-generated code without writing every test

The failures cluster, which is what makes this tractable

If AI-generated code failed at random you would have to test all of it evenly, which is impossible at the speed the code now arrives. It does not fail at random. The CodeRabbit State of AI vs Human Code Generation report compared AI-coauthored pull requests against human-only submissions.

Treat this as a direction of travel rather than a measurement. Other studies land in very different places. The reason the range is so wide is that each study counts something different, so arguing about the exact multiplier is a waste of a morning. What every one of them agrees on is the shape: error handling, edge cases, boundary conditions, concurrency and security boundaries. Skyramp’s analysis frames this well, arguing that model blind spots are distributional rather than novel, so the same weakness repeats across every function the model writes.

So the practical consequence is not “write more tests”. It is: aim the tests at the categories where the model is predictably weak, and make sure the aiming is not done by the model.

Tautological tests pass, and prove nothing

The single most repeated warning across the practitioner writing on this is also the easiest one to ignore, because ignoring it feels efficient. Do not let the same model write both the code and the tests. When it does, both outputs inherit the same misunderstanding, and the tests then verify what the code does rather than what it should do.

There is a good worked example of this in a developer’s account of shipping AI code to production: the model that wrote a shipping cost function also generated its tests, and those tests checked that various weights were handled, that negative inputs were rejected, and that currency was formatted correctly. All of them passed. None of them could catch the units conversion bug, because the tests encoded the same wrong belief about pounds and kilograms that the implementation did. A full green suite, a real financial error, and no signal anywhere in between.

This is not a quirk of one tool. It is the verification paradox that AI research has been circling for a while: a critic or reviewer drawn from the same model family shares the generator’s priors and blind spots, so self-verification can leave you worse off than no verification, and reliable improvement needs external checks rather than a second opinion from the same source. Switching providers helps, but less than you would hope. Independence is something you have to engineer, not something you get by opening a second tab.

Write the checks before you prompt

The cheapest way to get an external specification is to write it before the code exists, while you still have nothing to agree with. Write failing tests that define the behaviour you want, then pass those tests to the model as the constraint on the implementation, then run the suite immediately and note what the first attempt got wrong. Full Stack Data Solutions describes this loop as specification tests first, implementation second, full suite run third, adversarial review fourth, commit last, and reports the whole cycle taking about three minutes in practice for a small unit of work.

The volume here is manageable because a specification is much smaller than a test suite. You are writing the handful of assertions that express the contract, not the fifty that cover the permutations. Those fifty are exactly the work a model is good at, once the contract is fixed and it cannot quietly redefine it.

This also happens to be the highest-value thing you can put in a prompt. A failing test is a precise, executable statement of intent, which is why it beats several paragraphs of prose describing the same requirement. That is the same argument as writing a clear prompt with a concrete example, taken one step further: the example runs.

Properties and invariants do the work examples cannot

Example-based tests check the cases you thought of. The model’s weakest area is the cases nobody thought of, which means example tests and model blind spots fail in the same places. Property-based testing inverts that. Instead of asserting that a specific input gives a specific output, you assert a rule that must hold for every input, and let the framework generate hundreds of inputs trying to break it. Round-tripping, ordering, idempotence, conservation of totals, monotonicity: these are the statements a units bug cannot survive.

The research on model-generated tests explains why this matters so much here. Studies of large-scale LLM test generation have found models asserting against their pre-training knowledge of what a function called calculateShipping probably does, rather than against what this implementation actually does. There is a related and very specific weakness around boundaries: when a change shifts a conditional edge, model-written tests tend to check a representative invalid input and a valid input comfortably far from the boundary, and miss the value that sits exactly on it. Fuzzed and generated input does not have that preference.

One team’s post-incident rule, after AI-written unit tests passed and production still broke, was to require at least one property-based or fuzzed test per function plus contract tests validating invariants rather than example outputs. That is a rule you can enforce in review, and it costs one test per function rather than a suite.

Mutation testing tells you whether your suite can fail

Everything above still leaves one question open: is the suite you now have any good? Coverage will not answer it, because coverage measures which lines ran, not whether anything would have noticed if they were wrong. Mutation testing answers it directly. You deliberately break the code in small ways, one at a time, then check whether any test fails. A mutant that survives is a hole in your suite, named precisely.

This is also the one place where letting a model write tests at volume is straightforwardly safe, because the mutant supplies an external target the model did not choose. Meta applied exactly that principle, using models to generate mutation-derived tests, as described in a trial it ran. InfoQ’s write-up of the same work covers the same trial, at a scale where a human writing every test is simply not an option.

You do not need Meta’s infrastructure to use the idea. Running an off-the-shelf mutation tool against the two or three modules that would hurt most if they were wrong will tell you more about your test suite in an afternoon than a coverage report ever will.

Agent-written tests are scaffolding, not evidence

Agentic tools increasingly write tests without being asked, and it is worth being clear about what that buys you.

So an agent’s tests are useful. They catch typos, they pin behaviour against future changes, and the act of running them forces the agent to confront its own output rather than hand you something that has never executed. They are not evidence that the code does what you asked, because they were written by the thing that decided what the code does. Read them as regression scaffolding, and read the assertions rather than the pass count. That is a lighter version of the same discipline as reading the diff: you are not re-deriving the work, you are checking that the claim matches the intent.

The division of labour that holds

What survives contact with real work is a split by origin rather than by volume. You own the parts that encode intent, and you delegate the parts that encode coverage.

• You write, by hand and before prompting: the contract, the invariants, and the two or three assertions that would catch the failure you most fear.
• The model writes: the example cases, the fixtures, the boilerplate, the regression tests around behaviour you have already confirmed by running it.
• Something external decides: mutation runs, property-based generators, type checking, linting, dependency and secret scanning, and a review pass from a different provider’s model where the correlation data gives you at least some independence.
• The pipeline enforces: nothing merges on a green suite alone if the suite has never been mutated, and the gates live in continuous integration rather than in a prompt, for the same reason that guardrails only work in the runtime.

There is one honest limit to all of this. If you cannot state what the code should do, no amount of testing machinery will save you, because every check you generate will be derived from the implementation and will therefore agree with it. That is the real cost of the fast route, and it is covered further in the risks and limits of vibe coding and in the working practices under best practices. The specification is the part of the job that has not been automated. Everything downstream of it can be.

---

Written with AI assistance and reviewed under our editorial policy. Vibe coding changes quickly; details reflect sources current on 2026-09-20. Some passages are the author's opinion. Spotted an error? tell us.