Evaluate an AI coding tool by giving it a real work item from your own backlog and putting the result through your normal review. Everything else follows from that: score the output against the same rubric you use for human code, check how the tool handles your conventions and your CI pipeline, and measure rework and defect rate rather than how many tasks it finished. Public benchmarks and vendor demos tell you a tool is broadly capable. They do not tell you how it behaves in a codebase with your history, your test suite and your review culture, and that is the only question a buying decision rests on.
Write down how your team works before you open a vendor site
The most common failure in this kind of evaluation is treating it as ordinary software procurement. Teams read feature lists, sit through demos, compare pricing tiers, and pick the tool with the longest list of capabilities. Lumenalta’s guide to evaluating AI coding tools makes the point that this misses the variable that decides the outcome, which is fit with how the team already works. A platform engineering group weights criteria quite differently from a product team: pipeline integration, Terraform and Kubernetes support and repository-wide reasoning matter far more to them than clever object-oriented refactoring in a single file.
So the first artefact of a useful evaluation is two or three sentences describing your own work, written before anyone looks at a vendor page. What language and framework carries most of your value. Whether the work is mostly new features, mostly maintenance of code nobody currently understands, or mostly incident response. Who reviews what, and how strict that review is. Where your deployment gates sit. That paragraph becomes the filter. Half the tools on any shortlist will fail it on the first read, which saves you the demos.
Adoption is not the question any more. Most teams are already using something. The live question is whether the thing they are using suits the work, and whether anybody has checked.
Decide whether you are buying autocomplete or an agent
These are two different products with two different sets of criteria, and the market has now formalised the split. Gartner renamed its category from AI Code Assistants to Enterprise AI Coding Agents, and the 2026 Magic Quadrant under the new name covers 12 vendors. The change in criteria is the interesting part. The old category measured passive, in-editor assistance: suggestion quality, acceptance rate, keystroke latency. The new one measures whether an agent can take a defined task and finish it without being led by the hand.
That gives you a concrete list of things to ask an agentic vendor, and they are not things a demo volunteers. What share of defined tasks does the agent complete end to end without further prompting, and on what class of change. Does it read your pipeline output directly, or does someone have to paste failures back into a chat window. Which events does it log, and are those logs structured and exportable in a form your compliance tooling can actually ingest. Can approval gates be set at task, file or pull request level, and where does an escalation go when the agent exceeds its scope. What bounds execution, so it cannot edit files outside the work item or trigger a pipeline run nobody asked for.
If what you want is an assistant that completes the line you were already typing, most of that is irrelevant and you should not pay for it. The decision of when an agentic tool earns its place over inline autocomplete is worth settling before the trial rather than during it, because it changes what you are measuring. It is also worth deciding what bounds the agent will run inside, since the guardrails that hold are the ones in the runtime rather than the ones written into a prompt.
Push one real work item all the way through
Benchmarks are a sanity check. Pass rates on synthetic tasks, competitive programming scores and agentic leaderboards tell you a tool is not obviously broken. They predict very little about your repository, your conventions and your test suite, because none of those were in the benchmark.
The test that does predict something is boring: take one ordinary work item and run it through your normal process with the tool in the loop, then put that change through exactly the same linting, dependency scanning and pull request review rules any other change would face. Then count what it cost. How many review comments did the change attract compared with a typical one. How many revisions before it merged. How many follow-up fixes landed in the week after. That comparison is the number you are buying against, and no vendor can give it to you.
A variation worth running if you have several candidates: pick your team’s five most frequent daily jobs, something like bug investigation, feature scaffolding, review assistance, test generation and refactoring, then hand three developers the same real backlog items with one tool each. You get a comparison across the work you actually do rather than across the work the vendor chose to show you. Where the output touches anything that has to be verified, the same logic applies as with checks that come from outside the model: the tool cannot be the judge of its own work.
Score the output the way you score code
Acceptance rate on its own is a soft metric. Somebody accepting a suggestion means they did not object to it on screen, not that it was right. The more informative version is acceptance rate plus rework after acceptance, measured against a small set of cases that reflect your codebase patterns, and graded on a rubric you already use: correctness, readability, maintainability.
The single cheapest diagnostic in the whole exercise is making reviewers label why they rejected something. Wrong logic. Breaks our style. Adds risk. Unclear. Four buckets, one click, and after a fortnight you can see the shape of the tool’s weakness. A tool that is mostly rejected for style is a configuration problem you can fix with project rules. A tool that is mostly rejected for wrong logic is not a configuration problem. This only works if somebody is genuinely reading the changes, which is why the diff read that vibe coding skips has to be part of the trial rather than something you reinstate afterwards.
The capabilities that matter beyond speed
Speed is the easiest thing to demonstrate and the least useful thing to compare, because every candidate will look fast on a clean example. Glean’s framing of evaluation beyond speed weights six capabilities equally against it: how well the tool absorbs your systems and conventions, whether it improves quality past the first version it produces, how deeply it fits into your existing review and CI/CD flow, whether it helps a team rather than one individual, how it handles security and governance, and whether you can measure its effect on delivery with real data.
Two more dimensions come out of treating an assistant as a production system rather than a novelty. The first is latency as a contract with the user: past a certain response threshold, people stop reaching for the tool, and adoption collapses without anyone filing a complaint. The second is effective context, which is not the same as the advertised token limit. Retrieval, symbol graphs and indexing decide whether the tool looks at the right files. A very large context window that selects the wrong three files is worse than a small one that finds the correct one, and you can only tell which you have by watching it work in a repository with real history. Our survey of the current tool landscape covers the categories these products fall into.
Security, data handling and governance
This is where an evaluation should be least improvisational, because the answers are contractual and checkable. Where does your code go, who retains it, for how long, and whether it can reach a training set. What the tool can read and what it can write. Whether it fits your existing secure development lifecycle and assurance programme, or sits outside it. Witness AI’s account of the security risks of AI coding assistants is a reasonable checklist to test a vendor’s answers against.
The AI readiness framework from Augment Code takes the same ground from the buyer’s side, scoring six dimensions from nought to three across strategy, data, infrastructure, skills, governance and delivery. Its more uncomfortable questions are the valuable ones: do you have documented, stable APIs for the systems the tool would touch, can AI safely perform write actions against them, and has one team run a contained pilot before anyone talks about rolling it out. A low score is not a reason to avoid the tools. It tells you which order to fix things in. The specifics of what to write down belong in a policy that covers both security and ownership, settled before the trial ends rather than after.
Measure outcomes, not completed tasks
For agentic tools especially, task count is a vanity number. An agent that opens thirty pull requests, of which nineteen need rework and two cause incidents, has made the team slower while producing an impressive dashboard. DX’s research on measuring AI code assistants and agents points at the measures that survive contact with reality: acceptance rate, rework, defect rate, review time, test coverage and production incidents.
Two of those deserve special attention because they move in the wrong direction quietly. Review time is where the cost of generated code lands, since somebody has to read what nobody wrote. And defect rate after merge is the only measure that distinguishes genuinely faster delivery from moved work. If review time and defect rate both hold steady while throughput rises, you have a real gain. If throughput rises and review time rises with it, you have shifted effort from writing to reading and should be honest about that trade-off.
How long to run the trial, and what it will cost
The sources disagree here, and the disagreement is worth understanding rather than resolving. One argument for a two-week evaluation sprint rests on churn: these tools change materially every couple of weeks, so a three-month process is partly evaluating software that no longer exists by the time you decide. The counter-argument is that a fortnight measures novelty. Rework, maintenance burden and the cost of code nobody understands show up later, and a pilot that ends before the first awkward bug in generated code has told you the cheap half of the story.
A workable compromise is a short sprint to decide whether a tool is a contender, then a longer run with one team on real delivery work to decide whether to expand. Keep the scoring sheet across both, so the second phase extends the first rather than starting again.
On cost, be careful with any number you read, including the ones in comparison round-ups. Pricing in this market has been rewritten repeatedly, with seat-based, usage-based and hybrid models sitting side by side, and DX’s guide to assistant pricing and return on investment is more useful for its method than its figures. What matters for a budget is whether your spend is bounded by seats or driven by tokens, because agentic work consumes far more of the latter than autocomplete does, and a per-seat estimate built from assistant usage will understate an agent by a wide margin. Model the cost against the work you tested, not against the plan name. The working habits that make results repeatable are also part of the cost picture, since a tool used carelessly generates review load that no licence fee captures.
Written with AI assistance and reviewed under our editorial policy. Vibe coding changes quickly; details reflect sources current on 2026-09-30. Some passages are the author's opinion. Spotted an error? tell us.
Frequently asked questions
- Are public benchmarks worth looking at when choosing an AI coding tool?
- They are a sanity check that a tool is broadly capable, and nothing more. Benchmark tasks are not your codebase, your conventions or your test suite, so a high score does not predict how the tool behaves on your backlog. Use them to shorten a list, then test on real work.
- How long should a trial run before we decide?
- Opinion is split. One school argues for a two-week sprint because these tools change materially every couple of weeks, so a long process evaluates software that has already moved on. The other argues that rework and maintenance cost only surface later, which favours a longer pilot with one team on real delivery work.
- What is the single most useful thing to measure?
- Rework after acceptance, paired with review time. Acceptance rate alone only records that nobody objected on screen, while rework and review time show where the effort actually went. If throughput rises and both of those hold steady, the gain is real.
- Do we need different criteria for agents than for in-editor assistants?
- Yes, and the analyst category now reflects that split. An assistant is judged on suggestion quality and response time, while an agent is judged on end-to-end task completion, pipeline integration, approval gates, execution scope limits and exportable audit logs. Buying one against the other's criteria is how teams end up disappointed.