Can AI write my test suite for me?

It can write most of the code in one, but not the assertions that make it a test. Generated assertions are derived from what the code currently does, so they encode existing bugs as expected behaviour. The workable split is to write oracles by hand from the requirement and generate the scaffolding around them.

What is the oracle problem in testing?

The oracle is the part of a test that decides whether an output is correct. Generating inputs is mechanical, but deciding correctness is a claim about intent, and intent is not recoverable from the implementation. This is why AI helps most on tasks where an oracle already exists elsewhere — in a requirement, a passing test or an invariant.

Where does AI help most in quality engineering?

Triage and maintenance, not authoring. Clustering many failures into a few causes, repairing locators that broke without the element changing, and finding coverage gaps are all pattern work that needs no view on correctness — and they recur for the life of the suite, whereas writing a test is a one-off.

Forge · 9 September 2026

What AI changes about software testing, and what it does not

AI generates tests quickly, but generated tests assert current behaviour, which encodes existing bugs as expected. Where AI genuinely reduces QA effort, where the oracle problem stops it, and how to tell the two apart.

4 min read

AI is very good at producing test code and cannot, even in principle, tell you what the software is supposed to do. Almost everything useful about applying it to quality engineering follows from that one asymmetry.

The oracle problem

A test has two parts: an input, and a judgement about whether the resulting output is correct. The second part is the oracle. Generating inputs is mechanical and AI is excellent at it. The oracle is a claim about intent, and intent is not present in the code.

This is not an argument against generating tests. It is an argument about where the review effort belongs: on the assertions, not on the scaffolding. The generated code around the assertion is usually fine and rarely worth much attention. The assertion is the entire test.

Where the time actually goes

The case for AI in testing is usually made against writing tests, which is the part of quality engineering that consumes the least time. Maintaining a suite and triaging its failures consume far more, and they are also where AI helps most — which is not the pitch that gets made.

Illustrative proportions, not measured — the split varies by team and by how flaky the suite is. The ordering is the durable part: authoring is the smallest of the three, and it is the one most tooling is sold against.

What AI genuinely does well here

Task	Why it works
Clustering failures	Two hundred red tests are often six causes. Grouping by failure signature is pattern matching over text, with no judgement about correctness required.
Repairing brittle locators	When a selector breaks but the element still exists, the intended target is recoverable from context. The oracle is unchanged — only the path to the element moved.
Proposing cases from requirements	A written requirement is an oracle expressed in prose. Turning it into candidate cases is translation, and the source of truth is external to the code.
Finding coverage gaps	Comparing branches exercised against branches present is analysis over structure, and needs no view on what is correct.
Generating input data	Boundary values, malformed payloads and awkward unicode are input-side work. The oracle for most of them is "does not crash", which is already known.

Tasks where the oracle already exists, or is not needed

The pattern is consistent. AI performs well wherever the oracle is supplied from outside — by a requirement, by an existing passing test, by a property that must hold — and performs badly wherever it would have to invent one.

What it does not fix

Ambiguous requirements. A requirement that two engineers read differently will produce tests that encode one reading, confidently and without flagging that a choice was made.
Flaky infrastructure. A test that fails one run in twenty because of a race in the environment is not a test problem, and generating more tests against it multiplies the noise.
Deciding what matters. Coverage is a measure of what was executed, not of what was worth executing. Nothing in a model knows which paths carry the money.
Accountability. When a generated suite passes and production breaks, the person who reviewed the assertions owns it. Keeping that reviewable is a design constraint on how much you generate at once.

A workable division of labour

01Write the oracle by hand, from the requirement. This is the part that carries the intent and it is usually a few lines.
02Generate the scaffolding, the fixtures and the input permutations around it. This is the bulk of the code and the least of the thinking.
03Review assertions individually and scaffolding in bulk. Reviewing generated tests uniformly is how teams end up rubber-stamping.
04Never generate a test against unverified behaviour. If the code has not been confirmed correct, a generated test freezes whatever it currently does.
05Point the effort at triage and maintenance first. That is where the recurring cost sits, and the oracle question does not arise there.

Used this way the gain is real and fairly large. Used as "generate a test suite for this repository", it produces a suite that is expensive to maintain, agrees with every bug it was pointed at, and is trusted more than it has earned.

Common questions

Can AI write my test suite for me?: It can write most of the code in one, but not the assertions that make it a test. Generated assertions are derived from what the code currently does, so they encode existing bugs as expected behaviour. The workable split is to write oracles by hand from the requirement and generate the scaffolding around them.
What is the oracle problem in testing?: The oracle is the part of a test that decides whether an output is correct. Generating inputs is mechanical, but deciding correctness is a claim about intent, and intent is not recoverable from the implementation. This is why AI helps most on tasks where an oracle already exists elsewhere — in a requirement, a passing test or an invariant.
Where does AI help most in quality engineering?: Triage and maintenance, not authoring. Clustering many failures into a few causes, repairing locators that broke without the element changing, and finding coverage gaps are all pattern work that needs no view on correctness — and they recur for the life of the suite, whereas writing a test is a one-off.

When to use an AI agent, and when a script is the right answer

Agents are chosen for task complexity when the deciding factor is input variance. Where each one belongs, why an agent's failures are harder to catch than a script's, and the hybrid that usually wins.

4 min read

WhatsApp automation in India: what you are actually allowed to send

Opt-in, the 24-hour window and template approval decide what you can send on WhatsApp, not your CRM. What each rule means, why TRAI's DLT registration does not apply, and how a contact list shrinks to a reachable audience.

5 min read

All insights.

Browse insights Talk to us

Or write to us at [email protected]