A flaky test passes and fails on the same commit. Nothing in the product changed between the two runs. The test changed its mind.
Teams pay for this twice. First in reruns, then in trust. Once engineers learn that a red run means nothing, a real regression gets the same shrug as a stale selector.
This guide sorts flakiness by root cause, because each cause has a different fix. Retries help with one of them and hide the other three.
What causes flaky tests
Four causes explain almost every flaky failure in an end-to-end suite.
1. Timing and race conditions
The test acts before the application is ready. A button exists in the DOM but its click handler is not attached. A row renders before the fetch resolves. A fixed sleep passes on a fast machine and fails on a loaded CI runner.
Fix: wait for the state you need, not for a number of milliseconds. Wait for the network request, the disabled attribute, or the text you expect. Delete every fixed sleep you own.
2. Locators that describe the markup
A selector such as div.card > button:nth-child(2) records where an element sat on the day the test was written. A class rename or a new wrapper breaks it, and the failure says only that the element was not found.
Fix: address elements the way a user finds them, by role, label, or visible text. Where the product gives no stable handle, add a test id in the application code and treat it as an interface.
3. Shared state between tests
Two tests use the same account. One test leaves a modal open, a filter applied, or a record behind. The failure follows test order, so it appears when the suite runs in parallel and disappears when you run the file alone.
Fix: give each test its own data and its own session. Create the record the test needs at the start of the test, and never depend on a record another test made.
4. The environment under the test
A third-party script times out. A staging deploy lands mid-run. A CI container gets less CPU than your laptop, so animations and timeouts behave differently.
Fix: stub third-party calls that are not part of the assertion, pin the environment the suite runs against, and read run history before you blame the test.
How to tell a flaky test from a real bug
Do this before you change the test, because the cheapest wrong move is to make a genuine failure quiet.
- Rerun the same commit. A pass on rerun means the test is unstable, not the product.
- Read the failure evidence, not the stack trace. A screenshot at the failing step usually names the cause.
- Check the failure rate across the last thirty runs. One failure in fifty is a different problem from one in three.
- Run the file alone. A pass alone and a failure in the suite points to shared state.
What a testing tool can repair for you
Two of the four causes are structural, and no tool fixes them for you: shared state is a test design problem, and a broken environment is an infrastructure problem. Timing and locators are different. Both are mechanical, and both can be handled at run time.
Momentic tests are plain YAML in your repository. Steps describe intent, so a step says click the "Checkout" button, not a CSS path. The runner resolves that intent against the live page on every run, which removes the whole class of failures that a markup change causes. When a target still moves, self-healing repairs the step and the repair arrives as a diff you review in a pull request.
Timing is handled the same way. The runner waits for the page to settle before it acts, so the suite does not need fixed sleeps.
steps:
- action: navigate
url: "{{env.BASE_URL}}/cart"
- action: ai_action
description: click the "Checkout" button
- action: ai_check
description: the order confirmation shows an order numberAn unresolved flaky test does not have to block the merge queue while you work on it. Quarantine keeps it running and keeps its result out of the gate, so you still collect evidence.
Self-healing approaches, side by side
Every vendor in this space says self-healing. The differences that matter are where the test lives and who approves a repair.
| Tool | Where the test lives | How it handles a moved element |
|---|---|---|
| Momentic | YAML in your repository | Intent-based steps resolved at run time, plus healing that lands as a reviewable diff |
| Autonoma | Vendor platform | Managed AI agents maintain the tests for you |
| QA Wolf | Vendor platform, with Playwright code you can export | A managed service fixes broken tests as part of the contract |
| Functionize | Vendor platform | ML models pick a replacement element from historical run data |
| Maestro | YAML in your repository, mobile only | Built-in waits and retries; there is no AI repair of a step |
| Playwright | TypeScript in your repository | Auto-waiting and web-first assertions; you write the new locator yourself |
Read the middle column first. A repair you cannot read in a pull request is a change to your test suite that no engineer approved.
A plan for a suite that is already flaky
- Measure first. Rank every test by failure rate over the last thirty runs.
- Quarantine the worst offenders so the gate becomes trustworthy again today.
- Fix by cause, top of the list down. Timing and locators first, because they are the cheapest.
- Delete tests that assert nothing anyone reads. A flaky test with no owner is a test to remove.
- Keep the gate honest. A green run must mean the product works, or the suite goes back to being decoration.
Momentic runs the same tests locally and in CI with npx momentic run, and the exit code gates the merge. The quickstart takes about ten minutes.