---
title: "QA Bench: Evaluating QA Agents on Real Web Applications"
description: "QA Bench measures how many of 1,396 labeled bugs Mo, Codex, Claude Code and Devin find in four open-source web apps in 30 and 60 minutes, and at what cost."
canonical: "https://momentic.ai/blog/qa-bench"
last-updated: "2026-11-02T16:51:50Z"
---

# QA Bench: Evaluating QA Agents on Real Web Applications

URL: https://momentic.ai/blog/qa-bench

[blog](/blog) [/ engineering](/blog/category/engineering)  / qa-bench

Engineering  Deep dive

QA Bench measures how many real bugs a QA agent finds in a running web application within a fixed amount of time, and how many of the bugs it reports are real.

Wei-Wei Wu

CEO, Momentic

- [QA Bench results](https://momentic.ai/qa-bench)
- [Dataset on Hugging Face](https://huggingface.co/datasets/momentic/qa-bench)
- [Harness on GitHub](https://github.com/momentic-ai/qa-bench)

## 1. Executive summary

QA Bench measures how many real bugs a QA agent finds in a running web application within a fixed amount of time, and how many of the bugs it reports are real. It compares [Momentic's Mo](https://momentic.ai/mo) with Codex (GPT-6.1 Sol and GPT-6.1 Terra), Claude Code (Opus 5.5 and Sonnet 5.5), and Devin across 20 product areas in four widely used open-source applications, Gitea, Metabase, Twenty, and Saleor, which together have over 190,000 GitHub stars. Codex and Claude Code each ran with browser automation tools, Playwright MCP and agent-browser, driving hosted Browserbase browsers rather than local ones. The reference set holds 1,396 labeled bugs.

Each run gives the agent one product area in a fresh sandbox and either 30 or 60 minutes. An LLM judge then matches the agent's bug reports against the approved reference bugs for that area and a set of claims reviewers have already rejected. We report recall, precision, a severity-weighted score, and cost per matched bug. Each count averages three runs per area (except Devin, which had only a single pass), so it reflects a typical pass through the suite rather than any single pass.

### What we found

1. **Mo matched the most reference bugs at both budgets, in every application.** In an average 60-minute pass through the suite, Mo matched 269.3 of the 1,396 labeled bugs. The next best configuration, Codex with GPT-6.1 Sol and Playwright MCP, matched 86.3, and the best Claude Code configuration, Opus 5.5 with Playwright MCP, matched 59.0, so Mo found 3.1x and 4.6x as many bugs. Against the other configurations, the ratio ranges from 3.3x to 10.0x, and is even higher against Sonnet 5.5. At 30 minutes, the averages were 176.7, 74.7, and 38.0.
2. **More than half of the lead over Codex is already there at 30 minutes.** Going from 30 to 60 minutes, Mo's matches rose 52%, and Claude Code with Opus 5.5 rose 55% to 71%. Codex with GPT-6.1 Sol rose 16% with Playwright MCP and fell 5% with agent-browser. Codex with GPT-6.1 Terra went down.
3. **Mo's extra coverage costs $2.67 per bug.** At 60 minutes, Mo finds a reference bug for $2.67, 35% less than Claude Code with Opus 5.5 and Playwright MCP at $4.09, while finding 4.6 times as many. Codex with GPT-6.1 Sol and Playwright MCP is the most precise configuration at 86.0%, against Mo's 84.2%, and finds a bug for $0.80. Every other configuration except Devin is less precise than Mo.
4. **The browser tool changes the result as much as the model.** Playwright MCP found 10% to 58% more reference bugs than agent-browser for Codex and for Claude Code with Opus 5.5. Which tool was more precise and cheaper per bug depended on the model. On Playwright MCP, Codex Sol matched 10% fewer bugs than Codex Terra at 30 minutes and 4% more at 60.

In an average 60-minute pass, Mo matches 19.3% of the labeled bugs, the highest recall in the benchmark. A single pass leaves most of the labeled bugs unfound.

_Mo is Momentic's testing agent. Momentic built, designed, ran, and graded the benchmark._

## 2. Why this benchmark exists

If you point a coding agent at a running web app and ask it to find bugs, it returns a list. The list doesn't tell you how much the agent missed or how many items are real. The only way to know either is to have a labeled set of bugs for that app and grade the list against it.

That matters more than it used to. Static analysis and unit tests now catch a large share of defects before code ships, so the bugs that get through are increasingly the kind you only see by using the product, such as a button that does nothing, a filter that returns the wrong rows, or a settings page that shows a stack trace. End-to-end testing is where those bugs are, and it's the part of QA that agents are now being asked to do.

Momentic's experience is that agents miss many of these. We wanted a way to measure whether Mo was getting better at this and compare it with the agents people already have. No benchmark existed for the task, so we built one.

The benchmark asks how much of a labeled bug set each agent can find in a fixed time, how many of its reports are real, and what it costs. An agent that reports everything it sees scores well on the first and badly on the second, and an agent that reports almost nothing does the opposite.

## 3. Roles and fairness

QA Bench is a first-party benchmark, and we run every agent under the same rules.

| Rule | Detail |
| --- | --- |
| Same environment | Each area runs in its own E2B sandbox with the same seeded data and accounts, so every agent starts every run from the same state. |
| Same brief | Every agent gets the same area scope, entry points, and reporting obligations. |
| No source or internet access | During a run, agents can't read the application source, the ground-truth labels, or earlier bug reports, and have no internet or search access beyond the application under test. The published runs had two exceptions (see Web access in the GitHub README): Codex web search was on, including in Playwright MCP runs, and in the Codex and Claude Code agent-browser runs the agent’s shell had open web access. |
| Mo has more to do | Mo has to produce a reproduction video and a written report for each bug it finds. The other agents only have to identify the bug. |
| Out-of-area findings | Agents are told to stay inside the area. They keep bugs they find outside for labeling, but they don't count toward the area's score. |

The rules fix the time an agent gets, not how it spends it, so cost appears next to every coverage number.

## 4. The corpus

The benchmark uses four widely used open-source web applications, each pinned to an upstream commit and split into product areas. An area is a part of the product with its own scope and entry points, such as Gitea's issues, Metabase's dashboards, Twenty's workflows, or Saleor's vouchers.

| Application | What it is | GitHub stars | Areas | Labeled bugs |
| --- | --- | --- | --- | --- |
| Gitea | Self-hosted Git service, like GitHub. Repositories, issues, pull requests, wikis, releases. | 58.8k | 11 | 476 |
| Metabase | Business intelligence. Queries, dashboards, data modeling, admin. | 50.1k | 4 | 464 |
| Twenty | CRM, a Y Combinator company (S23). Records, custom views, data model, workflows. | 60.2k | 4 | 374 |
| Saleor | Headless e-commerce platform with an admin dashboard. | 23.5k | 1 | 82 |
| Total |  | 192.6k | 20 | 1,396 |

Upstream repositories: [Gitea](https://github.com/go-gitea/gitea), [Metabase](https://github.com/metabase/metabase), [Twenty](https://github.com/twentyhq/twenty) and [Saleor](https://github.com/saleor/saleor).

The areas differ in size. Gitea's eleven areas average 43 labeled bugs each, Metabase's four average 116, and Twenty's four average 94.

## 5. Systems and task conditions

Ten agent configurations ran the benchmark.

Mo ran with no custom prompt and no setup beyond the effort preset. This tested Mo as a new user would: signing up and telling it to test something. Its medium and high presets run for about 30 and 60 minutes, serving as its two budget conditions.

Codex and Claude Code each ran two models and two browser integrations, so we can compare results across models and browser tools. Both integrations drove a hosted Browserbase browser, not a browser on the agent's machine.

Devin ran in its own native browser at default settings, not on Browserbase.

| System | Model | Reasoning effort | Browser interface | Areas at 30 min | Areas at 60 min |
| --- | --- | --- | --- | --- | --- |
| Mo | Mo | Medium at 30 minutes, high at 60 | qa CLI | 20 | 20 |
| Codex | GPT-6.1 Sol | Medium | Playwright MCP on Browserbase | 20 | 20 |
| Codex | GPT-6.1 Sol | Medium | agent-browser on Browserbase | 20 | 20 |
| Codex | GPT-6.1 Terra | Medium | Playwright MCP on Browserbase | 20 | 20 |
| Codex | GPT-6.1 Terra | Medium | agent-browser on Browserbase | 20 | 20 |
| Claude Code | Opus 5.5 | Medium | Playwright MCP on Browserbase | 20 | 20 |
| Claude Code | Opus 5.5 | Medium | agent-browser on Browserbase | 20 | 20 |
| Claude Code | Sonnet 5.5 | Medium | Playwright MCP on Browserbase | 20 | 20 |
| Claude Code | Sonnet 5.5 | Medium | agent-browser on Browserbase | 20 | 20 |
| Devin | Devin | Default | Devin's native browser | 20 | 20 |

The agent gets the area's scope and entry points, then explores the area and reports the bugs it finds. It can't read the application source, the labels, or earlier reports. Each configuration ran each area three times at each budget, for a total of 60 runs per configuration per budget, with one exception:

- Devin ran each area once per budget

Where a configuration is missing an area, its totals cover the areas it ran, and any comparison with Mo uses the same areas for both.

## 6. How ground truth was built

The reference set is the list of bugs that each area is known to have. Candidate bugs come from exploratory bug bashes by Mo, Claude Code, and Codex, and each candidate goes through the same four steps before it becomes a label:

1. Collect the candidate reports from the agent bug bashes.
2. Reproduce each one with Mo.
3. Have Claude Code and Codex check it independently, in the browser and in the application source.
4. Human review to approve or reject it.

Approved bugs go into the area's ground-truth.json. Rejected claims go into known-invalid.json, which the grader also uses, so an agent that reports something a reviewer has already rejected gets marked down for it. The reference set lists confirmed bugs in these application versions, not every bug they contain.

Each approved bug has a stored severity weight based on the kind of failure. Scoring uses the stored weight, not the judge's estimate.

| Weight | Category | What it covers |
| --- | --- | --- |
| 1 | Cosmetic | Incorrect presentation or navigation, such as a typo, a layout problem, a stale count, or the wrong tab opening, with the operation itself working |
| 3 | Functional | Broken functionality or an unintended change, such as a button that does nothing, a wrong result, or a failed save that still persists |
| 5 | Broken | A page that is broken or unusable, or a raw error dump shown to the user |

The weights grade what the user runs into, not security impact, so a page crash the user can recover from by reloading is still a 5. Section 9 has examples at each weight.

## 7. How we score

An LLM judge, GPT-6.1 Sol at high reasoning effort, matches each submitted finding to the area's labels. It first groups duplicate reports of the same bug. Each finding is then a match (a bug in ground-truth.json) or an invalid match (a claim in known-invalid.json). Findings that match neither go to a review queue, where a reviewer labels each one as one or the other, so the final results classify every finding as one of the two.

Duplicates are counted separately, so a bug reported twice in one run is one match and one duplicate.

### Precision

Precision is the share of findings that turned out to be reference bugs:

Precision=matchesmatches+invalid matches\displaystyle \text{Precision} = \frac{\text{matches}}{\text{matches} + \text{invalid matches}}

Duplicates don’t count against precision. They cost one point each in the score.

### Recall

Recall is the share of an area's labeled bugs that a run matched:

Recall=distinct reference bugs matchedlabeled bugs in the area\displaystyle \text{Recall} = \frac{\text{distinct reference bugs matched}}{\text{labeled bugs in the area}}

Reporting the same bug twice still counts as one bug found. Suite-level recall is pooled: the sum of per-area mean distinct matches divided by the sum of labels in the areas run (1,396), so each labeled bug carries the same weight regardless of area.

### Severity-weighted score

Score=∑matchesseverity weight−invalid matches−duplicates\displaystyle \text{Score} = \sum_{\text{matches}} \text{severity weight} - \text{invalid matches} - \text{duplicates}

The score rewards an agent for finding bugs that matter and penalizes it by 1 point for each known-invalid claim and each duplicate report.

Every number in the results is the sum of per-area means over three runs, or one run for Devin. Mo's 269 matches at 60 minutes means that, on average, in a pass through all 20 areas, Mo matches about 269 of the 1,396 labeled bugs. It isn't a total across all 60 runs. Costs work the same way.

## 8. Results

### 8.1 Coverage

Figure 1. Matched reference bugs per pass through the suite, averaged over three runs per area (one for Devin).

| Configuration | Areas | Matched, 30 min | Matched, 60 min | Recall, 60 min | Score, 60 min |
| --- | --- | --- | --- | --- | --- |
| Mo | 20 | 176.7 | 269.3 | 19.3% | 588.7 |
| Codex, GPT-6.1 Sol, Playwright MCP | 20 | 74.7 | 86.3 | 6.2% | 207.0 |
| Codex, GPT-6.1 Terra, Playwright MCP | 20 | 83.3 | 82.7 | 5.9% | 177.0 |
| Codex, GPT-6.1 Sol, agent-browser | 20 | 68.0 | 64.3 | 4.6% | 144.3 |
| Claude Code, Opus 5.5, Playwright MCP | 20 | 38.0 | 59.0 | 4.2% | 122.7 |
| Codex, GPT-6.1 Terra, agent-browser | 20 | 59.3 | 57.7 | 4.1% | 126.3 |
| Claude Code, Opus 5.5, agent-browser | 20 | 24.0 | 41.0 | 2.9% | 81.0 |
| Devin | 20 | 16.0 | 27.0 | 1.9% | 65.0 |
| Claude Code, Sonnet 5.5, Playwright MCP | 20 | 12.0 | 15.0 | 1.1% | 26.7 |
| Claude Code, Sonnet 5.5, agent-browser | 20 | 8.7 | 13.0 | 0.9% | 20.0 |

Matched and Score are per-pass averages over three runs per area, one for Devin.

At 30 minutes, Mo matched 2.37x as many reference bugs as Codex with GPT-6.1 Sol and Playwright MCP and 4.65x as many as Claude Code with Opus 5.5 and Playwright MCP. At 60 minutes, the ratios are 3.12x and 4.56x.

On shared areas, the 60-minute ratios against the other configurations run from 3.3x (Codex Terra with Playwright MCP) to 10.0x (Devin). Sonnet matched 8.7 to 15.0 bugs per pass from 13.0 to 20.7 findings, and Mo matched 14.7x to 20.7x as many. Anecdotal observation from the Sonnet runs is that the model stops exploring early.

Mo also has the highest score at both budgets: 2.18x Codex Sol with Playwright MCP at 30 minutes and 2.84x at 60. The score ratio is smaller than the match ratio because Mo pays about 66 penalty points per pass at 60 minutes against Codex's 14, and because Mo's matched bugs have a slightly lower average severity weight (2.4) than Codex's (2.6).

At 60 minutes, that average sits between 2.3 and 2.6 for every configuration except Sonnet, so no configuration is finding a markedly different mix of severities.

Figure 2. Recall by product area, all configurations, at 30 and 60 minutes. Each cell is the percent of the area's labeled bugs matched per run.

Mo's lead holds in all four applications at both budgets. In an average 60-minute pass, Mo matched 23% of Saleor's labeled bugs, 26% of Gitea's, 22% of Twenty's, and 9% of Metabase's. Metabase is the hardest application for all models. It holds a third of the corpus across four areas, with 95 to 131 bugs each, and Codex Sol with Playwright MCP matched 3% of it.

At the area level, Mo's three-run average beat every other configuration's in 19 of 20 areas at 60 minutes. The exception is Gitea notifications, the smallest area, with 16 labeled bugs, where Claude Code Opus with Playwright MCP averaged 3.0 matches and Mo 2.3.

Recall is low across the board. Mo's 19.3% at 60 minutes is the best number in the table, meaning even Mo misses about 81% of labeled bugs on average.

### 8.2 Report accuracy

An agent that reports more will also report more non-bugs.

Figure 3. Findings per pass at each budget, averaged over three runs per area, split by what the grader resolved them to.

| Configuration | Precision, 30 min | Precision, 60 min | Known-invalid, 60 min | Duplicates, 60 min |
| --- | --- | --- | --- | --- |
| Mo | 85.4% | 84.2% | 50.7 | 15.3 |
| Codex, GPT-6.1 Sol, Playwright MCP | 87.5% | 86.0% | 14.0 | 0.0 |
| Codex, GPT-6.1 Sol, agent-browser | 77.9% | 76.3% | 20.0 | 0.0 |
| Codex, GPT-6.1 Terra, Playwright MCP | 74.0% | 81.6% | 18.7 | 0.3 |
| Codex, GPT-6.1 Terra, agent-browser | 73.9% | 76.2% | 18.0 | 0.0 |
| Claude Code, Opus 5.5, Playwright MCP | 78.6% | 79.5% | 15.3 | 0.3 |
| Claude Code, Opus 5.5, agent-browser | 78.3% | 77.0% | 12.3 | 0.3 |
| Claude Code, Sonnet 5.5, Playwright MCP | 66.7% | 72.6% | 5.7 | 0.0 |
| Claude Code, Sonnet 5.5, agent-browser | 66.7% | 65.0% | 7.0 | 0.0 |
| Devin | 66.7% | 84.4% | 5.0 | 0.0 |

Known-invalid and duplicate counts are per-pass averages. Precision is pooled over every run of the configuration.

Setting aside Sonnet, whose figures rest on nine to 15 matches per pass, Codex with GPT-6.1 Sol and Playwright MCP is the most precise configuration at both budgets, at 87.5% and 86.0%. Mo is next at 30 minutes (85.4%) and third at 60 (84.2%, behind Devin's 84.4%), and Claude Code with Opus 5.5 is at 77% to 80%. The other three Codex configurations are lower, at 74% to 82%, because they report 1.3 to 2.8 times as many known-invalid claims while matching fewer reference bugs. Devin's 66.7% at 30 minutes is the lowest outside Sonnet; at 60 minutes it is 84.4%.

Mo's known-invalid matches are concentrated. At 60 minutes, about 59% come from five areas: three in Twenty, one in Metabase and one in Saleor, where Mo's precision ranges from 62% to 84%. Twenty workflows is also one of Codex Sol Playwright's three least precise areas, at 64%.

### 8.3 Cost

Mo finds a reference bug for $2.67 at 60 minutes and $2.54 at 30.

| Configuration | Cost per matched bug, 30 min (USD) | Cost per matched bug, 60 min (USD) | Cost per pass, 30 min (USD) | Cost per pass, 60 min (USD) |
| --- | --- | --- | --- | --- |
| Mo | 2.54 | 2.67 | 448 | 719 |
| Codex, GPT-6.1 Sol, Playwright MCP | 0.79 | 0.80 | 59 | 69 |
| Codex, GPT-6.1 Sol, agent-browser | 0.83 | 0.96 | 57 | 62 |
| Codex, GPT-6.1 Terra, Playwright MCP | 0.56 | 0.53 | 47 | 44 |
| Codex, GPT-6.1 Terra, agent-browser | 0.29 | 0.27 | 17 | 16 |
| Claude Code, Opus 5.5, Playwright MCP | 2.86 | 4.09 | 109 | 241 |
| Claude Code, Opus 5.5, agent-browser | 1.73 | 2.45 | 42 | 101 |
| Claude Code, Sonnet 5.5, Playwright MCP | 1.66 | 2.08 | 20 | 31 |
| Claude Code, Sonnet 5.5, agent-browser | 0.77 | 0.68 | 7 | 9 |
| Devin | 4.51 | 3.89 | 72 | 105 |

Cost per matched bug is total spend divided by total matches across all configuration runs. Devin's spend was calculated separately from the harness, as its billed on-demand USD per session reported by the Devin API (on_demand_usage), excluding fixture hosting and grading; its cost per matched bug uses that spend divided by its matched bugs. Cost per pass is the average over three runs per area, one for Devin.

Claude Code Opus with Playwright MCP is at $4.09 per bug at 60 minutes, so Mo costs 35% less per bug and finds 4.6 times as many. Codex Sol with Playwright MCP finds a bug for $0.80, and Mo costs 3.3x that per matched bug. The cheapest way to find a reference bug in this benchmark is Codex with GPT-6.1 Terra and agent-browser, at about $0.30. The most expensive is Devin at 30 minutes, at $4.51, and Claude Code Opus with Playwright MCP at 60, at $4.09.

Per pass, Mo is the most expensive configuration, at about $720 for a 60-minute pass, compared with $241 for Claude Code Opus with Playwright MCP and $69 for Codex Sol with Playwright MCP. Against Claude Code, Mo spends 3.0x as much for 4.6x the bugs. Against Codex, 3.1x of the 10.4x is bug count, and the remaining 3.3x is the higher cost per bug.

Figure 4. Recall against cost per matched bug. Each arrow runs from the 30-minute result to the 60-minute result for one configuration. The cost axis is logarithmic.

Cost per matched bug also varies by area. For Mo at 60 minutes, it runs from $1.94 in Saleor vouchers to $6.95 in Gitea notifications and $10.69 in Metabase querying, the two areas where Mo matched the fewest bugs. Codex Sol with Playwright MCP runs from $0.27 in Gitea wiki to $1.75 in Metabase querying.

### 8.4 Time budget

| Configuration | Matched bugs, 30 to 60 min | Cost per matched bug, 30 to 60 min | Areas with a higher 60-min average |
| --- | --- | --- | --- |
| Mo | +52% | +5% | 19 of 20 |
| Codex, GPT-6.1 Sol, Playwright MCP | +16% | +1% | 12 of 20 |
| Codex, GPT-6.1 Sol, agent-browser | -5% | +16% | 8 of 20 |
| Codex, GPT-6.1 Terra, Playwright MCP | -1% | -5% | 10 of 20 |
| Codex, GPT-6.1 Terra, agent-browser | -3% | -7% | 10 of 20 |
| Claude Code, Opus 5.5, Playwright MCP | +55% | +43% | 11 of 20 |
| Claude Code, Opus 5.5, agent-browser | +71% | +42% | 15 of 20 |
| Claude Code, Sonnet 5.5, Playwright MCP | +25% | +25% | 9 of 20 |
| Claude Code, Sonnet 5.5, agent-browser | +50% | -12% | 9 of 20 |
| Devin | +69% | -14% | 12 of 20 |

Changes are measured on the areas each configuration ran at both budgets.

Mo's matches rose 52% from 30 to 60 minutes, with a higher average in 19 of the 20 areas, and its cost per matched bug rose only 5% because spend rose at about the same rate. Claude Code with Opus 5.5 rose 55% to 71%, but spend more than doubled, so its cost per bug rose 42% to 43%. Devin rose 69% and got cheaper per bug. Codex with GPT-6.1 Sol rose 16% with Playwright MCP and fell 5% with agent-browser, and both GPT-6.1 Terra configurations found fewer bugs at 60 minutes than at 30.

In five of the 20 areas, Codex Sol with Playwright MCP's 60-minute average is lower than its 30-minute average, so an area-level difference of a bug or two is within what three runs can produce by chance.

### 8.5 Browser integration

Codex and Claude Code each ran with two browser integrations. Playwright MCP gives the agent a set of browser tools over the Model Context Protocol, and agent-browser is a command-line client the agent drives from its shell. Both connect to a hosted Browserbase browser. Each row below uses the areas both integrations ran for that agent and model.

| Agent and model | Budget | Shared areas | Playwright MCP matches | agent-browser matches | Playwright MCP precision | agent-browser precision | Playwright MCP cost per bug (USD) | agent-browser cost per bug (USD) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Codex, GPT-6.1 Sol | 30 min | 20 | 74.7 | 68.0 | 87.5% | 77.9% | 0.79 | 0.83 |
| Codex, GPT-6.1 Sol | 60 min | 20 | 86.3 | 64.3 | 86.0% | 76.3% | 0.80 | 0.96 |
| Codex, GPT-6.1 Terra | 30 min | 20 | 83.3 | 59.3 | 74.0% | 73.9% | 0.56 | 0.29 |
| Codex, GPT-6.1 Terra | 60 min | 20 | 82.7 | 57.7 | 81.6% | 76.2% | 0.53 | 0.27 |
| Claude Code, Opus 5.5 | 30 min | 20 | 38.0 | 24.0 | 78.6% | 78.3% | 2.86 | 1.73 |
| Claude Code, Opus 5.5 | 60 min | 20 | 59.0 | 41.0 | 79.5% | 77.0% | 4.09 | 2.45 |
| Claude Code, Sonnet 5.5 | 30 min | 20 | 12.0 | 8.7 | 66.7% | 66.7% | 1.66 | 0.77 |
| Claude Code, Sonnet 5.5 | 60 min | 20 | 15.0 | 13.0 | 72.6% | 65.0% | 2.08 | 0.68 |

Playwright MCP found 10% to 58% more bugs than agent-browser for Codex and Claude Code with Opus 5.5 at both budgets. Sonnet follows the same pattern: Playwright MCP found 38% more at 30 minutes and 15% more at 60. Precision is about the same for Terra on both at 30 minutes, and 5.4 points higher on Playwright MCP at 60; for Codex with Sol, agent-browser matched 1.4 to 1.8 times as many known-invalid claims and lost about 10 points, and Claude Code Opus was 0.3 to 2.5 points more precise on Playwright MCP. agent-browser was cheaper per matched bug for Terra, Sonnet and Claude Code Opus, and 5% to 20% more expensive for Sol.

## 9. Example bugs from the reference set

The nine bugs below are accepted ground-truth entries, chosen to show what each severity weight means in practice. All of them describe the pinned application versions used in the benchmark, not current upstream releases. The examples are drawn from the final reviewed label set (revision e77c72b).

| Application | Area | What goes wrong | Severity |
| --- | --- | --- | --- |
| Gitea | Pull requests | Deleting a comment leaves the conversation count unchanged until reload | 1 |
| Twenty | Settings and data model | A rejected role save leaves the Save button spinning | 1 |
| Twenty | Settings and data model | Reloading a role's Assignment tab opens Permissions instead | 1 |
| Gitea | Repository code | Uploading a file named after an existing directory replaces the directory | 3 |
| Saleor | Vouchers | A failed voucher save still persists the rename | 3 |
| Twenty | Workflows | A Delay step left at its default of zero seconds fails the workflow | 3 |
| Twenty | Settings and data model | Sorting the AI Skills list by Type crashes the settings page | 5 |
| Saleor | Vouchers | Assigning an eligible category or product crashes voucher creation | 5 |
| Metabase | Data modeling | Retiring a segment breaks its open reference page | 5 |

Each bug links to its entry in the label set at revision e77c72b:

- [Deleting a comment leaves the conversation count unchanged until reload](https://github.com/momentic-ai/qa-bench/blob/e77c72b7ad024366ba194f0c3cc36a93c6209349/fixtures/oss/gitea/e2e/areas/pull-requests/ground-truth.json#L1016) (Gitea, Pull requests, severity 1)
- [A rejected role save leaves the Save button spinning](https://github.com/momentic-ai/qa-bench/blob/e77c72b7ad024366ba194f0c3cc36a93c6209349/fixtures/oss/twenty/e2e/areas/settings-data-model/ground-truth.json#L884) (Twenty, Settings and data model, severity 1)
- [Reloading a role's Assignment tab opens Permissions instead](https://github.com/momentic-ai/qa-bench/blob/e77c72b7ad024366ba194f0c3cc36a93c6209349/fixtures/oss/twenty/e2e/areas/settings-data-model/ground-truth.json#L1006) (Twenty, Settings and data model, severity 1)
- [Uploading a file named after an existing directory replaces the directory](https://github.com/momentic-ai/qa-bench/blob/e77c72b7ad024366ba194f0c3cc36a93c6209349/fixtures/oss/gitea/e2e/areas/repository-code/ground-truth.json#L468) (Gitea, Repository code, severity 3)
- [A failed voucher save still persists the rename](https://github.com/momentic-ai/qa-bench/blob/e77c72b7ad024366ba194f0c3cc36a93c6209349/fixtures/oss/saleor/e2e/areas/vouchers/ground-truth.json#L312) (Saleor, Vouchers, severity 3)
- [A Delay step left at its default of zero seconds fails the workflow](https://github.com/momentic-ai/qa-bench/blob/e77c72b7ad024366ba194f0c3cc36a93c6209349/fixtures/oss/twenty/e2e/areas/workflows/ground-truth.json#L1197) (Twenty, Workflows, severity 3)
- [Sorting the AI Skills list by Type crashes the settings page](https://github.com/momentic-ai/qa-bench/blob/e77c72b7ad024366ba194f0c3cc36a93c6209349/fixtures/oss/twenty/e2e/areas/settings-data-model/ground-truth.json#L1086) (Twenty, Settings and data model, severity 5)
- [Assigning an eligible category or product crashes voucher creation](https://github.com/momentic-ai/qa-bench/blob/e77c72b7ad024366ba194f0c3cc36a93c6209349/fixtures/oss/saleor/e2e/areas/vouchers/ground-truth.json#L4) (Saleor, Vouchers, severity 5)
- [Retiring a segment breaks its open reference page](https://github.com/momentic-ai/qa-bench/blob/e77c72b7ad024366ba194f0c3cc36a93c6209349/fixtures/oss/metabase/e2e/areas/data-modeling/ground-truth.json#L248) (Metabase, Data modeling, severity 5)

The Twenty crash shows what a 5 looks like. In a workspace with more than one AI Skill, clicking the Name header sorts the Skills list normally. Clicking the Type header replaces the settings tabs and the list with "Sorry, something went wrong" and a Reload button. Reloading restores the list, and clicking Type crashes it again. No data is lost, but the settings content is unusable for anyone who clicks that header. The source review found a Boolean field declared as a string, which the sorter passes to localeCompare.

## 10. Discussion

Mo found more of the labeled bugs than any other configuration, in every application and at both budgets. The results point to a few reasons.

- **Mo kept finding bugs in the second half hour.** Its matches rose 52% between 30 and 60 minutes, with a higher average in 19 of the 20 areas. Its cost per matched bug barely moved, from $2.54 to $2.67. Codex with GPT-6.1 Sol rose 16% with Playwright MCP and fell 5% with agent-browser, and both GPT-6.1 Terra configurations fell. Claude Code rose 55% to 71%, from a much lower base, and its cost per bug rose 42% to 43%. Sonnet 5.5 seemed to give up early and stop looking for bugs, leading to bad recall.
- **Every agent did worst in the largest areas.** In an average 60-minute pass, Mo matched 23% of Saleor's 82 labeled bugs and 9% of Metabase's 464. The coding agents' recall in Metabase was 0.5% to 3.4%. Those areas hold most of the bugs, so they dominate the suite-level figures.
- **The extra coverage costs $2.67 per bug.** At 60 minutes, Mo's cost per matched bug was $2.67, 35% less than Claude Code with Opus 5.5 at $4.09 and 3.3 times Codex Sol with Playwright MCP at $0.80. Mo's precision of 84.2% is third, behind Codex Sol with Playwright MCP's 86.0% and Devin's 84.4%.
- **The browser integration mattered as much as the model.** Playwright MCP found 10% to 58% more bugs than agent-browser for Codex and for Claude Code with Opus 5.5, and which tool was more precise and cheaper per bug depended on the model. On Playwright MCP, Codex Sol matched 10% fewer bugs than Terra at 30 minutes and 4% more at 60.

No configuration found a different mix of severities. The average stored weight of a matched bug was between 2.3 and 2.6 for every configuration except Sonnet, so the gaps in the severity-weighted score come from how many bugs each agent found.

## 11. Conclusion

In an average 60-minute pass, Mo matched 19.3% of the 1,396 labeled bugs, compared with 6.2% for the best Codex configuration and 4.2% for the best Claude Code configuration, at $2.67 per matched bug. The lead held in all four applications, in 19 of 20 areas at 60 minutes, and at both budgets, and it grew with the longer budget with a 5% rise in cost per bug.

The benchmark's highest recall is about one bug in five. A single pass over a product area, in any tested configuration, misses most of its labeled bugs.

These figures hold for this corpus, these application versions, these agent configurations, and these two budgets. Another application, a newer model, a higher-effort setting, or a longer window could change any of them. The labels and scoring are pinned to a commit so that later runs can be compared against these.

## 12. Limitations

**First-party evaluation.** Momentic builds Mo and designed, ran, and graded the benchmark.

**Agent-drafted labels.** The candidate bugs came from bug bashes by Mo, Claude Code, and Codex, three of the agents being graded. If none of them can find a bug, it has no way into the reference set, so the labels may share those agents' blind spots. The reference set also includes bugs discovered by the scored runs and subsequently reviewed and approved, giving agents that report more candidates more opportunity to expand the set.

**An LLM judges the matches.** GPT-6.1 Sol decides whether a report describes a labeled bug and whether two reports describe the same bug. Judging errors in either direction feed straight into the scores.

**Uneven corpus.** Eleven of the 20 areas are from Gitea, and one is from Saleor. Metabase holds 33% of the labeled bugs in four areas. The suite-level numbers weight bugs equally, so Metabase's low recall pulls everyone's total down.

**Three runs per area, no dispersion reported.** The exports give per-area means over three runs and run counts, without the individual runs, so there are no error bars, and area-level differences of a bug or two should be read as noise. Devin has one run per area.

**Equal time isn't equal work.** The time budget doesn't cap spend or compute, and Mo spends far more per run than the baselines. Mo also has to record a reproduction video and write a report for each bug, which the baselines don't.

## 13. Reproducibility and artifacts

The reference labels are pinned to qa-bench commit e77c72b. Each area has its own folder under fixtures/oss/<application>/e2e/areas/<area>/, containing the area definition, its ground-truth.json, and its known-invalid.json; the repository also holds the Compose recipes and seed scripts that build each sandbox. All nine example bugs in section 9 appear in the same final reviewed label set, with the same severities and verification notes.

Every run's score file carries the hash of the judge configuration and the label files it was graded against. Anyone regrading against a later label set will get a different hash, and you shouldn't compare the two score sets directly. This report reflects exports taken on October 31, 2026, after every finding had been through human review. The dataset is on Hugging Face at [huggingface.co/datasets/momentic/qa-bench](https://huggingface.co/datasets/momentic/qa-bench), the harness, labels and per-area results are on GitHub at [github.com/momentic-ai/qa-bench](https://github.com/momentic-ai/qa-bench), and a summary of the results is at [momentic.ai/qa-bench](https://momentic.ai/qa-bench).

## Keep reading.

[Engineering   Axing our Custom Harness for AI SDK: Tool-based Agents for E2E Testing     From brittle browser scripts to self-healing AI agents: how we rebuilt AI Action to discover, replay, and repair end-to-end test flows without rewriting test code.     Zach Bellay     10 min read](/blog/tool-based-agents-for-e2e-testing)[Engineering   How We Ditched Postgres for ClickHouse to Process 12 Billion Caches Per Day     From Postgres pain to ClickHouse speed: how we re-architected caching to serve 2M+ cache queries and 20B entries per day, while maintaining ~250ms average resolution latency.     Henry Haefliger     6 min read](/blog/postgres-to-clickhouse-migration)[Engineering   Most Browser Agents "See" the DOM. Ours Had to Understand User Intent.     Browser agents look at the DOM. Momentic understands what the user actually meant. Here's how we scaled intent-based caching to maintain a 95%+ cache hit rate and be significantly more reliable.     Henry Haefliger     8 min read](/blog/teaching-browser-agents-user-intent)

## Close the feedback loop.

Point Momentic at your app. Free to start, no credit card.

[Try for free](https://app.momentic.ai/signup) [Contact sales](/sales)