---
title: "QA Bench"
description: "Mo, Codex, Claude Code and Devin explore 20 areas of four open-source apps. Scored against 1,396 human-labeled bugs."
canonical: "https://momentic.ai/qa-bench"
last-updated: null
---

# QA Bench

URL: https://momentic.ai/qa-bench

Agents are tasked with finding bugs in four popular open-source apps. We score their reports against 1,396 human-labeled bugs across 20 product areas, at 30 and 60 minutes.

[Read the blog](/blog/qa-bench) [Hugging Face](https://huggingface.co/datasets/momentic/qa-bench) [GitHub](https://github.com/momentic-ai/qa-bench)

## Recall against cost

Each arrow runs from a configuration’s 30-minute result to its 60-minute result. Up and to the left is better: higher recall for less spend per bug.

## Four open-source apps

Each area runs on a pinned upstream commit with seeded data.

- Gitea **476** labels in 11 areas Language Go Created 2016 Stars 58.8k Contributors 1,896 Commits 21,896 Code lines 644k [Labels (Gitea)](https://huggingface.co/datasets/momentic/qa-bench/tree/main/gitea/e2e/areas) [GitHub (Gitea)](https://github.com/go-gitea/gitea)
- Metabase **464** labels in 4 areas Language Clojure Created 2015 Stars 50.1k Contributors 527 Commits 46,057 Code lines 3.15m [Labels (Metabase)](https://huggingface.co/datasets/momentic/qa-bench/tree/main/metabase/e2e/areas) [GitHub (Metabase)](https://github.com/metabase/metabase)
- Twenty **374** labels in 4 areas Language TypeScript Created 2022 Stars 60.2k Contributors 789 Commits 17,666 Code lines 2.92m [Labels (Twenty)](https://huggingface.co/datasets/momentic/qa-bench/tree/main/twenty/e2e/areas) [GitHub (Twenty)](https://github.com/twentyhq/twenty)
- Saleor **82** labels in 1 area Language Python Created 2013 Stars 23.5k Contributors 331 Commits 22,706 Code lines 760k [Labels (Saleor)](https://huggingface.co/datasets/momentic/qa-bench/tree/main/saleor/e2e/areas) [GitHub (Saleor)](https://github.com/saleor/saleor)

### Sample human-labeled bugs

| App | Severity | Bug | Area |
| --- | --- | --- | --- |
| Gitea | Sev 5 | Invalid regular expression produces a 500 in repository code search | Repository code |
| Metabase | Sev 5 | Retiring a segment crashes its open data reference page | Data modeling |
| Twenty | Sev 3 | Restoring Notes or Tasks revives removed relations and can create duplicate links | Record pages |
| Saleor | Sev 5 | Assigning an eligible category, collection or product crashes the voucher editor | Vouchers |

Contributors, commits and code lines (scc) from the Oct 29 upstream snapshot; stars from Oct 31.

## Method

Every agent tests one product area of a pinned app at a time, in its own sandbox, with the same brief and time limit. Agents can’t read the app source, the labels or earlier reports. Codex and Claude Code drive hosted Browserbase browsers, not local ones. Devin uses its own browser, and Mo uses Momentic’s hosted browsers.

## Close the feedback loop.

Point Momentic at your app. Free to start, no credit card.

[Try for free](https://app.momentic.ai/signup) [Contact sales](/sales)