---
title: "UI and end-to-end testing with OpenAI Codex"
description: "What Codex writes well for UI and end-to-end tests, where it stalls, and how AGENTS.md and an MCP test runner turn its output into a passing PR check."
canonical: "https://momentic.ai/blog/codex-ui-testing-guide"
last-updated: "2026-10-31T09:50:51Z"
---

# UI and end-to-end testing with OpenAI Codex

URL: https://momentic.ai/blog/codex-ui-testing-guide

[blog](/blog) [/ resources](/blog/category/resources)  / codex-ui-testing-guide

Resources

How far Codex gets on UI and end-to-end tests, and how AGENTS.md instructions plus a test runner over MCP turn its output into checks you can trust.

Wei-Wei Wu

CEO, Momentic

## TL;DR

- Codex can turn a user story into Playwright test scaffolding with selectors and assertions.
- Codex can run tests when its environment permits, but a passing test it wrote does not independently verify the UI behavior you intended.
- Put test conventions in `AGENTS.md`, then connect Codex to a runner over MCP. [Momentic’s setup guide](/blog/how-to-set-up-momentic-mcp-server) covers its MCP server and skills.
- Commit the resulting test and run it in CI. The PR check then reports whether that same test passes against the proposed change.

## What Codex writes well for UI and end-to-end tests

Codex can draft useful Playwright tests when you give it the user story and the relevant code. In the CLI, you can point it at a component and its existing tests. In Codex cloud, you can give it the same task against a repository. Codex can use those files and the PR description to choose a route, locate controls, and write assertions for the behavior you described.

Suppose the story says that submitting an invalid email on `/signup` must show “Enter a valid email address.” With the app running and Playwright’s `baseURL` configured, Codex could draft a test like this.

`import { test, expect } from '@playwright/test';  test('signup shows an error for an invalid email', async ({ page }) => {  await page.goto('/signup');  await page.getByLabel('Email').fill('not-an-email');  await page.getByRole('button', { name: 'Sign up' }).click();   await expect(  page.getByText('Enter a valid email address')  ).toBeVisible(); });`

The test expresses the story in browser actions and an observable assertion. Codex still needs to check the rendered form before trusting the label, button name, or error text. Writing a plausible test file does not establish that the test runs or catches a regression.

## Where Codex stalls on UI testing

A coding agent can write a Playwright test that passes while missing the behavior the user requested. Suppose a user story says a saved address must survive a page reload. The agent implements the form, then writes a test that clicks Save and checks for a success message. Both the code and the test pass, but neither checks whether the address persists.

The problem is what [closes the verification loop](/blog/verification-belongs-inside-the-agent-loop). When an agent chooses the assertions for its own change, a green result confirms those assertions, not the full user story. An independent check of the saved address after reload would catch the omission.

Execution creates a separate failure mode. Codex can run Playwright through a shell, but a test file alone shows no evidence of a browser run. A selector that matches the wrong Save button may also pass locally and fail against another page state. If the test never enters the repository’s CI suite, later PRs cannot use it to catch the same regression. For a useful PR check, the test must exercise the intended behavior in a real browser and run again on subsequent changes.

## Giving Codex test-writing instructions with AGENTS.md

Put test conventions in a repository-root `AGENTS.md` before asking Codex to implement a UI change. Codex uses the file as project instructions, so you can specify where tests belong and what a test must assert.

`# UI test instructions  - Write end-to-end UI tests with @playwright/test in tests/e2e/. - Name test files <feature>.spec.ts. - Name each test for the user behavior it verifies. - Prefer getByRole and getByLabel over CSS selectors. - Assert a visible outcome after each important user action. - Do not use waitForTimeout to make a test pass. - Run the new test with npx playwright test tests/e2e/<feature>.spec.ts. - If the app or test runner is unavailable, report the test as unverified.`

For a password-reset story, you can ask Codex to implement the form and add `tests/e2e/password-reset.spec.ts`. The test should submit the form and check for the confirmation message the user actually sees. Codex can use the component and its accessible labels to draft that test, then run the specified command if the app and Playwright are available.

`AGENTS.md` controls the instructions Codex receives, not the truth of its assertions. Codex could still test a confirmation message while missing a broken reset email. Review the behavior the test covers, and keep its result in a runner that CI can execute again on the PR.

## Connecting Codex to a test runner over MCP

Codex can write a Playwright test without opening a browser. To check the UI while it works, give Codex a browser connection through MCP. Add this server to `~/.codex/config.toml`.

`[mcp_servers.playwright] command = "npx" args = ["-y", "@playwright/mcp@latest"]`

Restart Codex, then check that it sees the server.

`codex mcp list`

With the server available, Codex can open the running app, inspect the rendered page, and try the interaction described in a user story. For example, if a story says a customer can remove an item from the cart, Codex can check the cart in the browser before writing assertions against it. The browser result gives Codex evidence that source code alone cannot provide.

**Playwright MCP provides browser access, not a persistent PR check.** Ask Codex to save the assertion in a Playwright test and run that file with the project’s test command. If the file lives at `tests/e2e/cart.spec.ts`, a direct run looks like this.

`npx playwright test tests/e2e/cart.spec.ts`

The browser session helps Codex investigate a failure. The saved test and its CI run determine whether the same behavior passes on the next commit. For a Momentic-backed check, use the separate [Momentic MCP setup guide](/blog/how-to-set-up-momentic-mcp-server) rather than treating a browser connection as the quality system.

## Setting up the Momentic MCP server and skills for Codex

Codex needs access to a test runner before it can check the UI change it wrote. Start by setting up Momentic in the repository.

`npx @momentic/wizard@latest`

Next, follow the [Momentic MCP setup guide](/blog/how-to-set-up-momentic-mcp-server) to install the server and skills. For Codex CLI, put the MCP server configuration in `~/.codex/config.toml`. The server lets Codex write, run, and fix Momentic tests. The skills provide instructions for using those tools. Keep the tests in the repository so a PR can review the same test files that CI runs.

Install the Momentic skills and register the MCP server with Codex:

`npx momentic skills -y --target codex npx momentic install-mcp --client codex --scope user --config ./momentic.config.yaml`

`install-mcp` writes the server entry into `~/.codex/config.toml`, so you do not edit TOML by hand. Run `codex mcp list` to confirm that `momentic` is listed.

## Example workflow: from a user story to a passing PR check

A passing PR check needs to run the browser test, not merely accept Codex’s account of what it tested. Consider a task-list feature where selecting the Draft filter updates the URL and the filter survives a refresh.

Ask Codex to implement the feature and write down the behavior to verify. Keep the test intent separate from the implementation so a browser run can check what a user sees.

`Implement a Draft filter on /tasks. Selecting Draft should show only draft tasks, add status=draft to the URL, and preserve the filter after a refresh. Draft a test intent for those behaviors. Do not report the feature as verified until a browser test passes.`

Once Codex has changed the app, ask it to use the configured [Momentic MCP server and skill](/blog/how-to-set-up-momentic-mcp-server) to turn that intent into an end-to-end test and run it. The browser check should select Draft, inspect the visible task rows and URL, refresh, and inspect the rows again. Review the test file and run result before committing them with the feature. Codex writes the change. Momentic verifies its behavior in a browser.

The PR check must run that saved test again. The example below assumes the app uses `npm run dev`, serves port 5173, and has Momentic authentication configured in CI. Add `MOMENTIC_API_KEY` as a repository secret. The job runs the saved tests with `npx momentic run --upload-results`.

`name: E2E on: pull_request  jobs:  momentic:  runs-on: ubuntu-latest  steps:  - uses: actions/checkout@v4  - uses: actions/setup-node@v4  with:  node-version: 22  - run: npm ci  - name: Start app  run: npm run dev -- --host 127.0.0.1 > /tmp/app.log 2>&1 &  - name: Wait for app  run: |  for attempt in $(seq 1 30); do  curl -fsS http://127.0.0.1:5173/tasks >/dev/null && exit 0  sleep 1  done  cat /tmp/app.log  exit 1  - name: Run saved Momentic test  env:  MOMENTIC_API_KEY: ${{ secrets.MOMENTIC_API_KEY }}  run: |  npx momentic install-browsers  npx momentic run --upload-results`

Set the `momentic` job as a required PR check in your repository settings. If the saved test fails or the runner cannot start, the job fails and the PR remains blocked. After Codex fixes the behavior and the same test passes in CI, the PR check reports **Success**.

## Codex and Momentic are complementary, not redundant

Codex and Momentic handle different parts of a PR. Codex can implement a UI change and draft a Playwright test. Momentic provides a separate way to run and verify end-to-end behavior. The split matters because a coding agent can write an assertion that passes while missing the behavior the user story requires.

For example, Codex might add a checkout confirmation message and test that the message appears after a click. An end-to-end check should also establish that the order was placed. Running that check on the PR gives reviewers a result tied to the change, rather than Codex’s account of what it tested.

The same division applies when you use Cursor or another coding agent. The [Momentic MCP setup guide](/blog/how-to-set-up-momentic-mcp-server) covers the agent connection. A separate PR check keeps verification available after the agent’s session ends.

## Engineering takeaway

Codex saying a feature is done does not establish that the UI works. Give it a runner that executes a test against the application and returns results Codex can use to fix the change. Keep the test in the repository and run it again in CI, so reviewers can inspect the same check that gates the PR. Verification then rests on a repeatable result, not the agent’s account of its own work.

## Frequently asked questions

Can Codex run Playwright tests without MCP?    Yes. If Playwright and its browser dependencies are installed, Codex can run the test command in its execution environment. MCP is one way to connect Codex to additional tools, not a requirement for Playwright.    Does AGENTS.md guarantee a good test?    No. Instructions can tell Codex where tests belong and which conventions to follow. A browser run and review must still check whether the assertions cover the user story.    Does the setup work with Codex cloud as well as the CLI?    Both can write tests, but each environment needs access to the required runner and dependencies. A server configured in your local ~/.codex/config.toml does not automatically configure a cloud task.    How is this different from asking Codex to “test this”?    A specified runner produces an executable check and a result Codex can act on. Keeping that check in CI lets the PR run it again after Codex finishes. The [Momentic MCP setup guide](/blog/how-to-set-up-momentic-mcp-server) covers the agent connection.    Where do Momentic tests live, and who owns them?    Momentic tests live as YAML files in your repository, where you can review and version them with the code they check. See the [Momentic documentation](/docs) for the supported test and CI setup.

Still have additional questions?

## Keep reading.

[Resources   Testing iOS and Android apps with Claude Code     Four ways to test iOS and Android apps with Claude Code, from simctl and adb to XCUITest, Espresso, mobile MCP servers and Momentic mobile tests.     Wei-Wei Wu     12 min read](/blog/claude-code-mobile-app-testing)[Resources   Migrating from Appium to Maestro     How to move an Appium suite to Maestro flows, with a worked login example, a concept map and the cases where Appium is still the better fit.     Wei-Wei Wu     6 min read](/blog/migrate-appium-to-maestro)[Resources   Migrating from Cypress to Playwright: a step-by-step guide     A practical plan for moving a Cypress suite to Playwright: command mapping, the async model change, what breaks, and running both suites during the move.     Wei-Wei Wu     10 min read](/blog/migrate-cypress-playwright)

## Close the feedback loop.

Point Momentic at your app. Free to start, no credit card.

[Try for free](https://app.momentic.ai/signup) [Contact sales](/sales)