# Momentic: Full Content > AI end-to-end testing for web and mobile. Write tests in plain English; Momentic builds, runs, and maintains them as your product changes. This is the full-text companion to /llms.txt, following the llmstxt.org convention. It concatenates the readable content of Momentic's core marketing pages and blog posts as Markdown. ## When to use Momentic Use Momentic for: - End-to-end web, iOS, and Android regression and smoke suites in CI. - Authoring tests from a code diff with the explore agent. - Auto-healing flaky locators as the UI changes. - Triaging failing test runs. Agents call Momentic with: - `npx momentic init` to create a project. - `npx momentic run` to run tests. - `npx momentic mcp --config /absolute/path/to/momentic.config.yaml` for editor agents. - `npx momentic-mobile mcp` for mobile editor agents. Do not use Momentic for: - Unit tests. - Load testing. - Backend-only assertions that do not exercise a UI. ## Developer resources - [Developer resources](https://momentic.ai/developers): Momentic documentation, CLIs, MCP, CI, and machine-readable files. - [Documentation](https://momentic.ai/docs): Product documentation. - [Docs index for agents](https://momentic.ai/docs/llms.txt): Machine-readable documentation index. - [Blog index for agents](https://momentic.ai/blog/llms.txt): Dated machine-readable index of blog posts. - [CLI reference](https://momentic.ai/docs/cli-reference): CLI commands and options. - [momentic on npm](https://www.npmjs.com/package/momentic): Web CLI package. - [momentic-mobile on npm](https://www.npmjs.com/package/momentic-mobile): Mobile CLI package. - [MCP server for coding agents](https://momentic.ai/docs/coding-agents/mcp-server): Run the local stdio MCP server. - [API keys and team settings](https://momentic.ai/docs/account/api-keys-and-team): Manage API keys and team settings. - [Machine-readable files](https://momentic.ai/developers): Links to the site machine-readable files. - [Agent Skills index](https://momentic.ai/.well-known/agent-skills/index.json): Discovery index for Momentic Agent Skills. - [Agent instructions](https://momentic.ai/agents.md): When to use Momentic, plus every machine-readable index. --- # Platform Momentic learns your product, authors and runs tests across web, iOS, and Android, and keeps them green as your app changes. ## Learns your product Momentic learns your product from your docs, guides, codebase, and tools like Jira, Linear, and Figma. Stat: 77,813 TESTS CREATED ## Self-healing specs Tests adapt as your product changes, eliminating flaky tests and manual maintenance. Stat: 8,932,104 AUTO-HEALS ## Tests that keep up with your code Watches every pull request and commit to close coverage gaps automatically. Stat: 70,607,819 TEST RUNS EXECUTED ## Failures you can act on Triage regressions with reproducible steps, session replays, and root cause analysis. Stat: 96% SIGNAL-TO-NOISE RATIO ## Continuous integration Run reliably across every commit, deploy, and environment automatically. Stat: 80,290 PRS VERIFIED --- # How Momentic works ## 01. Write tests in plain English Momentic tests are human-readable YAML files in your codebase. No XPath or CSS selectors to maintain. Learn more: https://momentic.ai/docs/core-concepts/test-format ## 02. Run anywhere with a single CLI command Integrate with any pipeline that can run Node.js. Run on hosted browsers, Android emulators, and iOS simulators, all in parallel. Learn more: https://momentic.ai/docs/cli-reference/momentic/commands/run ## 03. Find the root cause When a test fails, Momentic pinpoints what broke and why using your codebase context, then shows you exactly how to fix it. Learn more: https://momentic.ai/docs/reliability/failure-analysis ## 04. Auto-heal as your product evolves Momentic auto-heals intended UI changes and flags real regressions, so you only review genuine bugs. Learn more: https://momentic.ai/docs/reliability/auto-heal ## 05. Generate tests from your code changes Momentic turns your PRs and diffs into new tests, and explores your app to close coverage gaps. Learn more: https://momentic.ai/docs/ai/explore --- # Enterprise Security, compliance, and support for organizations. ## SOC 2 Type 2 Independently audited. Compliance reports in the Trust Center. ## 99.99% uptime SLA Tests run when you need them. Backed contractually. ## SSO & RBAC SAML SSO, fine-grained role permissions, and audit logs. ## Dedicated CSM 24/7 support and a dedicated success manager for enterprise accounts. --- # Pricing Usage-based pricing billed in credits, with no per-user seats. One credit covers one test step. ## Free: $0 / forever For individuals and small projects. Includes: - 2,000 credits/month (~200 test runs) - All core features and AI test authoring - 30-day results retention - No credit card required ## Pay-as-you-go: $125 / month + additional usage For teams running tests regularly. Everything in Free, plus: - 10,000 credits/month (~1,000 test runs) - Buy more credits anytime; no hard cap - 5 Android + 5 iOS mobile devices - 5 phone numbers for SMS/OTP testing - Shared Slack channel ## Enterprise: Custom For organizations with security and compliance needs. Everything in Pay-as-you-go, plus: - Test-based custom pricing - SAML SSO, SCIM provisioning, audit log - Unlimited mobile session length - Priority support, SLA, and onboarding - Dedicated account manager ## Pricing FAQ ### How does Momentic's pricing work? Pricing is usage-based, billed in credits, with no per-user seats. Each plan includes a monthly credit allotment: Free includes 2,000 credits (about 200 test runs), Pay-as-you-go includes 10,000 (about 1,000 runs), and Enterprise uses test-based pricing sized to your suite. ### What's a credit? One credit covers one test step, including steps that AI features generate and run, like AI actions, failure recovery, and auto-heal. A typical run is about 10 steps, so 10 credits. Iterating on tests in the editor is always free. ### Does Momentic charge per user or per seat? No. Pricing is based on usage, not headcount, so you can invite your whole team (engineers, reviewers, and stakeholders) without per-seat charges. You only pay for the credits your tests use. ### What happens when I run out of credits? On Free, runs pause once you reach the monthly limit, a hard stop with no surprise charges. On Pay-as-you-go, usage past your included credits bills at $0.01875 per credit. Credits reset at the start of each monthly billing cycle and don't roll over. ### Can I buy more credits without upgrading? Yes. On Pay-as-you-go you can top up anytime in 10,000-credit blocks for $125 ($0.0125 per credit), cheaper than the overage rate. ### How is mobile emulator usage billed? Free includes 30 minutes of mobile emulator time per month with a hard stop. On Pay-as-you-go, emulator time is metered in credits per minute, 8 credits/min for Android and 15 credits/min for iOS, alongside up to 5 Android and 5 iOS emulators running at once. Enterprise is custom. ### Do I need a credit card to start? No. The Free plan is free forever and needs no card. You only add billing details if you move to Pay-as-you-go or buy more credits. ### Does Momentic support SSO and user provisioning? Google SSO is available on every plan. Enterprise adds SAML single sign-on, SCIM provisioning, audit logs, and domain claiming. ### How is Enterprise priced? Enterprise uses test-based pricing tailored to your suite size and requirements, and adds priority support, an uptime SLA, onboarding, and a dedicated account manager. Talk to sales for a custom quote. --- # Customers ## See how Quora cut QA time from 7 hours to 30 minutes with Momentic. _Quora · Consumer_ URL: https://momentic.ai/customers/quora > Momentic gave us a fast and reliable way to validate Poe.com's AI responses, even when they weren't deterministic. When Quora launched , the goal was simple: to bring together the best AI models from leading AI companies in a single interface across all devices. Today, Poe has grown into a unified hub where anyone can access more than 100 of the best text, image, video, and audio models, along with millions of bots built on top of them, all in one place. Achieving this hinges on three essentials: reliability, responsiveness, and seamless interactions. With multiple production deployments a day, the team needed to continuously validate performance, stability, and user experience across their platform. With much of the interactions happening across an extremely dynamic chat interface, they needed a quality solution that can adapt intelligently. ### Challenge ## Manual testing couldn’t keep up Quora’s Product Operations team is responsible for a vast variety of things, including but not limited to product support, product operations, internationalization, and QA. QA was mostly manual, with almost 500 test cases documented in a Google Sheets spreadsheet. These were key to ensure releases from the engineering team had no regressions. Executing critical test cases daily consumed up to 7 hours. With these tests running every weekday, the time investment became substantial. It was time for a change. ## The search for the right AI QA solution What followed was a thorough RFP process, where the Product Operations team evaluated a range of agentic QA solutions based on the following key criteria: A truly native generative AI solution - Quora needed a modern solution that could simplify testing while handling the dynamic nature of’s AI. Quality of the tool & UI - The AI agent needed to feel intuitive for Quora’s internal teams. Depth of integration - The tool had to integrate seamlessly with Quora’s existing test case management system to ensure context about bugs never gets lost. After evaluation, Momentic was selected for the project. ### Solution ## Expanding coverage with minimum engineering lift Quora had a clear goal: give the Product Operations team the ability to build and automate end-to-end tests of critical user flows in. Momentic’s natural language test editor and multi-modal assertions meant the team could describe expected behavior in plain English, critical for a team that was deeply embedded in workflows but never trained in Playwright or Selenium code. By the end of the month, the team had automated all critical tests in Momentic with minimal support from the engineering team. As Poe.com continues to grow and evolve, Momentic has played a helpful role in their process, contributing to smoother deployments. ### Results ## A 14x speed up Over the next three months, the Product Operations team automated hundreds of critical test cases. Today, the daily test executions take a mere 30 minutes instead of 7 hours. This 14x speedup not only saved time but also streamlined processes and made updates smoother, reducing dependence on manual steps. ## Shipping fast without breaking things: How Retool builds reliable products _Retool · Dev tools_ URL: https://momentic.ai/customers/retool > It's like giving someone your QA checklist and watching them execute it for you! Thousands of companies, including Amazon, Lyft, Doordash, and Brex use Retool to power mission-critical processes. Customers run billing, customer support, performance evaluations, and even inventory management all on Retool. For Retool, upholding reliability while maintaining a blazing fast developer velocity is a top priority. Engineers need to be able to ship major projects, such as the recent , quickly and confidently. ### Challenge ## Existing automation tools couldn’t keep up Many of Retool’s most important features rely on complex infrastructure or third-party dependencies. For example, testing requires creating and connecting a account. Similarly, to thoroughly validate Retool , a stateful deployment is needed to accept and run batch jobs. Third-party platforms are difficult to automate in Cypress, as their UIs constantly change and often have unpredictable loading states. In addition, the complexity of these Retool features meant that Cypress scripts would be lengthy and difficult to reason about. As the number of tests grew, they became cumbersome to author and maintain. The team simply couldn’t keep up. ## Everyone dreaded the checklist Retool’s product is constantly growing and evolving. The team looked to manual testing as a stopgap to verify correctness and maintain coverage. Hundreds of test cases were documented in a 15 page Google Doc. This checklist was executed manually by on-call engineers before every release. The checklist was a testament to Retool's focus on quality and reliability, but it impacted velocity. The team could only release every two weeks. ### Solution ## I didn’t think testing could be fun This is when the Source Control team decided to bring in Momentic. Within days the team had instrumented Retool’s most complex user journeys, no Cypress scripts or XPath required. Engineers were impressed by Momentic’s test authoring experience. Developers could verify correctness through Momentic’s multi-modal AI and simply describe user flows in plain English. They didn’t have to open the Chrome DevTools panel every time to find a . By the end of the month, the team had automated over half of the manual checklist. Soon after, the checklist was formally retired and the Source Control team reduced the release process to a single step: . ### Results ## Getting to daily deploys The Developer Experience (DevX) team drove adoption across the broader product organization. The DevX team authored, managed, and coordinated ownership of Momentic tests across several critical product areas including Permissions, Resources, and Workflows. Over the next three months, Retool was able to completely phase out manual testing. As a result, Retool 8x’ed their release cadence, saving over 40 engineering hours per month. Now Retool releases four times a week, an incredible feat for a platform used by over half of the Fortune 500. ## 30% fewer production incidents: How CoverGo improves reliability with Momentic _CoverGo · Insurtech_ URL: https://momentic.ai/customers/covergo > We've already seen a 30% decrease in production incidents thanks to Momentic's automated testing. is a leading insurance technology platform for health, life, and P&C, empowering insurance companies to transform digitally in the most flexible, scalable, and cost-effective way. Backed by leading U.S. insurtech and fintech VC funds, CoverGo also offers the most comprehensive set of insurance APIs on the market, enabling its users to build robust insurance ecosystems that transform daily operations. ### Challenge ## Scaling end-to-end coverage to match the speed of development CoverGo powers critical workflows for some of the world’s largest insurers, MGAs, and brokers, all operating in heavily regulated environments. Any disruptions across distribution, policy servicing, and claims flows can rapidly erode policyholder trust, trigger regulatory scrutiny, or lead to costly remediation. Naturally, testing plays a crucial role within the company’s software delivery process. However, as CoverGo continued to expand its platform’s functionality, ensuring quality and uptime became an increasingly heavy lift. Biweekly regression testing would consume several working days for all involved, draining bandwidth that could have been spent on higher-value initiatives. While CoverGo had already invested in Playwright, building and maintaining coverage for complex, interconnected flows often took several days per scenario, making it difficult for automation to keep pace with development. Head of QA, , quickly realized current QA workflows weren’t keeping pace with CoverGo’s development velocity. She needed a solution that would help her team scale end-to-end coverage without adding overhead or require deep programming expertise. After evaluating several solutions, she discovered and knew the AI-powered testing platform was up to the task. ### Solution ## Establishing an AI verification layer that scales alongside CoverGo’s platform With Momentic, CoverGo established a scalable verification layer that empowers its team to accelerate without sacrificing coverage or quality. The partnership started with a turnkey onboarding, where Hanna worked with the Momentic team to replace all existing automations with Momentic tests. “The team was there offering support every single day,” Hanna shares. “Whether it was replying to questions, offering advice, or breaking down new updates, they really helped us acclimate in a short period of time.” Instead of relying on traditional frameworks like Playwright, Hanna’s team now creates tests using plain English instructions that describe exactly what flows need to be validated. These steps, combined with Momentic’s agentic actions and assertions, help validate both UI behavior and functional outcomes, enabling the team to test complex end-to-end workflows. Tests are organized into suites and executed using Momentic’s built-in scheduler, eliminating the need to manage separate cron jobs. Some test suites run daily to validate build stability, while others run weekly to cover broader regression scenarios. This cadence enables Hanna’s team to expand coverage without overwhelming engineers or introducing unnecessary noise. To ensure results are immediately actionable, notifications are delivered directly to Slack, alerting the team as soon as a failure occurs so issues can be addressed quickly. For Hanna, Momentic’s ease of use was the most powerful unlock. Now, team members don’t need deep programming or framework expertise to create and maintain tests, which makes automation easier to scale and sustain as CoverGo’s product evolves. “Even our less tech-savvy folks would have a fully functioning test case up within an hour,” she shares. “It’s one of the most intuitive testing platforms we’ve ever used.” ### Results ## A 30% reduction in production incidents through automated, end-to-end testing Partnering with Momentic enabled CoverGo to transform testing from a bottleneck into a scalable engine for development velocity. As end-to-end coverage expanded across critical workflows, fewer defects reached UAT and production environments — improving release confidence as the platform continued to scale. Looking ahead, Hanna plans to extend this foundation across CoverGo’s entire product surface area. With Momentic, a single testing platform can support end-to-end coverage across both web and mobile applications, allowing the team to scale quality consistently as new products and experiences are introduced. ## 80% faster release cycles: How GPTZero deploys multiple times a day while maintaining quality _GPTZero · AI_ URL: https://momentic.ai/customers/gptzero > With Momentic, we've caught bugs that would have eluded even our most diligent internal tests. is the world’s leading AI-text identification platform — founded to verify writing authenticity and preserve what’s human. Ranked as the #1 most trusted and reliable AI tool in 2025 by G2, GPTZero currently serves over 10 million individual users, 3,500 educational institutions, and hundreds of leading organizations across hiring, publishing, and legal. ### Challenge ## Shipping fast without breaking core user workflows As a first mover in AI text identification, GPTZero saw rapid and lasting adoption, led primarily by universities seeking to preserve academic integrity. As its user base quickly expanded across new industries (including HR, publishing, and legal), the development team sought to enrich the platform with deeper functionality, including team orchestration, document management, and subscription-based workflows. However, as GPTZero’s product evolved, its existing Cypress suite couldn’t keep pace. With a team consisting of only seven engineers at the time, manually creating and maintaining new tests was a considerable resource drain that directly impacted development velocity. This challenge only compounded with every release cycle, as each additional feature introduced a host of deeply interconnected workflows that needed to be validated before reaching production. The combination of rapid product expansion, mounting testing overhead, and rising regression risk signaled that GPTZero needed a more scalable testing foundation. Rather than throwing headcount at the challenge, Co-founder and CTO sought an AI-powered solution that would automate end-to-end testing for each new feature shipped. Having previously consulted with ’s Co-founder, , Alex was confident that their AI-powered testing platform was up to the task. ### Solution ## An AI verification layer that evolves alongside GPTZero’s product With Momentic, GPTZero began automating end-to-end verification at a pace that matched its rapid product development, covering both new and existing workflows as the platform expanded. Because Momentic translates natural language descriptions into self-healing tests that adapt to UI changes, the engineering team ramped up quickly. Within the first week, engineers were already validating core workflows end to end, protecting user-facing functionality without slowing feature delivery. “Momentic’s test editor UI is incredibly intuitive,” Alex shares. “None of our engineers were confused about how to use it, and unstable tests are reviewed and quarantined to keep our CI workflow stable.” As GPTZero’s product surface area continued to grow, Momentic became the foundation for verifying every workflow users relied on. From uncovering frontend re-render loops to flag inconsistencies, database provider race conditions, and performance issues tied to third-party libraries, Momentic consistently surfaced problems that would have otherwise demanded significant manual investigation. The relationship quickly evolved into a true engineering partnership. As GPTZero scaled rapidly from a seven-person team to more than fifteen engineers, Momentic scaled alongside them — supporting higher test volume, broader coverage, and more complex workflows without adding friction. Momentic regularly joined live debugging sessions with GPTZero’s team, shipping fixes and improvements in real time and helping the company maintain speed and reliability as both teams grew together. ### Results ## An 89% reduction in defect escape rate while accelerating development velocity With Momentic, GPTZero transformed testing from a recurring operational drain into a leverage point for the engineering team. By significantly reducing the time spent validating releases, the team reclaimed hours that could be redirected toward higher-value work — from iterating on new features to engaging directly with customers and shaping the product roadmap based on real-world feedback. As a result, GPTZero now moves faster across its development workflow while shipping with greater confidence. Engineers spend less time tracking down regressions and more time building, learning from users, and expanding the platform’s capabilities. Looking ahead, GPTZero plans to continue scaling its product surface area while maintaining the reliability educators and organizations depend on. Momentic’s platform remains a core part of that evolution, supporting the team as workflows grow more complex and release velocity increases. ## 85% reduction in production incidents: How Mutiny reliably ships enterprise-level GTM flows with Momentic _Mutiny · Martech_ URL: https://momentic.ai/customers/mutiny > Momentic gives us reliable end-to-end coverage, so we can focus on features instead of maintaining tests. is an AI platform that empowers GTM teams to create customer-facing assets that reflect their brand. Trusted by industry leaders including Uber, Amplitude, and GitLab, Mutiny supports high-touch enterprise GTM programs where reliability and execution confidence directly impact revenue and customer trust. ### Challenge ## Maintaining development velocity without compromising on quality As one of the fastest-growing enterprise GTM platforms, Mutiny leans heavily on continuous innovation to support its expanding roster of B2B clients. Because each interaction, from researching accounts to generating assets, has a direct impact on revenue, balancing development speed with release quality is their most important problem to solve. However, as the team continued to ship dynamic, agent-based flows, their existing testing solutions could not keep up. At the time, Mutiny’s engineers relied on a mix of service-level unit tests, Datadog synthetics, and light Playwright automation, but none of it covered end-to-end user flows that spanned multiple services. Every time the team shipped a change, those flows had to be retested manually. “We could never get an automated testing solution to work because only a few of our engineers had the know-how to write and maintain tests in Playwright, and they would quickly get stale as the app evolved,” says , Staff AI Engineer at Mutiny. “Without visual inspectability, discovering what failed was another major hurdle.” Rather than allowing instability and regressions to erode customer trust, Mutiny sought an end-to-end solution to replace its existing testing stack. The team’s ideal solution would not only match their iteration speed and product complexity, but also empower all engineers to own testing without commanding internal bandwidth. That’s when Mutiny discovered . ### Solution ## Dependable, engineer-owned testing that drives continuous innovation With Momentic, Mutiny introduced a robust testing suite that empowers its team to ship fast with confidence, even as its platform continues to evolve. To ensure Mutiny’s onboarding was quick and painless, the Momentic team created a shared Slack channel, weekly sessions with engineers, and a direct line to the CEO. Within the first week, Mutiny had a complete suite running in production. “Even after our initial onboarding call, I knew we made the right choice,” shares Alec. “The platform was intuitive, easy to use, and backed by a team that was always willing to work around our needs.” Today, engineers actively use Momentic to define and update tests for the workflows they change most frequently. Instead of writing brittle, locator-heavy scripts, the team describes expected behavior at a high level and effortlessly adjusts tests as the UI and underlying services evolve. This made it practical to keep coverage up to date, even as priorities shifted and new features shipped quickly. Momentic also enables Mutiny to validate complete end-to-end behavior across multiple services, something that had been difficult to verify consistently in the past. Tests routinely catch regressions caused by seemingly small changes in individual services that would previously have gone unnoticed until a demo or customer workflow failed. With clear visibility into where flows break and the application's state at the time, engineers diagnose issues earlier and resolve them before releases reach production. As testing became easier to understand and maintain, ownership naturally expanded beyond a small group of engineers. More team members began contributing updates and fixes to tests, turning end-to-end validation into a shared responsibility rather than a specialized task. This reduced the burden on any single engineer and allowed the team to move faster without sacrificing quality. By validating critical workflows earlier and more reliably, engineers are able to dedicate more time focused on shipping impactful improvements. Now, Mutiny continues to scale development velocity while reducing the risk of regressions reaching customer-facing environments. ### Results ## 85% fewer production incidents through a scalable, AI-powered testing layer Partnering with Momentic enabled Mutiny to accelerate development velocity without the constant concern of regressions making their way into production, even as the team continued to iterate quickly across a complex, multi-service product. Moving forward, Mutiny is confident that Momentic will remain a core part of its development workflow, empowering the engineering team to expand coverage into new product areas and move quickly without sacrificing reliability. ## From unscalable frontend tests to robust E2E coverage: How Pocus validates every deployment with Momentic _Pocus · Revenue data_ URL: https://momentic.ai/customers/pocus > Momentic lets our engineers describe the outcomes they want to verify in human terms, not fragile machine logic. (acquired by ) helps modern GTM teams hit their revenue goals, create repeatability, and future-proof their motion with the help of data, AI, and automation. Established in 2021, Pocus is trusted by leading organizations like Monday.com, Canva, and LaunchDarkly to generate over 70% of their pipeline and save reps 10+ hours every week. ### Challenge ## Reliable validation became increasingly difficult as Pocus’ product evolved As Head of Engineering at Pocus, leads a lean but agile team responsible for maintaining and evolving the company’s AI sales intelligence platform. With revenue teams across dozens of industry-leading enterprises relying on the platform daily, even minor disruptions across core flows can directly impact deal prioritization, signal identification, and outbound pipeline. Naturally, Aniruddha leans on continuous testing to ensure these revenue-generating flows remain airtight during each release. The problem? While Aniruddha’s team had a robust backend testing foundation in place, they were executing frontend tests entirely through Playwright, which struggled to keep pace with the platform’s continuous UI changes. Routine frontend updates caused locators to become outdated, leading to test failures even when user flows remained intact. Engineers would then have to investigate whether a failure indicated an actual issue or simply an outdated script. Over time, this flakiness reduced confidence in the Playwright test suite. As failures became harder to interpret, maintenance began to feel more like overhead than protection. Aniruddha knew that without scalable and consistent validation across real user flows, regressions that should have been caught earlier would slowly begin to surface in production. Rather than trying to salvage Playwright, he began searching for a new testing platform that could restore engineering confidence without introducing the brittleness the team had experienced previously. Fortunately, he didn’t have to search for long. After discovering through trusted colleagues within his network, Aniruddha immediately knew it was time to make the shift. ### Solution ## Transforming testing from a bottleneck to a force multiplier with Momentic Getting started with Momentic was a swift and collaborative process. Aniruddha worked directly with Momentic to familiarize his team with the platform, experiment with CI integration, and field any questions before their official rollout. Within a month, the team had deployed end-to-end tests covering core user flows without increasing technical overhead. “Having direct access to the Momentic team has been very helpful,” Aniruddha shares. “They’re super supportive, quick to answer questions, and always open to feedback.” Rather than wiring automation to static DOM selectors as they did in Playwright, Aniruddha’s team describes the exact flow they want to verify in natural language, and Momentic executes the test based on that intent. “We can define very specific actions with Momentic, like opening your list of accounts or playbooks from our dashboard,” Aniruddha shares. Because Momentic dynamically evaluates the rendered page and identifies elements based on context, tests remain stable even when the frontend is refactored or components are updated during normal development. Now, Aniruddha’s team knows that any failed tests represent true regressions rather than noise. Within months, Momentic has become a core element of Pocus’ CI workflow, validating critical user flows before every production deployment. By embedding automated end-to-end validation directly into the release process, Aniruddha’s team can verify that key functionality continues to behave as expected with each new release. ### Results ## More reliable tests, broader coverage, and validation on every deployment Partnering with Momentic enabled Pocus to eliminate Playwright flakiness, accelerate end-to-end coverage, and leverage automated validation across every production deployment. Looking ahead, Aniruddha is excited to further explore Momentic's CI/CD integrations as Pocus scales. ## From flaky tests to reliable releases: How Roo reclaims engineering time with Momentic _Roo · Healthcare_ URL: https://momentic.ai/customers/roo > Momentic lets us move quickly, keep control, and eliminate brittleness, exactly what we need. is the industry-leading veterinary staffing platform connecting Veterinarians, Veterinary Technicians, and Assistants with animal hospitals for relief work and hiring opportunities. Roo empowers the largest network of veterinary professionals to earn more on their schedule while providing relief coverage to help hospitals care for pets while their teams take time off. Roo’s team also founded and spun off , a leading open-source AI code editor used by developers at companies including Netflix, Microsoft, and Amazon. ### Challenge ## Managing flaky tests threatened development velocity Roo plays a critical role in veterinary care, helping hospitals secure relief coverage quickly so they can remain staffed when their clients need it most. Any disruption across core flows, such as logging in or viewing available shifts, can leave hospitals without support and veterinary professionals without work. Because of this, the team places a strong emphasis on validating key workflows before pushing new code to production. As the platform expanded, however, managing the growing Cypress test suite became increasingly difficult. Investigating flaky tests was time-consuming and repeatedly pulled engineers off higher-value work. Even a single false positive could delay a release and require hours of manual investigation. To maintain development velocity while protecting uptime, Lead Software Engineer in Test began evaluating AI-powered testing solutions that could complement his existing coverage and reduce the burden of managing flaky tests. Many options fell short. Some focused only on front-end UI behavior, while others relied entirely on text-based prompts that produced tests he couldn’t reliably customize or repeat across Roo’s detailed setup and teardown requirements. Michael ultimately chose because it offered something the other tools didn’t: stable, repeatable end-to-end tests that behaved consistently across runs without demanding extensive maintenance. The ability to validate complex workflows with far fewer interruptions immediately aligned with what his lean team needed. ### Solution ## Accelerating development with dependable, AI-powered testing Partnering with Momentic enabled Roo to establish a reliable AI testing layer that supports faster, more confident development cycles. Instead of pausing development to troubleshoot flaky test failures, engineers can now validate core workflows with consistency across every release. With Momentic’s hands-on support, Michael installed the CLI, created his first tests, and integrated them into Roo’s development workflow within days. “I can’t speak highly enough about my interactions with the Momentic team,” he shares. “They made the onboarding process painless and were always eager to iterate based on our feedback.” Michael now executes Momentic tests through the CLI in GitHub Actions, mirroring the stages where his Cypress tests previously ran. This enables him to compare behavior across identical end-to-end flows without altering Roo’s existing development process. Because Momentic’s tests have proven significantly more stable, Michael is evaluating how and when to transition more of Roo’s gatekeeper flows from Cypress to Momentic to reduce the amount of time lost to flaky test investigations. Beyond immediate improvements, Michael sees Momentic as a critical part of where Roo’s engineering practice is headed. Roo Code already helps engineers build features faster using AI — and Momentic opens the door to closing that loop by enabling AI to test those features as well. With Momentic’s MCP server, Michael envisions a workflow where Roo Code can both generate code and automatically generate the corresponding end-to-end tests, reducing manual effort and accelerating iteration cycles. It represents an early glimpse into how AI will increasingly empower engineers to build and test software more efficiently, something Michael is eager to explore. “It’s an incredibly innovative workflow that we’re working toward, but we’re confident we can bring it to life using Momentic,” he says. ### Results ## Hours of engineering time reclaimed and a 50% increase in test reliability With Momentic, Roo eliminated flaky tests that previously stalled releases and consumed countless hours of engineering time per incident. Tests now run predictably across core user flows, giving the team greater confidence as they continue to ship updates and expand the platform. Roo also sees Momentic as the missing piece in closing the feedback loop between AI-generated code and AI-generated tests. While AI coding tools like Roo Code already accelerate feature development, teams typically still write or maintain tests manually, a gap that slows iteration. With Momentic, Roo can now envision a workflow where AI not only builds new features but also produces the corresponding end-to-end tests with minimal human effort. It reflects where the broader discipline of software engineering is heading: a future in which AI empowers engineers to build and test new functionality far more efficiently than before. ## 80% coverage in 3 days: How Nuvo prevents regressions across critical onboarding flows _Nuvo · Data infrastructure_ URL: https://momentic.ai/customers/nuvo > Momentic changed how we approach frontend testing. It's now a core part of how we ship changes every day. is building the first digital trade network for the physical economy by connecting the businesses that make, move, and supply the world. The platform helps businesses onboard customers and vendors, verify companies and principals, and manage risk at scale. Today, more than 120,000 businesses use Nuvo to streamline onboarding and make faster, more confident credit decisions across complex supply chains. Trusted by companies such as AutoZone, Coca-Cola, and Nestlé, Nuvo sits in the critical path for industries including food & beverage, chemicals, construction, and manufacturing. ### Challenge ## Flaky frontend tests put critical onboarding flows at risk At Nuvo, frontend functionality is directly tied to how quickly buyers and suppliers can onboard and begin doing business. The company’s web application powers customer onboarding and trade workflows where any interruption, delay, or regression can slow approvals, block transactions, and defer revenue. Unlike engineers at larger companies optimizing for UI polish or feature velocity alone, the Nuvo engineering team builds product flows end-to-end that are both highly interactive and risk-sensitive. As Nuvo’s product matured, those workflows became increasingly complex. New onboarding paths, validations, and customization increased the number of possible states users could move through. Each release introduced more edge cases that needed to be verified before reaching production. While backend and unit tests covered isolated logic, they couldn’t reliably catch regressions in end-to-end user journeys where sequencing, state transitions, and asynchronous behavior determined whether onboarding succeeded or stalled. To safeguard these flows, the team initially relied on Playwright-based browser automation to validate critical paths. In practice, this setup was difficult to maintain. Animations, asynchronous UI updates, and timing dependencies caused tests to fail unpredictably. Engineers were forced to add hard-coded waits, adjust selectors, and rerun jobs just to determine whether a failure was a real regression or just a flaky test. That uncertainty carried a real cost. Each failed run forced the team to pause feature work, inspect logs, and manually debug. Over time, trust in the test suite eroded. Expanding coverage only made the problem worse. To reach true 100% frontend test coverage with Playwright, Nuvo’s team estimated it would require dedicating roughly 20–25% of engineering time to writing and maintaining tests. It was at this point that they began looking for an alternative, one that could validate Nuvo’s most critical onboarding flows without introducing more maintenance overhead or slowing development velocity. That search ultimately led the team to . ### Solution ## Reliable and scalable frontend testing with Momentic With Momentic, Nuvo established a reliable and scalable testing layer for validating revenue-critical workflows. Instead of limiting coverage to avoid flakiness, Nuvo gained a testing foundation it could trust to surface real regressions early and keep onboarding flows stable through frequent releases. Onboarding was hands-on and collaborative from the start. The Momentic team worked closely with Nuvo to set up tests, align them with existing development workflows, and integrate them cleanly into CI. That early support made it possible to adopt Momentic quickly without introducing friction or changing how the team shipped code. Once live, Nuvo migrated its most critical Playwright tests to Momentic. Those tests ran successfully out of the box with minimal setup, immediately eliminating the timing-related flakes that had undermined trust in previous tests. Creating the same end-to-end tests took significantly less time, and the tests remained stable as the frontend continued to evolve. Today, the Nuvo engineering team adds Momentic tests alongside every major code change, as part of the core development process. Tests run automatically in CI to validate that new features behave correctly before reaching production, while a broader suite runs nightly to cover additional edge cases and scenarios. By handling test execution and infrastructure, Momentic removed another source of overhead. The team no longer manages browser environments or reruns flaky jobs to confirm failures. Instead, test results consistently reflect real regressions, enabling the team to react quickly and resolve issues before they impact customers. As reliability improved, trust in the test suite was restored, and end-to-end testing expanded beyond a small set of critical paths into a dependable, shared part of the team’s workflow. Operating with a lean engineering team of 12, Nuvo now ships to production multiple times per day without dedicating a disproportionate share of engineering time to maintaining test infrastructure. ### Results ## Expanded coverage, hours of engineering time saved, and a scalable testing foundation Partnering with Momentic gave Nuvo a testing foundation the team could trust as the platform continued to evolve. With stable end-to-end tests running on every pull request, engineers no longer had to second-guess whether failures reflected real regressions or flaky execution. That reliability removed friction from the development process and made it easier to ship frontend changes without introducing risk into core onboarding workflows. Looking ahead, the Nuvo team is aggressively increasing Momentic test coverage, allowing them to accelerate development velocity without regressions. ## 99% pass rates within weeks: How Kibu ships complex EHR flows with confidence _Kibu · Healthcare_ URL: https://momentic.ai/customers/kibu > Momentic empowers us to ship fast without breaking things. It's become a crucial part of our PR cycle. is a modern web app that empowers disability providers to elevate the quality of their care while streamlining documentation flows in tandem. Beginning as a virtual fitness platform for individuals with IDD, Kibu has evolved into an end-to-end education, skill development, and EHR management solution for disability providers across the globe. ### Challenge ## Evolving app complexity strained the limits of manual testing Kibu supports hundreds of 24/7 care providers, where even brief outages can disrupt essential services and directly impact the individuals receiving care. As their platform expanded beyond virtual fitness to education, skills training, and full EHR management, the number of interconnected workflows grew rapidly, and so did the need for consistent, reliable uptime. With every release, engineers spent hours manually testing core user journeys to confirm new updates wouldn’t break existing functionality. This process was slow, repetitive, and only covered a small fraction of the paths real providers follow throughout the platform. As the product expanded, it became clear that manual testing simply couldn’t keep pace. To maintain reliability without adding headcount, Head of Product and COO began searching for a solution that could autonomously validate multi-step, real-world user flows end-to-end. The goal was to preserve the team’s bandwidth while ensuring providers could depend on a stable platform every hour of the day. ### Solution ## Solidifying coverage without slowing development With , Kibu turned hours of repetitive manual testing into a scalable automated suite, giving the team a foundation that speeds up development instead of slowing it down. Thanks to hands-on support from the Momentic team, Kibu was live within a day and had its first automated test running immediately. “They’re an absolutely incredible team to work with,” Ben shares. “Anything we needed, product knowledge, feature requests, even testing best practices, they were there to assist.” Today, Momentic’s automated tests are embedded directly into Kibu’s development workflow. New features are validated before reaching production, and tests run daily in production environments to surface issues early and keep essential user flows functioning reliably around the clock. After each deployment, a full suite of automated checks confirms that core user journeys continue working as intended, eliminating hours of manual regression work. Momentic’s integrations reinforced this workflow further. Kibu now triggers automated tests on every code push, pull request, and scheduled workflow through GitHub Actions, while Momentic’s Slack integration delivers real-time results to the engineering team. In Ben’s words, “engineers picked up our new workflow so quickly that our time-to-value was days, not months.” ### Results ## $100K+ annual savings and a 99% pass rate across automated daily tests With Momentic, Kibu regained engineering time that was previously spent on repetitive manual testing. Automated checks now validate core user journeys across the app, reducing post-release surprises and giving the team far greater confidence as they ship new features. With new support for Android and iOS rolling out, Kibu sees Momentic as a long-term pillar in how it scales testing across different platforms. ## From manual bottlenecks to automated scale: How Coframe validates thousands of website variants _Coframe · AI_ URL: https://momentic.ai/customers/coframe > Momentic saw our proof-of-concept as a legitimate value add and worked with us to formalize it. is an AI-native growth marketing platform that autonomously generates and deploys website variants that drive conversion lift and outperform both SaaS and CRO agencies. Trusted by teams at industry-leading companies such as Dropbox and Intuit, Coframe empowers its users to scale experimentation without adding operational overhead. ### Challenge ## Ensuring quality across an expanding number of customer-facing website variants Coframe was founded on the idea that teams don’t need a team of engineers, designers, and marketers to optimize their UI. By using generative models trained on high-performing web patterns, the platform enables companies to continuously and autonomously improve and personalize their UI at a velocity few internal teams or agencies could match. However, Coframe’s promise isn’t just speed; it’s delivering high-quality, production-ready optimizations at scale. Every UI variant must integrate cleanly with customer environments and perform as intended from the moment they go live. If a single bug reaches production, it can trigger a series of failures (from broken checkouts to failed forms) that block real purchases and muddy optimization insights. As Founding AI Engineer, was tasked with ensuring quality across each deployment. At the time, Coframe relied heavily on backend unit tests combined with developer-led manual checks during implementations. While backend coverage was robust, manual testing was slow, fragmented, and inconsistent. Over time, the risk of issues surfacing at the interaction layer continued to climb. Glavin first began experimenting with Playwright. While the tool helped standardize testing workflows, the technical overhead associated with maintaining suites for ephemeral variants was significant. Knowing he couldn’t simply throw headcount at the issue, Glavin began searching for an automated testing platform that could validate comprehensive frontend flows without introducing additional maintenance overhead. Fortunately, Glavin was already using in a limited capacity to test Coframe’s internal product workflows. As the number of website variants continued to grow, it became clear that the platform could also support the broader validation needs of Coframe’s experimentation engine. That realization led him to formalize Momentic’s adoption across the team. ### Solution ## Automating end-to-end testing with Momentic With Momentic, Coframe transformed frontend testing from a manual bottleneck to its competitive advantage. Instead of maintaining selector-heavy scripts tied to specific DOM structures, Glavin now authors tests by describing user workflows and expected outcomes. Momentic interprets those steps against the deployed interface, identifying elements contextually rather than binding to static identifiers. When multiple similar elements appear on a page, Momentic surfaces that ambiguity and prompts clarification before execution, preventing silent mis-clicks and reducing false positives. This shift reduces maintenance overhead and enables validation to scale alongside development. When UI structure changes, tests do not require constant rewrites because they are interpreted in context rather than rigidly coupled to implementation details. Within two weeks, Glavin had integrated Momentic into Coframe’s CI, and validation became a seamless part of the release process. Chrome extension compatibility, essential for Coframe’s product, was identified early, and the Momentic team moved quickly to enable local support, removing what could have otherwise been a blocker. Beyond validating internal product flows, Glavin focused on a larger objective: ensuring that the growing volume of customer-facing UI changes generated by Coframe could be validated automatically. He built a proof of concept that translated AI-generated variant descriptions into structured acceptance criteria, then executed those validations through Momentic runners. Rather than discouraging the approach, the Momentic team collaborated to explore a more formal integration path via an official SDK. ### Results ## Scalable validation, expanded coverage, and a future-proof testing foundation Partnering with Momentic enabled Coframe to replace manual checks with a reliable, behavior-driven testing foundation that scales alongside the company. By slashing time spent on validating UI variants, Glavin reclaimed engineering bandwidth that he now directs toward higher-value initiatives. Moving forward, Glavin is eager to expand Momentic’s influence across Coframe’s non-technical operators, empowering them to create and execute end-to-end tests that wholly align with customer behavior. ## How Best Parents got to 80% coverage in two days without writing a single line of code _Best Parents · Consumer AI_ URL: https://momentic.ai/customers/best-parents > It's all very visual and intuitive. A junior developer can do better in a week than I can. is a global education company that connects parents with internationally accredited summer and winter camps for their children. They work with over 2000 accredited education camps and recruit from over 70 countries. Successfully operating a global marketplace is a daunting engineering challenge: 1000s of pages, complex checkout flows, and supporting multiple locales. Providing a top-notch customer experience to their thousands of visitors each month requires a lot of testing. ### Challenge ## Existing testing tools were too hard to build and maintain Best Parents' engineering team was responsible for building their web application. Testing was critical to ensure their customers could always search and book camps. A single broken link or interaction can have enormous consequences. Aditya's team first used , but faced immediate challenges: it was too brittle and complex to set up. The constantly failing tests put a strain on Best Parents' engineering team and often required manual fixes for basic things like finding the right element. ### Solution ## Eliminating the testing overhead After finding Momentic by searching the web, Aditya's team began building their tests in Momentic to save time and improve their coverage. Engineers could now build complex tests in minutes without thinking about CSS selectors, test infrastructure, or even code! Momentic also upleveled the engineering team as a whole. ### Results ## From months to just days Momentic cut the time it took for Aditya's team to build end-to-end tests dramatically. The build process for tests got much shorter, but maintenance was also significantly easier - constantly updating UI no longer caused test failures. Manual release testing was automated and scheduled instead of frustrated and forgotten. Best Parents has 30+ active tests running hourly in Momentic. Regardless of technical expertise, engineers at Best Parents can now build and execute complex end-to-end tests by themselves without relying on help from other engineers. --- # Blog ## The Harness Will Not Save You _Quality · Wei-Wei Wu_ URL: https://momentic.ai/blog/the-harness-will-not-save-you AI "harnesses" are essential, but not enough to prevent bugs. Teams need to build an independent, behavioral verification layer that tests your software the way a real user would. Models will make mistakes. Right now, Fable, K3, Sol – all of them – do stuff wrong. It is inevitable. Model makers work on reducing that mistake space. But developers also have to work to reduce the mistake space around the model. Better guardrails, better context, better tools, better sandboxes, better checks the agent runs before it calls the work done. All of that is the harness, and it works. The mistake rate really does go down. But the harness will not save you. Harnesses build better code; they don't build a better product. To build a better product, you have to work with the model and the harness on verification. ## What is a Harness? Rules and walls. The rules are everything the agent is told: - The conventions in CLAUDE.md are rules. - The prompt templates and plan formats are rules. - The instruction to run the test suite before opening a PR is a rule. - Even tool descriptions are rules about when and how to act. The rest of a harness is walls. A sandbox doesn't ask the agent to stay inside it; there's nothing outside it to reach. Permission gates, egress controls, read-only mounts, and protected branches work the same way. They hold whether or not the model cooperates, which makes them the most dependable part of the whole setup. [Harness engineering](https://openai.com/index/harness-engineering/) is the work of improving both halves. Neither half examines the thing the agent produced. A rule can't confirm it was followed, and a wall can only tell you what didn't happen. [Endor Labs put numbers on all of this](https://www.endorlabs.com/learn/gpt-5-5-sets-a-new-code-security-record-with-cursor-not-codex-in-agent-security-league) by running the same models through two different harnesses in the same week: - GPT-5.5 scored 61.5% on functional correctness in its own Codex harness and 87.2% in Cursor's, a 25.7-point swing from the harness alone - Opus 4.7 went from 87.2% in Claude Code to 91.1% in Cursor Even the best pairing still failed roughly one task in eleven. And the only reason anyone knows that rate is that the benchmark grades the work from outside the harness. Production has no equivalent grader unless you build one. ## The Harness's Relationship with Testing Ask a team where verification lives in their setup, and they'll usually point at the harness. The rules file says to write tests and run them, the agent does, and the PR arrives green. Testing gets handled as one more rule, so it gets the same treatment as every other input. [Anthropic’s guidance on verification loops](https://claude.com/blog/building-verification-loops-in-claude-code-with-skills) defines the pattern as Claude checking its own work and fixing what fails, and shows how to encode your manual checks as skills so the agent closes its own feedback loop. There's real tooling behind it: - A /verify skill builds and runs the app to observe the changes. - Spec validation checks each change against a markdown spec in the repo. - Rubric graders run as a separate agent and loop failures back for rework. The Claude Code team chains them in its own work: code review, then simplify, then verify, then a design check. Every check is instructions the agent reads. The spec sits in the repo where the producing agent can edit it. The rubric is visible to the side being graded. The important part is this: **the loop is closed by the thing the loop is checking**. Why that matters starts at training time. Coding models are reinforcement-trained against pass/fail test rewards, and graders already discard agents' edits to test files because models comment out failing tests and splice in mocks. The reward is passing the visible check. Nothing in it asks whether a user can complete the workflow. The same behavior shows up at inference. [SpecBench](https://arxiv.org/abs/2605.21384) found every frontier model saturates the test suite it can see, while the gap against held-out end-to-end tests widens with scale, roughly 28 points for every tenfold increase in code size. One agent submitted a 2,900-line compiler that memorized the test inputs. Models align with incentives. If the incentive is passing the test, a capable model has many avenues for that, and writing correct code is only one of them. Just look at what just happened with an unreleased OpenAI model [escaping its sandbox and breaking into Hugging Face's production database](https://huggingface.co/blog/agent-intrusion-technical-timeline) for the answer key. Models align with incentives. A green suite the agent wrote, ran, and reported tells you the agent satisfied its checks. Whether the software works is a separate question, and answering it takes an oracle that sits outside the agent and that the agent can't edit. Everything in the harness sits inside. ## Build the verification layer yourself The gap is already showing up in production: - [Faros AI's report](https://www.faros.ai/research/ai-acceleration-whiplash) on the months since coding agents went mainstream shows monthly incidents up nearly 58%, bugs per developer up 54%, and roughly a third of PRs merging with no review at all. - A [May 2026 CloudBees study](https://www.cloudbees.com/lp/2026-state-of-code-abundance-report) found 81% of enterprise technology leaders reporting a direct increase in production issues tied to AI-generated code. Those are correctness failures, the exact thing all those green checks were supposed to catch. Verification has to be built as its own layer, with three properties no harness provides: - **It's independent**. The checks are authored and run outside the coding agent, and a separate grader doesn't clear the bar if it reads a rubric the producing side can see and edit. - **It's behavioral**. It exercises the running software the way a user would and judges the outcome rather than the diff. - **It's in the loop**. It gates every merge at the speed agents ship, because the mistakes arrive continuously now. The execution of this should be fully agentic. [Agents author the tests](https://momentic.ai/blog/agentic-testing-guide) from plain-language intent, and on every PR an exploration agent reads the diff and adds or updates coverage for the flows that changed. The runs then happen against the preview deployment, and the checks judge outcomes through the signals a user's browser produces: the DOM, the accessibility tree, screenshots, network traffic, the console. Finally, failures get triaged by an agent into real bugs, intentional changes, setup problems, or flakes. It repairs the tests behind intentional changes and quarantines the flaky ones. Nobody writes scripts, nobody maintains them, nobody manually tests. The only failures that reach a person are the ones classified as real. The judgment stays with the humans: - You state the intent: what the product is supposed to do, in plain language, before any agent turns it into a test. - You keep the [knowledge base](https://momentic.ai/docs/ai/knowledge-base) that records how the product is supposed to behave and what counts as a bug versus an intentional change. - You make the final call on what verification finds and whether a change ships. That definition is the oracle every check runs against, and it belongs to the team rather than the agent being graded. Most teams got the order backward. They automated the building and left verification to the builder. Build the layer, hand it your definition of correct, and then let the agents run. The harness is worth every bit of investment it's getting, and it will keep improving. It embeds rules. Verification is something you have to build. If you want to see what that layer looks like on your own app, the [setup wizard](https://momentic.ai/docs/quickstart/web) gets you from install to a passing test in a few minutes: run `npx @momentic/wizard@latest` in your project. Or [contact the team](https://momentic.ai/sales) to walk through it. ## The Set Theory of Agentic Testing _Quality · Wei-Wei Wu_ URL: https://momentic.ai/blog/set-theory-of-agentic-testing Why agentic testing isn't just another form of AI-assisted authoring, and how applying set theory helps you identify which tests belong in an agentic loop. Here’s our [definition of agentic testing](https://momentic.ai/blog/agentic-testing-guide): That is a tight definition. But a definition tells you what agentic testing is, not where it sits. In a real codebase, it shares a workflow with hand-written unit tests, AI-generated scripts, self-healing selectors, and manual passes. When thinking about implementing agentic testing, the obvious question that matters is where the line is. What counts as agentic testing? What's next to agentic testing? What resembles it but isn't? What does it overlap, and what does it leave alone? To answer those questions, start thinking in [sets](https://en.wikipedia.org/wiki/Set_(mathematics)). ## The testing set Agentic testing is a set. It sits inside a larger set, testing with AI, which sits inside the largest set of all, testing itself. The universe is T, every test from the first line. Testing with AI, call it W, is the subset where AI contributes anywhere in the workflow: authoring, execution, observation, diagnosis, or maintenance. Everything in T outside W is tested with no AI in it; the deterministic, manual work that's still where most testing lives. Inside W, AI-assisted testing (A) is human-authored test creation with AI's help. A person states the intent, usually in natural language, and AI turns it into the artifact: a test case, a Gherkin scenario, a Playwright script, an assertion, a checklist. The human owns the test. Once it exists, it runs like any other test. Agentic testing, G, is testing that lives inside an agent loop. The human gives a goal, and the agent plans a path, acts on the app, observes what comes back through the[ DOM, accessibility tree, screenshots, network requests, console logs, and browser state](https://momentic.ai/blog/how-agentic-testing-works), and adapts the path until it can reach or verify the goal. The agent owns the path. It can change a step, re-resolve a renamed button, or add an action the UI now demands, but it does not rewrite the goal to make the run pass. A and G both sit inside W, and neither one contains the other. ``` A ⊂ W ⊂ T G ⊂ W ⊂ T A ∩ G is small, maybe empty ``` The notation names three regions, but two sets crossing inside a third cut the space into five: Region | What it is Outside testing with AI | no AI: hand-written, deterministic, manual Inside the umbrella, outside both circles | AI assists: triage, test selection, judging, data, self-healing AI-assisted only | a human authored the test with AI's help; it runs fixed Agentic only | a human or a machine gives an intent, and an agent runs the whole loop Both | one test mixes AI-authored fixed steps with an agentic step Three of those five are the ones worth most of your attention: AI-assisted only, agentic only, and the overlap where both are true. The rest of this takes the five regions in turn, starting with the two that are often confused with each other. ## What's only AI-assisted testing? A person owns the test, and AI helps write it, but once written, it runs as a fixed artifact. No loop, no runtime adaptation. This is where most AI in testing actually ships today. - "Generate a Cypress test for this checkout user story," and you commit the result. - Natural-language test cases generated from a feature description. - Gherkin scenarios written from a requirements doc. - LLM-generated Playwright or Selenium scripts you run unchanged. - Assertions that an LLM writes for an API response you captured. - Edge cases an LLM suggests from a product spec, once you turn them into tests. - Acceptance criteria for an LLM converts into a regression suite. - A flow you record by hand, which AI then converts into a maintained script. The thing AI sped up here is writing the test, not living with it afterward. You get the script in seconds, but it's the same thing you'd have written by hand: a fixed list of selectors and assertions that does the same thing on every run. The model does its part once, and then you're left with a static file. So it breaks the same way it always did. Selectors snap when the UI moves, assertions go stale, and the models writing your app are changing that UI all the time. You got the first draft faster and kept every bit of the maintenance. There's also a catch with the test itself: the model can write an assertion that's just wrong, one that passes for the wrong reason, so you still have to read what it wrote before you trust it. ## What's only agentic testing? The test runs inside an agent loop. You give an intent in plain language, and the agent works out the test itself, the steps, the selectors, any code it needs, then runs it against the app and adapts as it goes. Nothing gets saved as a script you maintain. The agent owns the whole path, and the intent it works from can come from a person or from a machine. - "Verify a new user can sign up and reach the dashboard," and the agent plans the steps, drives the browser, and works out the checks itself. - It decides pass or fail based on several signals at once: the DOM, the accessibility tree, a screenshot, the network calls, and the console, so it's checking behavior rather than whether one element exists. - It hits a newly required field it didn't plan for and fills it instead of failing, or re-resolves a renamed button and caches the result so the next run skips the model. - A coding agent launches a browser and verifies the UI change it just made. - A CI agent reproduces a failing test, tries other paths, and proposes a fix. - An agent crawls the critical flows from a sitemap, builds a repro from a production error, and checks it locally. - A Momentic [agentic step](https://momentic.ai/docs/core-concepts/agentic-testing), where you state a goal and the agent plans, acts, observes, and adapts to reach it. This is why [agentic testing is its own category](https://momentic.ai/blog/agentic-testing-guide), not a kind of AI-assisted testing. Agentic testing isn't traditional automation, and it isn't codegen from natural language that you then maintain. The difference is the loop and the missing artifact. There's no saved script someone has to keep alive, so the breakage from a renamed class or a moved element mostly goes away. The agent re-resolves it and keeps going. The one thing the agent can't touch is the goal. Say the goal is "user reaches the dashboard," and signup fails. If the agent quietly decides "signup form showed up" is good enough, it just moved the goalpost to pass itself. It can change the route, but the bar for success stays yours, which is why this kind of test needs a real oracle it checks against and can't edit, rather than an assertion it could quietly rewrite. Once a test can reach its goal more than one way, a single hardcoded assertion stops being enough. You give something up for it. A fixed script is brittle but easy to read; an agent loop is sturdier but harder to pin down, and it costs you in speed and in runs that don't come out identical every time. You make the trade when the UI churns enough to keep breaking scripts, or when a coding agent needs to check its own work as it goes. Self-healing doesn't reach this far. If a model only patches a broken locator within a script that otherwise runs correctly, the script is still calling the shots, and there's no real loop. That's assistance, and it sits in the margin. ## Does anything land in both? Most of what looks like both is just agentic. A person giving an intent and an agent running the loop is agentic testing, not a blend, because the person never authored a fixed test for the AI-assisted side to claim. To land in both, one test has to carry an AI-assisted authored part and an agentic part at the same time. That's narrow, and it may turn out to be empty, but a couple of things fit. - A test that's mostly AI-generated, with deterministic steps and an agentic step or two dropped in where the flow won't pin down. The fixed steps came from AI-assisted authoring and run the same way every time; the agentic step runs the loop. One test, both circles. - An AI-generated script handed to an agent as a starting plan, which the agent runs and adapts instead of executing line by line. The script came from AI-assisted authoring; the run is agentic. Though once the agent re-plans freely, it's fair to ask whether the script still matters or it's just agentic with a head start. Outside cases like these, the two sets mostly sit next to each other rather than on top of one another. ## Where does AI assist without authoring? AI helps with testing, but it didn't write the test, and it isn't running it. A person authored the test, it runs the way it always did, and AI works around it. - AI triages a failing run, clustering errors and proposing a root cause. - AI classifies which failures are flaky and which are real. - An ML model picks which tests to run for a given change, the test-impact-analysis case. - AI clusters logs after a failure to pinpoint the cause. - An LLM summarizes a CI run so a human can read it fast. - AI generates synthetic test data for an otherwise static suite. - An LLM-as-judge on outputs, scoring them against a rubric. - AI flags visual anomalies in rendered output. - Self-healing selectors patch a locator inside a scripted test. What determines whether a test stays in this box rather than the AI-assisted circle is whether AI actually wrote the test. If it generated the data your hand-written assertion checks, that's help. If it wrote the assertion, that's authoring. LLM-as-judge is the one that trips people up: the model is acting as the judge of pass or fail, not writing the test and not driving it, so it stays here until you wire it into a loop. None of the trade-offs from the other boxes show up here. The test still runs fixed and deterministic and easy to read, and AI just trims the work around it. For many teams, this is the safest place to introduce AI, and it's worth treating it as its own thing rather than a watered-down version of AI-assisted testing. ## What is testing without AI at all? No AI anywhere. A person wrote it, a person or a runner executes it, and it behaves the same way every time. OG testing. - Human-written unit and integration tests. - Hand-coded Selenium, Cypress, or Playwright scripts. - Manual exploratory testing and manual test cases. - Performance and load tests. - Contract tests between services. - Snapshot tests with no AI in the diff. - Prompt regression tests a human wrote and a human judges, even though the system under test is AI. It's easy to write this box off as the old way of doing things, but most of it should stay because it's deterministic. A hand-written unit test is fast, easy to read, and only goes red when something actually broke, which is what lets it block a merge with nobody watching. Property tests and fuzzers throw inputs at your code that you'd never think to list out by hand. None of this is waiting around to be replaced by an agent. This box is also what everything else leans on. When an agent runs a loop, the thing it checks its work against is usually a plain deterministic assertion. So these tests stick around as testing gets adaptive, just in a new role, as the oracle the loop measures itself by. The real question is which tests have a good reason to leave, not how to empty the box. And it's a short list: end-to-end and integration flows over a UI that won't sit still, where keeping a fixed script alive finally costs more than the determinism is worth. Unit tests, perf tests, and property tests mostly have no reason to move. ## How to place any test Four questions settle almost everything. 1. Is it producing evidence about expected behavior? If yes, it's in T. 1. Is AI used anywhere in the testing work? If yes, it's in W. 1. Is AI mainly helping a human author the test, or acting inside the execution loop? Authoring from intent goes in AI-assisted; a plan-run-observe-adapt loop goes in agentic; and both go in the overlap. 1. Is the system under test itself AI? This one moves the test nowhere. It's a separate axis, and it's why a human-judged prompt eval lands outside W. Two tiebreakers catch the rest. Self-healing that only patches a locator inside a fixed script is assistance, not an agent loop. And adaptivity alone doesn't make something agentic: fuzzing and exploratory testing change course at runtime without any agent making the calls, so they fall outside W. ## The set your tests sit in, and where they're moving to A definition tells you what agentic testing is. The sets tell you where it sits: inside testing with AI, beside AI-assisted testing, apart from the deterministic work that's still most of any suite. That's what the definition leaves out, and it's what lets you point at any test you own and name its set. The sets aren't fixed, either. As agents churn your UI and write more of your code, tests shift out of the brittle fixed-script sets toward the agentic loop, while the deterministic base stays put. Knowing the sets is what makes that shift a decision rather than something flaky selectors force on you. ## A New Era of Software Quality Starts Today _News · Wei-Wei Wu_ URL: https://momentic.ai/blog/a-new-era-of-software-quality Today we're announcing a new Momentic - a major platform update, a new brand, and a way for every developer to experience the future of testing for themselves. Today we're announcing a new Momentic - a major platform update, a new brand, and a way for every developer to experience the future of testing for themselves. ### Why we rebuilt the platform Jeff and I founded Momentic around a problem we lived with every day. As former engineers at Robinhood, Qualtrics, WeWork, and Retool, we watched the widening gap between how fast teams could write code and how confidently they could ship it. AI coding agents have only made this gap impossible to ignore. More code is shipping faster than ever, and that means [more bugs and incidents](https://www.geekwire.com/2025/how-the-aws-outage-happened-amazon-blames-rare-software-bug-and-faulty-automation-for-massive-glitch/) reaching production. According to [Faros AI's 2026 AI Engineering Report](https://www.faros.ai/blog/ai-acceleration-whiplash-takeaways?utm_source=fnf&utm_medium=linkedin&utm_campaign=faros-june&utm_term=kbeck&utm_content=whitepaper), monthly incidents are up nearly 58% since AI adoption accelerated. And a [May 2026 CloudBees study](https://www.cloudbees.com) found that 81% of enterprise technology leaders say they've seen a direct increase in production issues tied to AI-generated code. This isn't a prediction, it's already happening. The code is shipping, but that was never the real bottleneck. QA, which was already painful, is now falling more behind. ## What's new in Momentic Over the past few weeks, we've been working closely with AI-native engineering teams whose applications collectively serve hundreds of millions of users, spanning productivity, media streaming, consumer applications, and professional services. They've been invaluable partners in shaping how agentic testing really should work alongside rapidly evolving developer toolchains. One of our customers said it best: That insight became the foundation for everything we built next. The teams we worked with weren't asking how to write better test scripts. They were asking for an always-on, autonomous system that actually understands their product, knows how it's supposed to behave, and gets smarter over time. Humans just have to review a report, not manually test. So here’s what Momentic customers now have access to: ### Introducing memory and knowledge base The best QA tester you’ve ever worked with didn’t just catch bugs. They knew your product's terminology, the flows that were brittle, the edge cases nobody had documented. That knowledge lived in their head, and when a new hire onboarded, they simply ‘got it’ too. We built a way for that knowledge to live inside the platform instead, with our new [Knowledge Base](https://momentic.ai/docs/ai/knowledge-base#knowledge-base). Teams define how their product is supposed to behave, what counts as a bug versus an intentional change, what terminology means in their specific context. Every agent, whether it's writing new tests, triaging failures, or proposing fixes, runs on that shared understanding. The more your team puts in, the smarter it gets. Every team has a different bar for what "quality" means. Some want pixel-perfect accuracy; others just need the user to reach the destination. Now you can customize the platform accordingly. ### Coverage that grows with your product Writing tests is usually the last thing engineers want to do. And with AI accelerating the number of commits daily, the gap between what's shipping and what's been verified keeps growing. [Explore Agent](https://momentic.ai/docs/ai/explore) closes that loop automatically. Every time a PR lands, Momentic reads the diff, identifies what changed, and proposes new or updated tests, already scoped to the flows that matter, already consistent with your existing suite. It notices the new features, the renamed components, the edge cases that weren't there last sprint. Over time it gets better at this, because it's learning your product as it goes. ### Failures that mean something Flaky tests are one of the worst problems to have in your test suite. Not because they're annoying, because they train engineers to ignore failures. And that's exactly when real bugs slip through. Every failure in Momentic now gets triaged through our [Failure Classification Agent](https://momentic.ai/docs/reliability/failure-recovery#failure-recovery) and analyzes its root cause. Is it a real bug, intentional application change, test setup issue, or transient error? If it's an intentional change that triggered a test failure, Momentic opens a pull request to fix the test itself. If it's a real bug, the team gets a high-signal alert with full context on what broke and why. The result is a test suite that compounds trust over time. ### The spec _is _the test For too long, test scripts have been artifacts that only the engineer who wrote them can understand. The [new Momentic test format](https://momentic.ai/docs/get-started/migrate-to-simplified-format#6-author-simplified-format-files) is intent-based and readable by both humans and AI agents. Engineers just have to describe what they want to test in plain English. Plus, AI agents can parse, build, and modify tests more effectively - making the entire development loop faster and more autonomous. ### A new brand Our platform evolution meant Momentic needed a new expression to match, one built around the same two qualities we engineered into the product itself: ease of use and guardrails that can evolve autonomously with your product. The grid structure you'll see throughout our new site is intentional. Structure and reliability are still the foundation. But there's motion in it now, an energy that reflects what it feels like when your agent catches a bug before production, when tests update themselves, and the agent unblocked itself mid-execution and kept going. That's the energy the new brand is built to match. ## Quality for everyone Quality used to be a function of how big your QA team was. Every developer, on every team, regardless of size, deserves quality. And it should be easy, enjoyable, and built into the foundation of how software gets made. Today, Momentic is open to every software engineering team. It is free to try yourself. Our philosophy hasn't changed since day one: we don't want anything standing between teams and shipping. Try it out yourself: `npx @momentic/wizard@latest` ## How We Ditched Postgres for ClickHouse to Process 12 Billion Caches Per Day _Engineering · Henry Haefliger_ URL: https://momentic.ai/blog/postgres-to-clickhouse-migration From Postgres pain to ClickHouse speed: how we re-architected caching to serve 2M+ cache queries and 20B entries per day, while maintaining ~250ms average resolution latency. _This is part 2 of our series on how Momentic scaled from 80k to 1B caches and successfully migrated from Postgres → Clickhouse. Read part 1: _[_Most browser agents "see" the DOM. Ours had to understand user intent._](https://momentic.ai/blog/teaching-browser-agents-user-intent) ## **Growing pains using Postgres** Adding more values to the cache key fixed many of our consistency problems, but it also meant that we went from having around 80k active cache entries to now approximately 1B. Our previous setup was simple: we stored the caches in a single table in Postgres, but this started to show cracks pretty quickly. Because of the high-write, high-read workload, we were running into both elevated resource usage as well as lock contention from queries trying to read and write to the cache concurrently, and as we increased the number of entries by several orders of magnitude, this only became worse. ## The decision to move to ClickHouse We decided to move our storage backend to ClickHouse in order to improve the performance of the cache lookup query. As Momentic grew, we were running it 600k times per day. Because caches are such a critical piece of our infrastructure, we needed to make sure that we could serve them with 100% uptime and sub-second latency. In theory, because we essentially only had a single query (select caches where test ID, CLI version and branch name and commit timestamp match some filters) we could leverage ClickHouse’s primary key to make that query super performant. In order to understand this decision, it’s important to understand what Momentic’s cache data looks like. The cache is keyed by test ID, step ID, Momentic version, git branch, and commit timestamp. For a given test ID, Momentic version, git branch, and commit, there are essentially a fixed number of values (the number of steps in the test, most often between 10 and 100). Postgres indexes are B-tree based, which means that the query cost will always increase with the amount of data. On the other hand, ClickHouse uses a sparse primary index which means that if we know the four key values for every query, we can very efficiently narrow down the search space to just a few granules. ## How we optimized our ClickHouse architecture ### Choosing the right primary key ClickHouse’s primary key got us 90% of the way there. On **non-main branches**, we were able to easily identify the data part containing the cache entries for our test, read it into memory, and compute the matching entries (10k-30k rows). But on **main branches**, we weren’t so lucky. We were still searching over all entries with a lower commit timestamp than the current commit’s (potentially 500k+ rows). This meant that most queries were reading 1-2 data parts, but some outliers were reading almost all of the parts on every query, leading to spiky memory usage and disk operations. This also made it very difficult to serve traffic reliably since query performance was heavily dependent on the specific customer’s data and usage pattern, and a few rogue queries could send our infra into a tailspin. ‍ To address this problem, we used a **materialized view** to precompute all of the available commit timestamps for a given test ID. This view was significantly smaller than the main table and could trivially be used to determine the best available commit timestamp. Once we had that we could search the main table for entries with that specific timestamp, thus narrowing it back down to one or two parts. ## Replacing with for extending TTL In Postgres, we would make three queries for every test run: `SELECT` to get the caches, `UPDATE` to increase the TTL for the caches that were used (this was debounced using Redis), and `INSERT/UPDATE` to store the updated caches. This didn’t play super well with ClickHouse because potentially 2/3 queries are updates, which aren’t very performant. Instead we switched to using only `INSERTS` combined with ClickHouse’s `ReplacingMergeTree`: `SELECT` to get the caches, re-`INSERT` the caches that were used to extend TTL, `INSERT` the new caches after the test run, and let ClickHouse take care of deduplicating entries asynchronously. This was such an improvement that we were able to fully eliminate the Redis layer, which at this point had limited value due to the higher cache key cardinality. From this: To this: ‍ ## How we migrated from Postgres → ClickHouse ### Double write In order to ensure that no data was lost during the migration and to validate our new approach, we started by double writing caches to both Postgres and ClickHouse. The 14 day stale-cache expiration guaranteed that after two weeks, the two databases contained exactly the same values. ### Double read + consistency check Once the same data was stored in both systems, we began validating the correctness and performance of our ClickHouse queries. During this phase, users were served caches from Postgres, but in the background we executed the same query against ClickHouse and compared the results, flagging any discrepancies. This allowed us to: 1. Make sure that our results were consistent between the two databases, and 1. Validate that our ClickHouse architecture was performant and held up at scale It was during this phase that we were able to iterate and improve performance by creating the materialized view of commit timestamps. ## The switch to ClickHouse Once we were confident in our ClickHouse setup, we gradually cut over production traffic from Postgres to ClickHouse, still keeping the double write in case we needed to roll back. After an initial validation period, we stopped writing to Postgres in the background. ## The results Migrating to ClickHouse has allowed us to handle over two million cache queries per day, processing almost 20 billion cache entries every day while maintaining ~250ms resolution latency on average. This new infrastructure allowed up to roll out the accuracy improvements we wrote about in our [previous article](https://momentic.ai/blog/teaching-browser-agents-user-intent) at scale. ## Next steps Going forward, some challenges that we still want to solve include stronger relativity checks and better validation for SVGs: - While we currently capture information about related elements, we still don’t do a good job encoding how they need to be related to the target. In an ideal world, we would also enforce things like how close the elements are, where they’re positioned relatively, or how they’re related in the DOM. - Many of our customers’s sites contain a lot of icons or icon buttons. Often, these buttons are positioned closely together and are easy to mistake for each other. In some cases, when there are aria attributes or other labels, we can handle these well. We are working towards the ability to validate that our cache resolves to the same icon every time using the SVG itself, even if the site isn’t accessible. ‍ _Thanks for reading! Read part 1 of our series on caching: _[_Most browser agents "see" the DOM. Ours had to understand user intent._](https://momentic.ai/blog/teaching-browser-agents-user-intent) _If this kind of problem interests you: connect with us on _[_**LinkedIn**_](https://www.linkedin.com/in/hungweiwu/)_, _[_**X**_](https://x.com/MomenticAI)_, or check out our open opportunities at _[_**momentic.ai**_](https://momentic.ai/)_._ ‍ ## Most Browser Agents "See" the DOM. Ours Had to Understand User Intent. _Engineering · Henry Haefliger_ URL: https://momentic.ai/blog/teaching-browser-agents-user-intent Browser agents look at the DOM. Momentic understands what the user actually meant. Here's how we scaled intent-based caching to maintain a 95%+ cache hit rate and be significantly more reliable. _Part 1 of our series on teaching browser agents to understand user intent at scale: how we scaled from 80k to 1B caches, serving 1M queries per day, while migrating from Postgres to ClickHouse_ Caching has always been key to keeping Momentic more cost-effective, fast, and reliable. As an end-to-end quality and testing platform, the two key metrics that build customer trust are a) how quickly we can identify when things break, and b) how consistently we deliver the same results given the same inputs. Both come down to the same underlying problem: teaching the agent to understand and remember what the user actually meant. **Speed**. Fast feedback keeps the development loop tight: PR checks finish quickly, and engineers can keep merging code. **Accuracy.** Consistency protects trust: if tests are always passing locally but flaking in CI, eventually the entire suite gets ignored. Using AI allows Momentic to intelligently adapt to customers’ product changes, reducing false positives (unnecessary failures). Fewer failures mean engineers spend less time context switching and product ships faster. However, AI is also expensive to run, and doesn’t always behave the same way. If we were to rely only on AI, it would be incredibly costly to run, each step would take several seconds to execute, and sometimes it would hallucinate clicking the wrong button. ## How Momentic generates and resolves caches In Momentic, any step that interacts with an element (think click, hover, or even element checks) can be cached. After using AI to locate an element based on the user’s intent, Momentic generates a list of unique, matching CSS selectors for that element. We also snapshot the DOM node. During subsequent runs, we evaluate all the selectors against the current page state and group them by matched element. The element with the most matching selectors is evaluated against the previous snapshot to determine if it’s similar enough to the original to be reused. ## Problems with cache accuracy as Momentic scaled As Momentic scaled to millions of daily runs on its platform, across 1000+ engineer organizations, we started to encounter a couple of common failure modes. ### **Failure mode 1. Cache pollution across branches** Different branches contained various minor product changes which might cause a cache that resolves on one branch to not work on a different branch. Since all branches shared a single cache, it was common for elements to get stuck in loops of cache busting, where one branch would regenerate a selector that worked with it’s changes, but didn’t work with another’s, causing the other to in turn regenerate the cache. This was impacting users because it led to tests touching product areas where many engineers were making changes running slowly because they were almost never cached. It also drove up our own costs because every time the cache busted, we had to pay for an AI completion. ### **Failure mode 2. Cache pollution across Momentic versions** Newer Momentic versions generated caches optimized for the latest release. If a repository contained branches with different versions, the newer ones corrupted caches for older ones. When customers changed Momentic versions, branches still using the older version would often see very low cache hit rates because the older CLI couldn’t use caches generated by the newer one. This meant that every upgrade led to a period of slower execution and higher costs until everyone on the team rebased onto the main branch. ### **Failure mode 3. False cache misses (validation too strict)** Because of the way we compared snapshots of DOM nodes, it was very common to bust the cache over inconsequential changes. It’s common for sites to have randomized classnames, IDs, or hrefs. When these changed, it often caused the cache to bust despite the element itself still appearing the same to the user. Minor styling changes or even className randomization across releases caused unnecessary cache busting and slowness. ### **Failure mode 4. False cache hits (validation too loose)** For short elements without many attributes, often in lists and tables, it was common for our validation to miss the features that mattered to the user’s intent. For example, the user might say “the menu button in the Alice row.” In this situation, the majority of the selectors would often be nth child based, causing them to select the wrong element if the ordering of the rows changed in later runs. This is especially hard to debug for users who aren’t very familiar with how Momentic works. This would lead to either incorrectly passing tests, or tests failing later on inexplicably. For example, if I accidentally decided to open the menu for Bob, I might not realize until ten steps later when checking that Bob is in my team that I actually edited the wrong user. ## How we stopped cross-branch cache pollution In order to prevent interference across branches and versions, we had to isolate their caches. We started by storing a different cache for each branch/version combination. This made the caches fully isolated, but still had a few limitations: 1. When you create a new branch, there’s no cache, so inevitably, the first test runs on that branch are slow and unpredictable. 1. Conversely, when a pull request gets merged back into main, the cache changes from that branch are forgotten, leading to unpredictable behavior when testing changes that were made on the branch. We observed several instances where tests would pass on a feature branch but fail on main due to AI consistency issues. 1. Similarly, when you change Momentic versions, results are slow and unpredictable because the cache is reset. In order to solve the branching problems, we had to allow branches to inherit caches from other branches. We did this by tracking when they were created, and seeding their caches with the latest values from the merge base commit on main. We ensured continuity on main by checking if a commit was created by merging a branch, and if it was, also pulling in the latest caches from that branch to make sure that we had the complete picture. We also enforced that as new versions of our CLI were released, they could only use caches generated by versions less than or equal to their own version. This prevented them from attempting to use caches that were incompatible. We did this by filtering out caches that had a version greater than the current CLI version during resolution. ## How we taught the cache to respect user intent When caches were created, there was nothing explicitly verifying that the resolved element matched the user’s original intent. This was particularly challenging because the validation rules need to change depending on the user’s prompt. To illustrate this issue, consider the following scenario: If the user’s prompt was “the add to cart button,” the cache can be used in both cases, but if the user’s prompt was “the blue button,” the cache must be busted. Our similarity check was a best-effort attempt at a generic criteria, but it was far from perfect. In order to solve this problem, we needed to generate some conditions to dynamically verify that we remained consistent with the original query. We landed on two types of conditions which can be used to validate the element: 1. Attributes are properties of the element itself that must remain the same in order for the element to match the user’s query. These could include text, color, or any arbitrary HTML attribute. For example, the user might ask for “The large profile image in the middle of the page.” In this case, the element must be an image, be large, and be located in the middle of the page. Depending on the user’s site, it might also need to have alt text like “profile image.” 1. Related elements are other elements on the page that the user used to identify their target. Related elements can themselves have required attributes. For instance, the user could say “the login button above the sign up button.” In this case, there’s one related element: the sign up button, which must say “sign up” and be below the main element. In order to generate these, we modified our element locator agent to classify which attributes it used in its reasoning to identify the element. We also had it generate targets for all of the additional elements that it considered. In our experimentation, we found the best results when these classifications were the last thing generated, after the reasoning and the target element itself. This new approach is a bit of a departure from how we’ve historically validated caches. In effect, we stopped asking "does this look like the element we saw before?" and started asking "does this element still match what the user meant?" Tying this back to the original button above, if the user specifies “the blue button,” we now strictly enforce that the button is blue. ## The result We’ve now maintained a cache hit rate over 95% while limiting false positives, which allows us to execute steps in 300ms on average, while a completely uncached step takes over 5s due to LLM latency. In Feb 2026, our attribute-based invalidation flagged 1M potential flakes across 200M resolutions. The new caching allows Momentic to be significantly more reliably than other tools that rely on hardcoded selectors. As a result of these detections, Momentic is able to use AI to adapt and heal the test. Flaky tests erode trust in the whole platform. Solving them isn't a caching problem, it's an intent problem. That's what we set out to fix with Momentic from the start. _Thanks for reading! Part 2 on the migration from Postgres to Clickhouse is coming soon._ _If this kind of problem interests you: connect with us on _[_LinkedIn_](https://www.linkedin.com/in/hungweiwu/)_, _[_X_](https://x.com/MomenticAI)_, or check out our open opportunities at _[_momentic.ai_](https://momentic.ai/)_._ ## Speed at the Cost of Quality _Quality · Wei-Wei Wu_ URL: https://momentic.ai/blog/speed-at-the-cost-of-quality AI coding tools boost velocity but add debt. Learn why behavioral tests and truth-driven development keep quality high at Cursor speed. That’s the finding of a [CMU study](https://arxiv.org/abs/2511.04427) that tracked 806 GitHub repositories that adopted Cursor, compared with 1,380 matched controls. The velocity is real, briefly. The quality isn’t, consistently. But we all know this isn’t going to stop teams from using AI. So, the question is: how do we keep the velocity and quality high when using tools like Cursor, Claude, and Codex? ## Velocity up, quality down, and a ton of vibe-debt Most AI productivity claims come from surveys (i.e., vibes) or lab experiments. The CMUers used a [difference-in-differences design](https://arxiv.org/abs/2511.04427), matching Cursor-adopting repos against similar non-adopting repos based on six months of pre-adoption data. The closest thing to a controlled experiment you can run at this scale. The highlights: 1. **The velocity spike is huge and temporary**. Lines added jumped 281% in month one, 48% in month two, and back to baseline by month three. This isn't the modest 6-20% sustained improvement that earlier Copilot studies found. It's a burst that fades. 1. **The quality decline is smaller and permanent**. Static analysis warnings up 30%, code complexity up 42%, both still elevated six months later. The velocity came and went. The debt moved in. 1. **The debt actively kills the velocity**. A ~3x increase in code complexity or ~5x increase in static analysis warnings fully cancels the velocity benefit. The debt compounds. 1. **AI-generated code is inherently more complex**. Even after controlling for “more code = more complexity” (trivially true), Cursor adoption produced a 9% baseline complexity increase. The code itself is harder to understand, not just more plentiful. 1. **Sustained tool investment matters**. Repos where developers actively refined their .cursorrules kept velocity gains longer. Repos that just turned Cursor on followed the boom-bust curve. Configuration isn't optional. 1. **Heavy usage makes quality worse, not better**. Robustness checks showed that high-confidence, intensive Cursor usage amplified the quality effects. The signal gets stronger the more you use it. The warning breakdown is worth a look, too. The categories that spiked hardest were: - Naming conventions (+21/repo/month) - Code hygiene (+16) - Code complexity (+15) - Code style (+15) All contextual violations: valid code that doesn't match how the project does things. Meanwhile, API usage and concurrency warnings actually decreased, suggesting LLMs do improve some patterns. ## AI has scaled code overnight, but not verification In traditional development, this mismatch is manageable. Teams write code at human speed and review it at human speed. The two roughly keep pace. But look at the numbers from the CMU study: With AI-assisted development, production velocity jumped nearly 4x in a single month while quality verification infrastructure stayed exactly where it was. This is a scaling gap, and one that compounds. Every month you produce faster than you verify, you accumulate debt that makes next month's verification harder. The GMM models put a number on the tipping point: ~3x increase in complexity or ~5x increase in warnings, and the velocity benefit is gone entirely. You're not just back to where you started. You're behind, with a harder codebase. A natural response to a QA scaling gap is to throw people at it. More reviewers. More manual testing. But the scaling doesn’t work. You are bringing a linear scaling to an exponential fight. The warning distribution explains why this doesn't work. The categories growing fastest–naming conventions, code hygiene, code complexity–are all contextual. They're violations of how this particular project does things, not violations of universal programming rules. A reviewer who doesn't carry deep codebase context will catch the easy stuff (unused imports, obvious type errors) and miss the contextual stuff, which is exactly what's growing fastest. ## **What actually scales** Here's what's interesting about those warning categories. The CMU paper measured structural quality in terms of naming, complexity, and style. It didn't measure behavioral correctness at all. It can't tell you whether the login flow still works, whether the checkout completes, or whether the API returns the right data after 281% more code landed in a month. The structural stuff is the _easy_ layer to catch. The behavioral drift underneath is what actually breaks things for users. This is why we talk about [truth-driven development](https://momentic.ai/blog/truth-driven-development). When code changes faster than anyone can inspect it, you need a stable definition of correctness that doesn't depend on reading every diff. Behavioral tests that define what must be true: - “A user can log in.” - “Billing updates reflect in the dashboard.” - “A logged-out user can't reach /billing.” Not line coverage (you can have 90% and still ship broken flows). Not code review (you can't review at 3x the usual rate). The implementation can change constantly. The truth stays fixed. That's what scales with AI velocity, because it doesn't care who or what wrote the code. It cares whether the system does the right thing. The question has shifted from "can we build it?" to "can we trust it?" AI answered the first question. Nobody has answered the second one at the velocity AI now demands. This is a gap in this research. The paper recommends scaling test coverage with lines of code added, which is correct. But it frames testing as something that happens after code lands, a verification step at the end of the pipeline. When code lands at 281% of the usual rate, downstream verification will always be behind. Testing has to move from a phase you pass through to infrastructure that runs continuously, at the same cadence as code generation. In practice, that looks like: - Every PR generated by Cursor or Claude gets tested against behavioral truths before it merges. Not “does this function return the right type,” but “does the user journey still work end to end.” - Tests that[ define the specification](https://momentic.ai/blog/the-test-is-the-truth), not just verify it. - Tests that self-heal when the DOM shifts, so your team isn't spending 40% of their sprint maintaining brittle selectors while the AI generates another thousand lines. The CMU paper found that repos actively refining their .cursorrules kept velocity gains longer. The same logic applies to testing: teams that invest in their testing infrastructure, not just their code generation tools, are the ones that sustain the speed. AI should test the code it writes. That's the only way the math works. ## The window The CMU data shows velocity gains concentrate in the first one to two months. After that, debt dynamics take over. The window to get testing infrastructure in place is _before_ you adopt AI coding tools, or, at worst, at the same time. Teams adopting AI coding tools without a plan for how verification scales with production are running the exact experiment this paper measured. The results are in: a transient spike, a permanent quality decline, and a codebase that's harder to work with than the one you started from. The velocity is real. The 281% spike is real. And the debt that eats it is real, too. The teams that keep the speed will be the ones that defined their truths before they turned on the AI. Everyone else is just generating technical debt faster. ## A Migration Guide To Go From Outsourced QA To Engineering-Owned Tests _Quality · Wei-Wei Wu_ URL: https://momentic.ai/blog/outsourced-qa-to-engineering-owned-tests Move from outsourced QA to engineering-owned tests with a practical migration plan for faster feedback, stronger coverage, and better releases. _The promise was so beautiful. A team dedicated to your quality. Turnaround in hours, nay, minutes. 99.999recurring percent coverage. Outsourcing QA meant your team could focus on the product while the agency handled testing._ The reality was so different. Bug reports that arrived two days late, written by someone who'd never used the product. Slack threads longer than the code changes they were about. A coverage number that looked great on a dashboard and meant nothing in production. Outsourcing QA meant your team spent half its time explaining context to people who would forget it by the next sprint. Outsourced QA doesn’t work. At this point, we’ve seen dozens of customers move from outsourced, agency QA to in-house testing, allowing them to own quality, ship with confidence, and move faster than they ever did with a vendor in the loop. The hard part isn't making that call. The hard part is making the transition without shipping bugs during the gap. Here's how to make that transition without losing coverage along the way. 1. **Think in behaviors, not test scripts**. Reset your framing from implementation-level scripts to behavioral statements about what your product must do. 1. **Audit what you have**. Figure out what your vendor actually covers, because it's almost certainly less than the dashboard says. 1. **Define what “engineering-owned” means for your team**. Align on the ownership model that fits your team size and structure. 1. **Run the new system in parallel before you cut the old one**. Build confidence in the new system while the old one is still catching things. 1. **Migrate coverage, not tests**. Write new behavioral tests from scratch instead of porting the vendor's scripts. 1. **Restructure your feedback loops**. Compress the cycle from 72-hour bug reports to minutes-long CI checks. 1. **Measure the transition**. Track DORA and operational metrics to prove the migration is working. ## 1. Think in behaviors, not test scripts Before you start migrating anything, reset how you think about what you're migrating. Outsourced QA trains you to think in scripts. Your vendor delivered a spreadsheet of test cases: click this button, type into this field, assert this element appears. That framing is implementation-level. It describes how to test, not what to test. When you start the migration, the instinct will be to recreate those scripts in a new tool. **Resist**. You're not porting a test suite, but defining what must be true about your product. Start from what matters. Vendors cluster coverage around flows that are simple to script: login, password reset, basic CRUD, and static pages. Your test strategy should start from the other direction: - What would cause a customer to churn? - What would cause an incident at 3am? - What would cause a deal to fall through during a live demo? Behaviors come first. A test that verifies “a user can complete checkout with a promo code applied to a bundled subscription” matters more than ten tests that verify form validation on a settings page. Thus, tests are behavioral statements. [A well-written test describes the outcome your product must produce](https://momentic.ai/blog/the-test-is-the-truth): “a user can log in,” “a workspace can be created,” “billing updates reflect in the dashboard.” When your UI changes, a behavioral test still holds. A script breaks. Treat tests like product specifications. They are! They are a part of the specification of your product, just after the fact. When your team owns the tests, they stop being a checklist someone runs after development and become the contract that defines correctness. [Writing them deserves the same strategic attention as writing a PRD.](https://momentic.ai/blog/value-of-quality) The migration is your opportunity to build that foundation correctly, not to carry forward the vendor's assumptions about what was worth testing. ## 2. Audit what you have Before you cancel the vendor contract, figure out what your current setup actually covers. It's almost certainly less than the dashboard says. - **Map coverage to critical flows**. Get a list of all the tests your vendor runs. Map each one to a real user flow in your product. You'll typically find that coverage clusters around easy-to-test flows (login, basic CRUD, static pages) and undercovers the flows that matter most: checkout, billing, permissions, integrations, and anything with conditional logic. The dashboard might say 80% coverage. The reality is that 40% of flows would actually cause a production incident if they broke. - **Evaluate test health**. Pull pass/fail rates over the last 90 days. If more than 10% of tests are flaky, failing intermittently for reasons unrelated to real bugs, those tests are noise. They consume triage time without catching anything. Don't plan to migrate flaky tests. Plan to replace them. - **Check test freshness**. When was each test last updated? If your product shipped a major feature three months ago and the test suite hasn't changed, the vendor isn't keeping up. Stale tests give false confidence, which is worse than no tests at all, because your team assumes coverage exists where it doesn't. - **Identify what's not tested**. This is the most important finding. Ask your engineers one question: "What breaks in production that we never catch before it ships?" The answers tell you where your real coverage gaps are and where to focus first when you stand up the new system. The goal of this audit is not to replicate the existing test suite. It's to understand which flows matter, which are actually covered, and which are theater. ## 3. Define what “engineering-owned” means for your team “Engineering-owned tests” can mean different things depending on team size and structure. Before you start migrating, align on the model you're targeting. ### Developer-written, developer-maintained Every engineer writes and owns tests for the features they ship. This works best for teams under 15 where everyone has full-stack context and can reason about the product end-to-end. The prerequisite is tooling. If writing a test requires learning Playwright selectors, debugging flaky waits, and configuring browser environments, developers won't do it. The testing tool has to be low-friction enough that writing a test feels like writing a PR description rather than a side project. Natural language test authoring makes this viable in a way that code-based frameworks never did: an engineer who just built a billing feature can write "verify that upgrading from free to pro updates the invoice immediately" and have a working test in minutes. The risk is coverage gaps. No single engineer sees the whole product, so cross-feature interactions fall through the cracks. Mitigate this with: - A shared test health dashboard visible to the whole team - A recurring (brief, async) review of coverage gaps by a tech lead - Clear ownership of integration-level tests that span multiple features This model works even for non-engineering teams when the tooling is right. At [Quora](https://momentic.ai/customers/quora), the Product Operations team, not engineers, automated 500+ manual test cases for Poe.com using natural language tests, cutting daily test execution from 7 hours to 30 minutes. ### Embedded QA engineer(s) One or two QA-focused engineers own test strategy, write the complex tests, and review the overall coverage map. Developers write basic smoke tests for their own features. This works well at 15 to 50 engineers, where the product surface area is too large for any single developer to hold in their head. The key difference from outsourcing: these QA engineers sit in your standups, absorb product context, and accumulate institutional knowledge about failure modes. After six months, an embedded QA engineer knows that: - The discount code logic breaks when combined with annual billing - The search index takes 30 seconds to update after a bulk import - The mobile web checkout has a race condition on slow connections An outsourced vendor never builds this knowledge. The role itself is evolving. Embedded QA engineers increasingly spend less time executing tests and more time defining test strategy, writing behavioral specifications, and architecting coverage. Think of the role as a test architect who ensures the right things are tested, while the tooling handles execution and maintenance. ### Platform team with product team ownership At scale (50+ engineers), a small platform or quality team manages the testing infrastructure: CI/CD integration, test environments, parallel execution, test health metrics, and tooling standards. Product teams own coverage for their own features. The platform team doesn't write product tests. They make it easy for everyone else to. This model works because it separates two distinct problems: | Product teams | Platform team Strength | Domain knowledge of what needs testing | Infrastructure expertise to make testing fast and reliable Owns | Test definitions, coverage for their features | CI/CD integration, execution environments, and tooling standards Success metric | Critical flows are covered, and tests are meaningful | Product teams write and run tests without filing infra tickets When these responsibilities are merged into one team, both suffer. Product coverage stalls because the infra work is urgent, or infrastructure degrades because product tests are the priority. None of these models involves handing tests to an external team. All of them keep quality decisions inside the org. And all three are moving in the same direction: engineers owning quality directly, with dedicated QA roles evolving from manual test execution toward test strategy, coverage architecture, and behavioral specification. The traditional QA engineer who clicks through flows and writes bug reports is [being replaced by the test architect](https://momentic.ai/blog/killing-qa) who defines what must be true about the product and ensures the system verifies it continuously. Be explicit about which model you're targeting today, but design for where your team is heading. It changes how you staff the transition, how you onboard the tool, and what success looks like in three months. ## 4. Run the new system in parallel before you cut the old one This is where most migrations go wrong. Teams cancel the QA vendor, then scramble to build coverage from scratch. There's a gap. Bugs ship. Confidence drops. Engineers lose trust in the new approach before it has a chance to prove itself. Instead, run both systems simultaneously for 6 to 8 weeks. Yes, it costs. But it will allow you to understand your new workflow while still leaning on theirs. - **Weeks 1–2: Cover your top 10 critical flows**. Pick the 10 user flows that would cause the most damage if they broke in production. Signup, checkout, core workflow, billing, permissions, the flows where a bug means revenue loss or customer churn. Write tests for these first. Run them in CI on every PR. This is your minimum viable test suite. - **Weeks 3–6: Expand to 30–50 flows while the vendor still runs.** You're building confidence in the new system while the old one still catches things. Track what each system catches during this window. You'll likely find two things: the new system catches issues faster (because it runs in CI rather than on a 24-hour cycle), and the vendor catches fewer unique issues than you expected. Most of what the vendor reports will already be flagged by the new system, often hours earlier. **Weeks 7–8: Compare results and cut over.** By now, you have data. What the new system catches, what the vendor catches, and where the overlap is. If the new system covers the vendor's critical findings and catches them faster, you can confidently end the contract. If there are gaps, you know exactly where they are and can close them before pulling the trigger. [Retool](https://momentic.ai/customers/retool) followed a similar pattern. Their engineering team ran a 15-page manual QA checklist before every release, alongside a Cypress suite that couldn't handle the complexity of their product. After standing up Momentic tests in parallel, they retired the checklist entirely and went from biweekly releases to four times a week. The parallel period costs money, but it costs far less than a production incident caused by a coverage gap during the transition (and that makes you scramble back to the vendor). ## 5. Migrate coverage, not tests Don't try to recreate your vendor's test suite line by line. Their tests were written for their workflow, their tooling, and their understanding of your product, which was probably shallow. Instead, migrate by flow: 1. List every critical user flow in priority order (from the audit). 1. For each flow, write a new test from scratch in the new tool. 1. Validate that the new test catches the same class of issues as the old test. 1. Move to the next flow. This is faster than translating old tests and produces better results. A test written by someone who understands your product, describing behaviors rather than clicking through scripts, will be more reliable than a ported test carrying assumptions and debt from the old system. This is the moment to apply the framing from your test strategy: write each new test as a [behavioral statement about what your product must do](https://momentic.ai/blog/test-driven-development), not as a reimplementation of the vendor's procedural script. For teams with large existing Playwright or Selenium suites, whether homegrown or vendor-managed, the same principle applies. Don't try to make old scripts work in a new context. The maintenance burden you've accumulated in those scripts is the problem you're solving. Porting the scripts ports the burden with them. Start fresh. Write tests that reflect how your product works today, not how it worked when someone wrote the original test six months ago. ## 6. Restructure your feedback loops With outsourced QA, the feedback loop looked like this: build the feature, hand it to the vendor, wait for their cycle, read the bug report, context switch back to the code, try to reproduce, fix it. That loop takes 24 to 72 hours on a good week. With engineering-owned tests, the loop compresses to: build the feature, run tests automatically on the PR, see results in minutes, and fix before merge. Making that compression real requires a few concrete changes: - **Tests run on every PR**. This is non-negotiable. If tests only run nightly or on a schedule, you've recreated the outsourced model with different tooling. The value is in immediate feedback on the code that's about to merge. - **Results show up where engineers already work**. GitHub check, Slack notification, and IDE integration. Not a separate dashboard that nobody checks until deploy day. The information should be impossible to miss without adding a new tab to anyone's workflow. - **Critical failures block merge**. For your top 10-20 flows, a test failure should prevent the PR from being merged. This is the quality gate that replaces the vendor's bug report. It's faster, more reliable, and doesn't require a meeting to interpret. - **Non-critical failures are visible but non-blocking**. For lower-priority flows, surface failures as warnings. Engineers can assess whether the failure is relevant to their change or a pre-existing issue. - **Cancel the QA sync meeting**. If you had a recurring meeting to coordinate with the vendor (what's in scope, what's blocking, what's the status of that bug report from Tuesday), cancel it. The information the meeting provided now comes directly from the CI pipeline. One thing teams notice after making this transition: engineers start testing earlier in the development process, not because of a mandate, but because the feedback is fast enough to be useful while they're still working on the feature. When writing and running a test takes minutes instead of days, testing stops being a phase at the end and becomes something that happens continuously. Quality moves earlier in the development cycle as a [natural consequence of the tighter loop](https://momentic.ai/blog/outsourcing-quality-is-a-velocity-trap). ## 7. Measure the transition Deployment frequency and change failure rate are [DORA metrics](https://dora.dev/guides/dora-metrics/) that are well benchmarked and directly affected by a QA migration. The others are operational metrics specific to the transition. You won't find industry benchmarks for them, but you should baseline them before the migration starts so you can track the delta. **DORA metrics:** Metric | Before (outsourced) | Target (engineering-owned) Deploy frequency | Weekly or biweekly | Daily or per-PR Change failure rate (production incidents per release) | Baseline before migration | Track reduction over 3–6 months **Operational metrics to baseline:** Metric | Before (outsourced) | Target (engineering-owned) Time from bug introduced to detection | Typically 24–72 hours | <1 hour (caught in CI) Time from detection to when the engineer sees it | Typically 2–8 hours (report, triage, Slack thread) | < 5 minutes (PR check) Engineering hours/week on QA coordination | Varies, but often 10–25 hours across the team | 0 (no sync meetings, no clarification threads) Test maintenance hours/week | Vendor manages (you pay via coordination overhead) | Near-zero with self-healing tools; 1–2 hrs/week without Don't expect all of these to improve in week one. The operational metrics, time to detection, and time to engineer, improve immediately. The DORA metrics improve over one to three months as coverage builds and engineers start trusting the new system enough to ship more frequently. If you're not seeing improvement in the operational metrics within the first two weeks, something is wrong with the setup, not the approach. Check that tests are actually running on every PR, that results are visible in the right places, and that the critical-flow tests are stable enough to be meaningful signals. ## The transition is the easy part The mechanical steps of moving from outsourced QA to engineering-owned tests are straightforward: audit, build in parallel, cut over. The harder shift is cultural. This migration is the moment leadership signals that quality is a first-class engineering value, not a line item on a vendor invoice. Your engineers need to believe that quality is their responsibility, that the tests are trustworthy, and that the tooling won't become another maintenance burden they resent. The audit builds conviction that the old system wasn't doing what everyone assumed. The parallel period builds evidence that the new system works. The faster the feedback loop, the stronger the habit. And the behavioral test suite you build along the way becomes something the vendor never gave you: a durable, team-owned specification for how your product is supposed to work. Give each of those stages enough time to do its job, and the rest follows. ## Agentic Testing for Engineering Teams: A Definitive Guide (Part 2 of 2) _Quality · Wei-Wei Wu_ URL: https://momentic.ai/blog/how-agentic-testing-works This is the second part of our guide on agentic testing, covering how agentic testing works in practice and best practices for implementing inside your organization. _This is the second part of our guide on agentic testing, covering how agentic testing works in practice and best practices for implementing inside your organization. You can check out the first part to understand what agentic testing is: _[_https://momentic.ai/blog/agentic-testing-guide_](https://momentic.ai/blog/agentic-testing-guide) ## How does agentic testing work in practice? Underneath the four-verb definition, the technical architecture is converging fast. A few patterns are now well-established across vendors and open-source projects. ### Multi-modal page reading The dominant pattern is accessibility-tree-first with a vision fallback. The a11y tree is the semantic representation of the page (the same one screen readers use), which makes it more stable across CSS and DOM refactors than raw selectors. The agent reads the page by capturing a snapshot, a serialized text version of the tree at that moment. It looks something like this: The LLM reads that snapshot, decides "I want to type in the textbox," and acts against ref=e5. Those reference IDs are local to this single snapshot: the next time the agent observes the page, it gets a fresh tree with fresh refs. Selector drift, the source of most flakiness in scripted suites, can't happen because there's no persistent selector to drift. For surfaces the a11y tree can't describe well (HTML canvas, custom-rendered components, charting libraries), the agent falls back to a multimodal vision model interpreting the screenshot directly. ### Intent as the durable contract Rather than page.click('#submit-btn-v2'), the test author specifies the intent: click the submit button. The agent's resolver, an LLM with multi-modal access to page state, picks the actual element at runtime. Intent is what the test author writes. Selectors are implementation details the agent works out for itself. This matters because intent survives refactors. “Click the submit button” stays valid when the button changes from a