Performance
Momentic’s performance characteristics and benchmarks
Execution speed
The runtime of a Momentic test largely depends on how it is crafted and the state of the system under test.
A Momentic test that only attempts to achieve feature parity with a traditional testing tool such as Playwright or Cypress will run at approximately the same speed. Due to intelligent caching, over 99% of steps that run on the Momentic platform execute in under 500ms:
| Preset action | Average runtime |
|---|---|
| Click | 250ms |
| Type | 340ms |
Choosing from a <select> element | 275ms |
| Pressing a key | <5ms |
| Scroll | <5ms |
| Page check attempt | 220ms |
| Element check attempt | 210ms |
| Visual diff | 620ms |
However, any enhanced steps that require AI completions can incur greater runtime costs. These steps provide outsized value, power, and flexibility to developers at the cost of duration. The approximate first-time execution speed of each AI-enhanced action supported by Momentic is listed below. In many cases, Momentic caches AI results to reduce subsequent runtime.
| AI-enhanced action | First-time runtime range |
|---|---|
| Locating an element | 4-8 seconds |
| Evaluating an assertion once | 5-8 seconds |
| Extracting data from the page | 5-8 seconds |
| Generating a single command in an AI action | 6-12 seconds |
| Classifying a test failure | 20-30 seconds |
| Auto-healing a section | 30+ seconds |
Benchmarks
Overview
We have published a basic benchmark comparing Momentic against Playwright in this publicly accessible test automation environment.
The results illustrate that cached Momentic steps are only 52ms slower on average than comparable Playwright functions. Non-cached steps that require AI to execute run on average 6354ms slower. Over 99% of all steps executed on the Momentic platform are cached.
Note that this benchmark does not exhaustively test all Momentic step types, many of which do not have analogs in Playwright, Cypress, or any traditional tooling (e.g. AI check, Visual diff).
Method
We built a Momentic test that logs into the practice automation site, as well as an equivalent Playwright script that performs the same sequence of actions. We obtained three different sets of measurements:
- The “Steps only” category only measures the time spent executing steps in both software.
- The “End-to-end” category includes Momentic’s fixed bootstrap (e.g. API key check) and test result upload times. For Playwright, the end-to-end time includes CLI initialization time but does not involve any upload of data.
- The “First-run” category ran with caching explicitly disabled and thus includes the runtime of 4 fresh AI completions.
All measurements were completed on a M3 Max Macbook Pro with 36GB RAM running Mac OS Sonoma.
Results
All values are P50 milliseconds measured over 10 independent samples.
| Playwright | Momentic | |
|---|---|---|
| Steps only | 961ms | 1173ms |
| End-to-end | 1870ms | 3998ms |
| First-run steps only | N/A | 26379ms |
The source used for this benchmark is provided below: