Skip to main content

​
Speeding up large test suites

By default the CLI runs tests one at a time on a single machine, so a big test suite runs as slowly as the sum of its tests. Two independent knobs cut wall-clock time:
  • Parallelism (one machine): --parallel <n> runs n tests at once, each in its own browser instance. Raise it until the runner’s CPU/memory is saturated. 
    npx momentic run --parallel 4
  • Sharding (many machines): split the test suite across runners with --shard-count and a per-runner --shard-index, then combine the outputs with momentic results merge before uploading. This is how you turn a multi-hour sequential run into minutes across a CI matrix. 
    # runner 1 of 4
npx momentic run --shard-index 1 --shard-count 4 --output-dir test-results/shard-1
Combine both: shard across runners, and use --parallel within each runner.
Sharding and parallelism multiply: 4 shards x --parallel 4 is up to 16 tests running at once. See the GitHub Actions guide for a ready-to-copy matrix + merge workflow, and run / results merge for every flag.
Also make sure step caching is saving in CI (--save-cache): cached steps run in well under a second, so a warm test suite is much faster than a cold one.

​
Mobile test suites

Mobile runs use the same --parallel / --shard-count / --shard-index flags through the momentic-mobile CLI, with a few platform caveats:
  • --parallel AUTO saturates the current shard. On remote emulators each test gets its own session, so parallelism is safe and any org quota is enforced server-side.
  • Local Android runs need a distinct AVD per parallel test: running multiple tests against the same --local-avd-id conflicts.
  • Local iOS runs prefer --parallel 1 because concurrent Appium instances share a driver manifest.
# remote emulators, sharded across runners and parallel within each
npx momentic-mobile run --shard-index 1 --shard-count 4 --parallel AUTO
Merge shards with momentic-mobile results merge before uploading, exactly like web. See momentic-mobile run for every flag.

​
Execution speed

Runtime depends on how a test is written and the state of the system under test. A Momentic test that only aims for feature parity with Playwright or Cypress runs at about the same speed. Thanks to step caching, over 99% of steps execute in under 500ms:
Preset actionAverage runtime
Click250ms
Type340ms
Choosing from a <select> element275ms
Pressing a key<5ms
Scroll<5ms
Page check attempt220ms
Element check attempt210ms
Visual diff620ms
AI-enhanced steps are slower on first run but most are cached for subsequent runs. Approximate first-time runtimes:
AI-enhanced actionFirst-time runtime range
Locating an element4-8 seconds
Evaluating an assertion once5-8 seconds
Extracting data from the page5-8 seconds
Generating a single command in an AI action6-12 seconds
Classifying a test failure20-30 seconds
Auto-healing a section30+ seconds

​
Benchmarks

​
Overview

We have published a basic benchmark comparing Momentic against Playwright in this publicly accessible test automation environment. The results illustrate that cached Momentic steps are only 52ms slower on average than comparable Playwright functions. Non-cached steps that require AI to execute run on average 6354ms slower. Over 99% of all steps executed on the Momentic platform are cached. Note that this benchmark does not exhaustively test all Momentic step types, many of which do not have analogs in Playwright, Cypress, or any traditional tooling (e.g. AI check, Visual diff).

​
Method

We built a Momentic test that logs into the practice automation site, as well as an equivalent Playwright script that performs the same sequence of actions. We obtained three different sets of measurements:
  • The “Steps only” category only measures the time spent executing steps in both software.
  • The “End-to-end” category includes Momentic’s fixed bootstrap (e.g. API key check) and test result upload times. For Playwright, the end-to-end time includes CLI initialization time but does not involve any upload of data.
  • The “First-run” category ran with caching explicitly disabled and thus includes the runtime of 4 fresh AI completions.
All measurements were completed on a M3 Max Macbook Pro with 36GB RAM running macOS Sonoma.

​
Results

All values are P50 milliseconds measured over 10 independent samples.
PlaywrightMomentic
Steps only961ms1173ms
End-to-end1870ms3998ms
First-run steps onlyN/A26379ms
The source used for this benchmark is provided below: 
fileType: momentic/test/v2
id: log-in-practice-test-automation
url: https://practicetestautomation.com/practice-test-login/
retries: 1
steps:
  - type:
      text: student
      into: the username input
  - type:
      text: Password123
      into: the password input
  - click: the submit button
  - checkPageContains: Logged In