By default the CLI runs tests one at a time on a single machine, so a big suite
runs as slowly as the sum of its tests. Two independent knobs cut wall-clock
time:
Parallelism (one machine):--parallel <n> runs n tests at once, each
in its own browser instance. Raise it until the runner’s CPU/memory is
saturated.
npx momentic run --parallel 4
Sharding (many machines): split the suite across runners with
--shard-count and a per-runner --shard-index, then combine the outputs
with momentic results merge before uploading. This is how you turn a
multi-hour sequential run into minutes across a CI matrix.
# runner 1 of 4npx momentic run --shard-index 1 --shard-count 4 --output-dir test-results/shard-1
Combine both: shard across runners, and use --parallel within each runner.
Sharding and parallelism multiply: 4 shards x --parallel 4 is up to 16 tests
running at once. See the GitHub Actions
guide for a ready-to-copy matrix +
merge workflow, and run /
results merge for every flag.
Also make sure step caching is saving in CI
(--save-cache): cached steps
run in well under a second, so a warm suite is dramatically faster than a cold
one.
Mobile runs use the same --parallel / --shard-count / --shard-index flags
through the momentic-mobile CLI, with a few platform caveats:
--parallel AUTO saturates the current shard. On remote emulators each
test gets its own session, so parallelism is safe and any org quota is
enforced server-side.
Local Android runs need a distinct AVD per parallel test: running multiple
tests against the same --local-avd-id conflicts.
Local iOS runs prefer --parallel 1 because concurrent Appium instances
share a driver manifest.
# remote emulators, sharded across runners and parallel within eachnpx momentic-mobile run --shard-index 1 --shard-count 4 --parallel AUTO
Runtime depends on how a test is written and the state of the system under test.
A Momentic test that only aims for feature parity with Playwright or Cypress
runs at about the same speed. Thanks to step caching,
over 99% of steps execute in under 500ms:
Preset action
Average runtime
Click
250ms
Type
340ms
Choosing from a <select> element
275ms
Pressing a key
<5ms
Scroll
<5ms
Page check attempt
220ms
Element check attempt
210ms
Visual diff
620ms
AI-enhanced steps are slower on first run but most are cached for subsequent
runs. Approximate first-time runtimes:
We have published a basic benchmark comparing Momentic against Playwright in
this publicly accessible test automation
environment.The results illustrate that cached Momentic steps are only 52ms slower on
average than comparable Playwright functions. Non-cached steps that require AI
to execute run on average 6354ms slower. Over 99% of all steps executed on
the Momentic platform are cached.Note that this benchmark does not exhaustively test all Momentic step types,
many of which do not have analogs in Playwright, Cypress, or any traditional
tooling (e.g. AI check, Visual diff).
We built a Momentic test that logs into the practice automation site, as well as
an equivalent Playwright script that performs the same sequence of actions. We
obtained three different sets of measurements:
The “Steps only” category only measures the time spent executing steps in both
software.
The “End-to-end” category includes Momentic’s fixed bootstrap (e.g. API key
check) and test result upload times. For Playwright, the end-to-end time
includes CLI initialization time but does not involve any upload of data.
The “First-run” category ran with caching explicitly disabled and thus
includes the runtime of 4 fresh AI completions.
All measurements were completed on a M3 Max Macbook Pro with 36GB RAM running
macOS Sonoma.
