Failure recovery

Momentic can automatically recover from certain test failures by proposing and executing step changes. If the auto-healing is successful, you will have an opportunity to review and approve the changes in the results viewer. If the auto-healing is not successful, the test run will fail and you will be notified normally.

​

Enabling

Failure recovery can be enabled on a per-test basis in the test options dialog. For a test to be eligible for failure recovery, it must also have passed at least once in the last 7 days. This restriction ensures that deterministically failing tests do not go through failure recovery, which can incur up to a 60 second delay.

​

Recovery behavior

When an eligible test fails, Momentic’s AI will generate one or more step changes aimed to allow the rest of the test to continue while preserving the test’s original intent.

Momentic will then automatically execute the proposed changes using the current state of the browser. If successful, Momentic will continue running the rest of the test normally. If not, the test will be marked as failed. Failure recovery operates independently of any retries that may be configured on the test.

A single test run is eligible for failure recovery only once. If a later step in the test also fails, Momentic will not attempt to recover using AI to avoid compounding errors.

​

Approving changes

​

Reviewing changes

If a run is recovered successfully, the results viewer will display any steps added or modified by our AI with a unique purple status. The results viewer displays the exact sequence of actions taken over the course of the test, so any steps that were re-attempted after the failure recovery will appear multiple times in the results viewer:

The results viewer will also include a purple “AI changes for review” button that opens a detailed review dialog when clicked. This dialog will summarize the actions taken by the AI and display the exact step diffs in a YAML format:

The diff viewer for tests running on Momentic Cloud will compare the proposed steps against the live state of the test. For tests running locally, the diff viewer will compare the proposed steps against the state of the test at the time of the run. Be careful not to overwrite any changes that were made to the test after the test run.

​

Applying changes

For tests that exist on Momentic Cloud, you can apply the proposed changes by simply clicking the “Apply changes” button at the bottom of the dialog. This will instantly update the test. Please note that if any changes were made to the test after the test run, those changes may be overwritten. Please review the diffs carefully before approving.

For tests that exist locally, the failure recovery dialog will contain a separate tab labelled ‘Apply changes’. This tab will contain a CLI command that you can run on your machine to apply the proposed changes to your local test files:

This command must be executed in the project that contains the recovered test. When ran, the CLI will display a final diff against the local state of your test before prompting for approval:

​

Limitations

​

Excluded scenarios

As a rule of thumb, Momentic’s AI will decline to recover any of the following scenarios:

• Infrastructure issues (e.g. network timeouts, server errors, full-page errors, 500 response codes): there is usually no way to recover from these by adding or modifying existing test steps. We recommend you configure retries on your test to handle these cases.
• Assertion failures: we treat assertions as user-defined “ground truth” as to how the application should behave, so our AI will never modify existing assertions. However, we may modify test steps before assertion to produce the correct application state.
• Configuration errors: our AI will not salvage test steps that are missing inputs (e.g. type values), contain nonsensical values (e.g. negative timeout values), or are otherwise invalid (e.g. non-existent URL).

​

Patch content

To avoid affecting multiple tests at the same time, failure recovery will not suggest patches to any step containers such as modules and AI actions.

At the moment, we have also constrained our AI agents to only suggest patches to the test step that failed or the steps immediately preceding the failed step. As we gather more data, we plan to expand the scope of allowed patches to include steps further way from the failure point.