Failure recovery

Momentic can automatically recover from certain test failures by proposing and executing step changes. If the auto-healing is successful, you will have an opportunity to review and approve the changes in the results viewer. If the auto-healing is not successful, the test run will fail and you will be notified normally.

​

Enabling

Failure recovery can be enabled on a per-test basis in the test options dialog. For a test to be eligible for failure recovery, it must also have passed at least once in the last 7 days. This restriction ensures that deterministically failing tests do not go through failure recovery, which can incur up to a 60 second delay.

​

Recovery behavior

When an eligible test fails, Momentic’s AI will generate one or more step changes aimed to allow the rest of the test to continue while preserving the test’s original intent.

Momentic will then automatically execute the proposed changes using the current state of the browser. If successful, Momentic will continue running the rest of the test normally. If not, the test will be marked as failed. Failure recovery operates independently of any retries that may be configured on the test.

A single test run is eligible for failure recovery only once. If a later step in the test also fails, Momentic will not attempt to recover using AI to avoid compounding errors.

​

Approving changes

You will then be able to review and approve any changes made by the failure recovery process.

​

Limitations

​

Excluded scenarios

As a rule of thumb, Momentic’s AI will decline to recover any of the following scenarios:

• Infrastructure issues (e.g. network timeouts, server errors, full-page errors, 500 response codes): there is usually no way to recover from these by adding or modifying existing test steps. We recommend you configure retries on your test to handle these cases.
• Assertion failures: we treat assertions as user-defined “ground truth” as to how the application should behave, so our AI will never modify existing assertions. However, we may modify test steps before assertion to produce the correct application state.
• Configuration errors: our AI will not salvage test steps that are missing inputs (e.g. type values), contain nonsensical values (e.g. negative timeout values), or are otherwise invalid (e.g. non-existent URL).

​

Patch content

To avoid affecting multiple tests at the same time, failure recovery will not suggest patches to any step containers such as modules and AI actions.

At the moment, we have also constrained our AI agents to only suggest patches to the test step that failed or the steps immediately preceding the failed step. As we gather more data, we plan to expand the scope of allowed patches to include steps further way from the failure point.