---
title: "Gating merges and deploys on end-to-end tests: required checks, merge queues and flaky-test policy"
description: "Decide which end-to-end tests block a pull request, a merge queue and a deploy, with working GitHub Actions, GitLab, CircleCI, Buildkite and Jenkins config."
canonical: "https://momentic.ai/blog/ci-e2e-test-gating"
last-updated: "2026-11-02T18:21:27Z"
---

# Gating merges and deploys on end-to-end tests: required checks, merge queues and flaky-test policy

URL: https://momentic.ai/blog/ci-e2e-test-gating

[blog](/blog) [/ resources](/blog/category/resources)  / ci-e2e-test-gating

Resources

Which end-to-end tests should block a merge, a merge queue and a deploy, how to make each gate required in five CI systems, and how retries and quarantine change what a failure blocks.

Wei-Wei Wu

CEO, Momentic

An end-to-end test blocks a merge only when two things are true: the CI job exits nonzero when a counted test fails, and the repository requires that job's check before merging. Everything else is policy. Decide which suite gates which event, how long each gate may take, and how a flaky test counts, then write the config.

This guide covers that policy layer. For the basics of running a suite in CI, sharding it and sending results to Slack, start with [CI/CD pipeline testing](/blog/cicd-pipeline-testing).

## The gate matrix: what runs where and what it blocks

A merge gate runs a small suite against the code that is about to merge. Broader regression coverage runs off the merge path, because its runtime and its failure triage would hold up unrelated pull requests.

| Event | Suite | Example time budget | What a failure blocks |
| --- | --- | --- | --- |
| Pull request | critical | 10 minutes | The merge, through a required check |
| Merge queue (merge_group, GitLab merge train) | critical | 10 minutes | The queued merge; the PR leaves the queue |
| Push to main and nightly | regression, sharded | 30 minutes | Nothing merged. Alerts the owning team, can hold release promotion |
| After a production deploy | smoke | 5 minutes | No PR. Pages on-call or triggers a rollback |

The budgets are starting points to tune against your own run times. Momentic's [gate guide](/docs/guides/use-cases/gate-pull-requests-on-critical-flows) recommends keeping the `critical` label under roughly ten tests: sign-in, checkout, the primary write action. If a failure in a flow would not make you roll back, it belongs in the nightly suite.

The labels live in the test files, which sit in the repository next to the code:

`fileType: momentic/test/v2 id: checkout-critical url: https://staging.shop.example.com labels: [critical, regression] before:  - module: ../modules/log-in.module.yaml steps:  - click: Add to cart  - click: Checkout  - click: Place order  - assert: An order confirmation is visible`

A test can carry several labels. Each gate selects its slice with `npx momentic run --labels critical`, `--labels regression` or `--labels smoke`.

To shorten a slow pull request gate without shrinking coverage, [AI test selection](/docs/ai/select) (`--ai-select --ai-select-base origin/main`) runs the tests that cover the diff. When selection is unavailable, Momentic runs every test. Keep the full nightly run so tests that selection skipped still execute.

## GitHub Actions: pull_request, merge_group and one required check

A merge queue tests the merge commit it is about to create, not the PR's head commit. The workflow has to trigger on `merge_group` as well as `pull_request`. If it listens only for pull requests, the required check never reports for the queue's commit and the queue waits until it times out.

The workflow below shards the critical suite across two jobs and makes a single aggregation job the required check:

.github/workflows/e2e-gate.yml`name: E2E gate  on:  pull_request:  merge_group:  permissions:  contents: read  jobs:  critical:  name: Critical shard ${{ matrix.shard }}  runs-on: ubuntu-latest  timeout-minutes: 15  strategy:  fail-fast: false  matrix:  shard: [1, 2]  env:  MOMENTIC_API_KEY: ${{ secrets.MOMENTIC_API_KEY }}  steps:  - uses: actions/checkout@v4  - uses: actions/setup-node@v4  with:  node-version: 22.12.0  cache: npm  - run: npm ci  - run: npx momentic install-browsers chromium  - name: Run critical flows  run: >-  npx momentic run  --labels critical  --shard-index ${{ matrix.shard }}  --shard-count 2  --reporter steps  -y  --upload-results   gate:  name: Merge gate  runs-on: ubuntu-latest  needs: critical  if: ${{ always() }}  steps:  - name: Require every shard to pass  run: test "${{ needs.critical.result }}" = success`

Require `E2E gate / Merge gate` in the branch ruleset (or classic branch protection), not the individual shards. Two reasons:

- Matrix job names change when you change the shard count, and a required check that no longer exists blocks every PR.
- GitHub treats a skipped job as passing for required checks. If the gate job used `needs` without `if: always()`, a failed shard would skip the gate and the PR would merge. The `test` step turns any shard result other than `success` into a failure.

If you enable the merge queue in the same ruleset, the same check name has to pass on the queue's commit. Because the workflow runs on both events, it does.

## GitLab: merge request pipelines, merge trains and "Pipelines must succeed"

A merge request pipeline runs the tests, but on its own it does not block the merge. GitLab enforces the result only with the **Pipelines must succeed** setting, under **Settings > Merge requests > Merge checks**.

.gitlab-ci.yml`workflow:  rules:  - if: $CI_PIPELINE_SOURCE == "merge_request_event"  - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH  image: node:22.12  stages:  - test  - report  e2e_critical:  stage: test  parallel: 2  rules:  - if: $CI_PIPELINE_SOURCE == "merge_request_event"  script:  - npm ci  - npx momentic install-browsers chromium  - >-  npx momentic run  --labels critical  --url-override "$REVIEW_APP_URL"  --shard-index "$CI_NODE_INDEX"  --shard-count "$CI_NODE_TOTAL"  --output-dir "test-results/shard-$CI_NODE_INDEX"  artifacts:  when: always  paths:  - test-results  expire_in: 12 hours  e2e_regression:  stage: test  rules:  - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH  script:  - npm ci  - npx momentic install-browsers chromium  - npx momentic run --labels regression --upload-results  upload_results:  stage: report  needs: [e2e_critical]  rules:  - if: $CI_PIPELINE_SOURCE == "merge_request_event"  when: always  script:  - npm ci  - npx momentic results merge --output-dir test-results/merged test-results  - npx momentic results upload test-results/merged`

Store `MOMENTIC_API_KEY` as a masked CI/CD variable. `REVIEW_APP_URL` is whatever URL your review app or preview deploy exposes. `CI_NODE_INDEX` starts at 1, so it maps directly to Momentic's `--shard-index`. The `upload_results` job runs even when a shard fails, so the dashboard gets one run group for the pipeline, and its own success does not hide the failed shard: the pipeline still fails.

Merge trains run merge request pipelines too (`CI_MERGE_REQUEST_EVENT_TYPE` is `merge_train`), so the same `e2e_critical` job gates the train's combined result. Do not set `allow_failure: true` on a gating job. GitLab counts an allowed failure as a passing pipeline for **Pipelines must succeed**.

## CircleCI: one workflow, 0-indexed parallelism

CircleCI reports each job to GitHub as a status check named `ci/circleci: <job name>`. Require that name in the ruleset. Run the critical job on every branch except `main`, and the regression job on `main`:

.circleci/config.yml`version: 2.1  orbs:  node: circleci/node@5.0.2  jobs:  e2e:  parameters:  label:  type: string  shards:  type: integer  docker:  - image: cimg/node:22.12-browsers  parallelism: << parameters.shards >>  steps:  - checkout  - node/install-packages  - run: npx momentic install-browsers chromium  - run:  name: Run Momentic tests  command: |  npx momentic run \  --labels << parameters.label >> \  --shard-index "$((CIRCLE_NODE_INDEX + 1))" \  --shard-count "$CIRCLE_NODE_TOTAL" \  --reporter junit \  --reporter-dir reports \  --upload-results  - store_test_results:  path: reports  workflows:  e2e:  jobs:  - e2e:  name: e2e-critical  label: critical  shards: 2  filters:  branches:  ignore: main  - e2e:  name: e2e-regression  label: regression  shards: 4  filters:  branches:  only: main`

`CIRCLE_NODE_INDEX` starts at 0 and Momentic's `--shard-index` starts at 1, so the command adds one. `store_test_results` runs after a failed test step without changing the job's result. CircleCI reports one check for the job across all parallel containers, so `ci/circleci: e2e-critical` fails if any shard fails.

## Buildkite: parallel steps and Test Analytics

Buildkite numbers parallel jobs from 0 in `BUILDKITE_PARALLEL_JOB`. Add one before passing it to Momentic:

.buildkite/pipeline.yml`steps:  - label: ":momentic: Critical %n"  key: critical  parallelism: 2  commands:  - npm ci  - npx momentic install-browsers chromium  - >-  npx momentic run  --labels critical  --shard-index "$((BUILDKITE_PARALLEL_JOB + 1))"  --shard-count "$BUILDKITE_PARALLEL_JOB_COUNT"  --reporter buildkite-json  --reporter-dir reports  --output-dir "test-results/shard-$BUILDKITE_PARALLEL_JOB"  artifact_paths:  - "test-results/**/*"  plugins:  - test-collector#v1.12.0:  files:  - "reports/*.json"  format: "json"   - wait: ~  continue_on_failure: true   - label: ":momentic: Upload results"  commands:  - npm ci  - buildkite-agent artifact download "test-results/**/*" .  - npx momentic results merge --output-dir test-results/merged test-results  - npx momentic results upload test-results/merged`

The `buildkite-json` reporter writes the format [Buildkite Test Analytics](https://buildkite.com/docs/test-analytics) reads, and the test collector plugin uploads it. The `wait` step with `continue_on_failure: true` lets the upload step run after a failed shard, while the build itself still fails. Buildkite posts one commit status per pipeline to GitHub (for example `buildkite/web-e2e`). Require that status in the ruleset.

## Jenkins: keep the exit code, upload in `post`

Jenkins marks a stage failed when its `sh` step exits nonzero. Wrappers break that: `npx momentic run || true` or a pipe into `tee` without `pipefail` reports success after a failed test. Upload results in a `post { always { } }` block so they reach the dashboard without touching the stage result.

Jenkinsfile`pipeline {  agent any   tools {  nodejs 'NodeJS 22'  }   environment {  MOMENTIC_API_KEY = credentials('momentic-api-key')  }   stages {  stage('Install') {  steps {  sh 'npm ci'  sh 'npx momentic install-browsers chromium'  }  }   stage('Critical E2E') {  steps {  sh '''#!/usr/bin/env bash  set -euo pipefail  npx momentic run --labels critical --reporter junit --reporter-dir reports \  | tee momentic.log  '''  }  post {  always {  junit testResults: 'reports/**/*.xml', allowEmptyResults: true  sh 'npx momentic results upload test-results'  }  }  }  } }`

`set -o pipefail` keeps the exit code of `momentic run` when the output goes through `tee`. With the GitHub Branch Source plugin, Jenkins reports the build to GitHub as `continuous-integration/jenkins/pr-merge`. Require that context in the ruleset.

## Flaky tests: retries, quarantine and what counts as a block

Retries and quarantine answer different questions. A retry asks whether a failure repeats. Quarantine says a test's result should not count until someone fixes it.

- `--retries 1` gives a failing test one more attempt. A test that passes on retry returns exit code 0, so the gate passes. Keep the retry budget on the critical suite at 0 or 1, and review tests that needed a retry, because each one is either a flake or an intermittent bug.
- By default, `npx momentic run` still runs quarantined tests, but their results do not affect the exit code. The run reports them so you can watch them.
- `--skip-quarantined` drops quarantined tests from the run. Use it on the merge gate when quarantined tests eat into its time budget.
- `--only-quarantined` runs only quarantined tests and counts their results. Use it in a nightly triage job that tells you which tests are ready to come back.
- `--ignore-quarantine` counts every result. Use it on a release branch when you want every known issue to block.

Add a test with `npx momentic quarantine add <test> --reason <reason>`, or set the **On failed heal** triage setting to quarantine tests the healing agent cannot repair. See [auto-maintenance](/docs/reliability/auto-maintenance) for the rules, and [How to fix flaky tests](/blog/how-to-fix-flaky-tests) for the root causes worth fixing first.

Other runners make the same choice under different names:

| Runner | Retry setting | What a pass on retry does to the exit code | Quarantine equivalent |
| --- | --- | --- | --- |
| Momentic | --retries <n> | Exit 0 | Quarantine with --skip-quarantined, --only-quarantined, --ignore-quarantine |
| Playwright Test | retries in config or --retries | Exit 0, test reported as flaky. --fail-on-flaky-tests makes it exit 1 | test.fixme() or test.skip() in code, or a tag excluded with --grep-invert |
| Cypress | retries.runMode in config | Exit 0 | it.skip() in code, or a test-management service |

Whatever the runner, a required check must not report success without running the tests it gates. Watch for three patterns: `|| true` on the test command, a pipeline whose last command is an upload, and a condition that skips the whole gate job.

## After deploy: smoke tests that alert or roll back

A post-deploy smoke test catches what pre-merge tests cannot see: production config, real third-party integrations, a CDN or DNS change. It runs after the merge, so it cannot block one. Its failure goes to on-call or to your rollback step, never to a PR check. The [smoke testing guide](/blog/smoke-testing-in-software-development) covers what belongs in the suite.

.github/workflows/production-smoke.yml`name: Production smoke  on:  deployment_status:  jobs:  smoke:  if: >-  github.event.deployment_status.state == 'success' &&  github.event.deployment_status.environment == 'production'  runs-on: ubuntu-latest  timeout-minutes: 10  env:  MOMENTIC_API_KEY: ${{ secrets.MOMENTIC_API_KEY }}  TARGET_URL: ${{ github.event.deployment_status.environment_url }}  steps:  - uses: actions/checkout@v4  - uses: actions/setup-node@v4  with:  node-version: 22.12.0  cache: npm  - run: npm ci  - run: npx momentic install-browsers chromium  - name: Run smoke tests  run: >-  npx momentic run  --labels smoke  --url-override "$TARGET_URL"  --retries 1  -y  --upload-results  - name: Alert on-call  if: failure()  env:  SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}  run: |  curl -fsS -X POST -H 'Content-type: application/json' \  --data "{\"text\":\"Production smoke failed for ${GITHUB_SHA::7}: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}\"}" \  "$SLACK_WEBHOOK_URL"`

Smoke tests against production should use dedicated test accounts and avoid irreversible actions such as real payments. Replace the Slack step with your deploy tool's rollback command if it can restore the previous release safely. For preview deployments before merge, see [smoke test preview deploys](/docs/guides/use-cases/smoke-test-preview-deploys).

## Frequently asked questions

How do I block a pull request when an end-to-end test fails?    Make the CI job that runs the suite a required check. In GitHub, add it to a branch ruleset or branch protection. In GitLab, turn on Pipelines must succeed. The job must exit nonzero on a failed test, so never append `|| true` to `npx momentic run`.    Do end-to-end tests work with a GitHub merge queue?    Yes. Trigger the workflow on `merge_group` as well as `pull_request` and require the same check name. Without the `merge_group` trigger, the check never reports for the queue's merge commit and the queue stalls.    Should a flaky test block a merge?    No. Quarantine it so it keeps running without affecting the exit code, give it an owner, and fix it. Keep retries on the merge gate at 0 or 1 so a retry does not hide an intermittent bug.    Should end-to-end tests run against staging or after a production deploy?    Both, with different jobs. The merge gate runs the critical suite against staging or a preview deploy. A smaller smoke suite runs against production after each deploy and alerts on-call or triggers a rollback.    How do I run end-to-end tests in Buildkite?    Add a step that runs `npm ci`, `npx momentic install-browsers chromium` and `npx momentic run --upload-results`. To shard it, set `parallelism` and pass `$((BUILDKITE_PARALLEL_JOB + 1))` as `--shard-index` and `$BUILDKITE_PARALLEL_JOB_COUNT` as `--shard-count`. See the Buildkite setup.    Which end-to-end tests should run on every pull request?    The flows you would roll back for, labeled `critical` and kept under about ten tests. Edge cases and long journeys run nightly with `--labels regression`.

Still have additional questions?

## Keep reading.

[Resources   CI/CD Pipeline Testing: Run, Gate and Alert on End-to-End Tests     Wei-Wei Wu     11 min read](/blog/cicd-pipeline-testing)[Resources   How to Fix Flaky Tests: Root Causes, Fixes, and What to Automate     A flaky test passes and fails on the same code. Here are the four root causes, the fix for each one, and the parts a testing tool can repair for you.     Wei-Wei Wu     8 min read](/blog/how-to-fix-flaky-tests)[Resources   A Complete Guide to Parallel Testing     Your 101 guide to all things parallel testing – what parallel testing is, what the benefits are, and which tools to consider to make it easier.     Wei-Wei Wu     7 min read](/blog/complete-guide-to-parallel-testing)

## Close the feedback loop.

Point Momentic at your app. Free to start, no credit card.

[Try for free](https://app.momentic.ai/signup) [Contact sales](/sales)