---
title: "CI/CD Pipeline Testing: Run, Gate and Alert on End-to-End Tests"
description: "Run end-to-end tests in GitHub Actions, CircleCI, Jenkins or Buildkite; gate merges on failures, shard across parallel jobs, and send Slack alerts."
canonical: "https://momentic.ai/blog/cicd-pipeline-testing"
last-updated: "2026-10-03T08:33:08Z"
---

# CI/CD Pipeline Testing: Run, Gate and Alert on End-to-End Tests

URL: https://momentic.ai/blog/cicd-pipeline-testing

[blog](/blog) [/ resources](/blog)  / cicd-pipeline-testing

Resources

Wei-Wei Wu

CEO, Momentic

## TL;DR

- Run the end-to-end suite as a CI step on each pull request. Momentic keeps plain-English tests in the repo and runs them with `momentic run`.
- Gate merges on the CI job. A failed test returns a non-zero exit code, which fails the required PR check.
- Shard larger suites across parallel CI jobs to reduce feedback time. Tests must avoid shared state and ordering dependencies.
- Send Slack alerts on failure rather than every successful run. Include the failed check and run details so engineers can investigate.
- Run required checks against a staging or preview deployment before merge. Run a narrower smoke suite against production after deploy.

## Why e2e tests belong in the pipeline, not just on a laptop

Coding agents and continuous delivery increase the number of changes engineers must verify before merge. Manual review cannot confirm that every user flow still works. An e2e suite can perform those checks consistently, but only when the pipeline runs it for each relevant commit.

A test that runs only on a laptop provides information without enforcing a merge policy. Developers can forget to run it, use a different environment, or merge after a failure. CI/CD turns the same suite into a gate because the runner records a pass or failure for the commit.

Configure the CI runner to start the suite for pull requests and for any branch pushes that need independent verification. The test command should return zero when every test passes and a non-zero exit code when any test fails. CI converts that exit code into a status check, and branch protection can require the check before allowing a merge.

The gate should cover critical behavior rather than every possible scenario. A slow or flaky suite delays unrelated changes and teaches engineers to distrust failures. Reliable checks on authentication, checkout, or another core user flow provide a better starting point than broad coverage with poor signal.

## Running end-to-end tests in your CI runner

A CI runner needs a reachable test environment, the Momentic CLI, and any required credentials before it can execute end-to-end tests. Keep the plain-English test definitions in the repository so each commit runs the corresponding version of the suite.

The following GitHub Actions job fragment runs the suite for every pull request. It assumes that the runner image already contains the Momentic CLI and that the job can reach the target environment.

Install the Momentic CLI before the final step when the runner image does not already contain it. Configure the target environment and credentials according to your Momentic project settings, and store sensitive values in the CI provider’s secret manager rather than the repository.

A portable CI integration depends on the CLI process returning a meaningful exit code. `momentic run` exits successfully when the tests pass and returns a nonzero status when they fail. GitHub Actions converts that status into a failed job, which can then serve as a required pull request check.

Run `momentic run` as a shell step after checkout, dependency installation, and deployment of the test environment. The surrounding configuration differs by provider:

- [GitHub Actions workflow syntax](https://docs.github.com/actions/writing-workflows/workflow-syntax-for-github-actions)
- [CircleCI configuration reference](https://circleci.com/docs/configuration-reference/)
- [Jenkins Pipeline syntax](https://www.jenkins.io/doc/book/pipeline/syntax/)
- [Buildkite pipeline configuration](https://buildkite.com/docs/pipelines/configure)
- [GitLab CI/CD YAML syntax](https://docs.gitlab.com/ci/yaml/)
- [Bitbucket Pipelines configuration](https://support.atlassian.com/bitbucket-cloud/docs/configure-bitbucket-pipelinesyml/)

Each runner uses the command's exit status to determine whether the step passed.

Shell wrappers must preserve the test command’s exit code. A script that catches a failure and exits with status zero will make CI report a passing job despite a failed suite. Avoid suppressing errors with constructs such as `momentic run || true`, and enable `pipefail` when piping command output through another process.

## Blocking a pull request on test failure

A passing test run must control whether code can enter the protected branch. Running end-to-end tests in CI without making the job a required check provides feedback, but developers can still merge after a failure.

The gate follows a direct causal chain.

1. The CI runner invokes the test command, such as `momentic run`.
2. A failed test makes the command exit with a non-zero status.
3. The CI runner records the job as failed and reports a failed check on the commit.
4. The repository’s branch protection rules require that check to pass before allowing the merge.

Configure the CI step so it preserves the test command’s exit status. Shell wrappers, report-generation scripts, and commands such as `|| true` can accidentally turn a failed test run into a successful job. If later steps need to upload traces or screenshots, capture the original exit status and return it after those steps finish.

A required check also needs a stable job name. GitHub branch protection associates the merge rule with a named check, so renaming the CI job can leave the rule pointing at an obsolete check. Keep the gate name consistent across CI configuration changes, and verify that skipped or cancelled jobs cannot satisfy the rule.

GitHub enforces the gate through [required status checks in protected branches](https://docs.github.com/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches). GitLab can require a successful pipeline before merge through its [merge request settings](https://docs.gitlab.com/user/project/merge_requests/). Available controls vary by plan and repository configuration, so use the current provider documentation for exact setup steps.

Flaky tests require an explicit policy because every failed required check blocks the merge. Automatic retries can filter occasional infrastructure failures, but repeated retries can also hide regressions. Quarantine a known flaky test outside the required suite, assign an owner, and restore it only after fixing the source of nondeterminism. A smaller reliable gate provides more useful protection than a broad suite developers learn to rerun until it passes.

## Sharding e2e tests across parallel CI jobs

As an e2e suite grows, one sequential job can dominate CI time and delay every required check behind it. Sharding divides the suite across concurrent jobs, which reduces wall-clock time when the runner has enough compute available. Each job receives a shard index and total shard count, then executes only its assigned tests.

Most CI systems expose matrix or parallelism-style mechanics for creating these jobs. The exact configuration differs by runner, but the pattern stays consistent. The CI configuration starts several equivalent jobs, and the test command uses each job’s shard values to select a distinct subset of the suite. The required status check should pass only after every shard succeeds.

Parallel execution requires isolated tests and predictable environments. Two shards that update the same account, order, or database row can produce inconsistent failures or corrupt test data. Give each run unique users and records, or provision separate databases when practical. When tests must share a resource, use explicit synchronization instead of fixed delays or assumptions about execution order.

Interdependent tests create another failure mode. A test that assumes an earlier test created state may pass sequentially but fail when CI assigns both tests to different shards. The preferred fix is to make each test atomic. Each test should create its own prerequisites, verify one behavior, and clean up its state. When a legacy flow cannot be separated, keep its dependent steps together and enforce their order within one shard.

Equal test counts do not guarantee equal runtimes. A shard containing authentication setup, file uploads, or long polling may run much longer than shards containing short navigation checks. Partition tests using historical duration when your test runner supports it. Otherwise, group known slow tests deliberately and review shard timings in CI. A balanced split keeps one resource-heavy shard from becoming the new sequential bottleneck.

## Staging vs. production: when each check should run

Run fast end-to-end checks against a preview or staging deployment before merge. After deployment, run a narrower smoke suite against production. The staging job should remain a required PR check. A failed test returns a non-zero exit code, which fails the existing check and prevents the merge.

Staging catches behavioral regressions before customers encounter them. Each PR run must test the candidate commit, usually through a preview environment or a staging deployment tied to that commit. A shared staging environment can produce misleading results when another deployment replaces the code under test.

Production smoke tests cover failures that staging cannot reproduce. Production may use different configuration, infrastructure, data, or third-party integrations. Keep the suite focused on critical flows, and avoid tests that create destructive or irreversible data.

A production check runs too late to gate the original PR. Instead, its failure should stop a progressive rollout, trigger a rollback when your deployment system supports one, or alert the engineer responsible for the deploy. A production failure should not retroactively change the completed PR check.

## Send actionable Slack alerts after failed CI runs

Slack alerts surface failures to engineers who are not watching the PR check. Configure failure-only notifications by default so successful runs do not reduce the channel’s signal-to-noise ratio. Each alert should identify the commit and environment, then link to the failed run and its logs.

Post one summary after all shards finish so engineers receive a single result for the suite. Configure the notification step to run even when an earlier test step fails, but preserve the original nonzero exit status so the alert does not turn the CI job green.

The message should include the repository and commit, target environment, failed shard or test, and links to the PR check and retained artifacts. Keep Slack credentials in the CI provider's secret manager, and restrict the webhook or app token to the intended channel.

## Comparing CI-integrated e2e testing approaches

Choose a CI-integrated approach based on test ownership, suite partitioning, execution infrastructure, and the amount of browser-level control engineers need. Each approach can contribute a required check, but it assigns authoring and maintenance work differently.

| Approach | Gate mechanic | Parallelism and sharding support | Who maintains the tests | Where tests live |
| --- | --- | --- | --- | --- |
| Momentic CLI | momentic run returns an exit status that the CI provider can report as a required PR check. | CI jobs can run separate test groups in parallel. | Engineers maintain plain-English test definitions in the repository. | Tests live in the repository. |
| Playwright in CI | The Playwright command returns a nonzero exit code when tests fail, which fails the CI check. | Playwright supports parallel workers and suite sharding across CI jobs. | Engineers write, debug, and maintain the test code, selectors, fixtures, and test data. | Tests live in the repository. |
| QA Wolf | The managed service reports run results through its repository and CI integrations. Repository rules determine whether the result blocks a merge. | QA Wolf manages parallel execution within the service, subject to the configured engagement and capacity. | QA Wolf provides managed test creation and maintenance, while engineers review failures and product behavior. | Tests and execution primarily live in the external service. |
| BrowserStack CI plugins | The underlying test runner reports failure through the CI job, while BrowserStack provides remote browser and device execution. | BrowserStack runs concurrent sessions according to account capacity. The test framework or CI configuration partitions the suite. | Engineers maintain the Playwright, Selenium, or other supported test suite. | Test code lives in the repository, and execution runs on BrowserStack infrastructure. |

No testing tool blocks a merge by itself. Your repository’s branch protection rules must require the CI or service-generated check.

Playwright offers direct control over browser automation, but your engineers own the surrounding maintenance. BrowserStack expands browser and device coverage without taking ownership of test authoring. QA Wolf moves more test creation and maintenance into a managed service. Momentic keeps test intent in the codebase and exposes execution through a CLI, which fits repositories where engineers want reviewable tests without maintaining equivalent browser scripts.

## FAQ

**Can I run e2e tests on every commit or only on pull requests?**

Yes. Configure your CI trigger for pushes, pull requests, or both. A smaller suite on each commit gives faster feedback, while the required pull request suite can cover every flow needed for the merge gate.

**How does an e2e failure block a merge?**

The test command must return a nonzero exit code when a test fails. CI converts that exit code into a failed status check, and branch protection blocks the merge when that check is required.

**How do I rerun only failed shards?**

Persist each shard identifier with its result, then rerun the failed CI jobs against the same commit and environment. Some runners rerun failed jobs natively, but rerunning individual tests requires support from the test tool or a report-based retry step.

**Does a flaky test block merges?**

A flaky test blocks the merge whenever it returns a nonzero exit code through a required check. Limited automatic retries can reduce transient failures, but repeated flakiness should trigger quarantine with an owner and removal deadline rather than silent retries.

**Can I gate on staging and skip production checks?**

Yes. Run the merge-blocking suite against staging, then deploy after it passes. A smaller post-deploy smoke suite can catch production-specific failures involving configuration, infrastructure, or data, but that suite reports deployment health rather than gating the earlier merge.

**Why did a manual test run fail without sending a Slack alert?**

The Slack step may run only for CI events, specific branches, or scheduled jobs. Check the notification condition and confirm that the manual trigger reaches the same reporting step. If every failed run needs an alert, route supported trigger types through one notification job rather than relying on separate test-runner integrations.

## Setting up your first CI gate

Your first CI gate should cover one critical flow, such as authentication or checkout. A narrow test keeps initial failures easy to diagnose and gives the check a clear owner.

Store the test in the repository, run it against the candidate deployment, and make the CI job a required status check. To try the Momentic path, start with `npx @momentic/wizard@latest`, then use `momentic run` in CI.

## Frequently asked questions

How do I run end-to-end tests in GitHub Actions?    Run the test command as a workflow step after checkout and environment setup. Momentic suites run with `momentic run`, which exits non-zero on failure so GitHub Actions marks the job failed and the check red.    How do I block a pull request when an end-to-end test fails?    Make the test job a required status check in branch protection. The test command's non-zero exit code fails the job, and the required check then prevents merge until tests pass.    How do I shard end-to-end tests across parallel CI jobs?    Split the suite by directory or tag across jobs that each run a subset with the same test command. Tests must avoid shared state and ordering dependencies so each shard is independent.    How do I get test results into Slack after a CI run?    Add a notify step that runs on failure and posts the failed check plus a link to run details. Alert on failures, not every success, so the channel stays signal-rich.    Should end-to-end tests run against staging or production?    Run the required gate against a staging or preview deployment before merge, and a narrower smoke suite against production after deploy. Blocking suites belong pre-merge; post-deploy checks verify release health.

Still have additional questions?

## Keep reading.

[Resources   Cross-Platform Test Automation: One Suite for Web, iOS and Android     Wei-Wei Wu     10 min read](/blog/cross-platform-test-automation)[News   Mo: A Future Without Tests     We spent more than two years making test automation easier to build with Momentic. Now you don't have to touch one ever again.     Wei-Wei Wu     2 min read](/blog/mo-launch)[Resources   Best Visual Regression Testing Tools: 10 Compared for 2026     Ten visual regression testing tools compared for 2026: Applitools, Percy, Chromatic, BackstopJS, Argos, Storybook test runner, Playwright toHaveScreenshot, Lost Pixel, Meticulous and Momentic, with pricing model, CI integration and diffing method for each.     Wei-Wei Wu     12 min read](/blog/best-visual-regression-testing-tools)

## Close the feedback loop.

Point Momentic at your app. Free to start, no credit card.

[Try for free](https://app.momentic.ai/signup) [Contact sales](/sales)