Problematic Playwright pitfalls

Playwright is a powerful browser automation framework that recently overtook Cypress as the most popular E2E testing solution in the JavaScript ecosystem.

While full-featured and easy to get started with, we have ran into our fair share of unexpected, puzzling and dangerous bugs while working with Playwright at Momentic.

In this post, we're detailing the four most frustrating pitfalls we've encountered that have cost us dozens of engineering hours to remediate. Hopefully you can avoid them in your CI/CD pipelines as well!

Pitfalls

High page.screenshot() latency

Playwright's page.screenshot() method can hang for extremely long periods of time if your site loads custom fonts. Even though your site may not require these fonts to function correctly, by default Playwright will wait for any font loading network requests to complete before taking a screenshot.

If you notice that your screenshots are taking longer than 100ms to complete, turn on Playwright's debug logging and see if the output is similar to the following:

  pw:api waiting for fonts to load... +250ms
  pw:api waiting for fonts to load... +324ms
  pw:api waiting for fonts to load... +274ms
  ...

How do you bypass this error? With a little sleuthing inside the Playwright codebase, we discovered that you can force Playwright to bypass waiting for fonts to load with the PW_TEST_SCREENSHOT_NO_FONTS_READY environment variable. For example, JavaScript / Typescript users can simply add the following line before invoking .screenshot():

process.env.PW_TEST_SCREENSHOT_NO_FONTS_READY = "1";

Setting this environment variable improved screenshot speed by more than 10X for some Momentic customers!

Hanging page loads

Did you know that the default options for page.goto, page.waitForLoadState, and page.waitForURL can actually cause testing scripts to hang indefinitely?

By default, these functions wait until the page fires the load event. This event typically fires when all dependent resources have finished loading, including fonts, images, iframes, and scripts.

However, many sites never fire this event due to long-running ads, dynamic iframe content, or background analytics events that continuously "reset the clock" on the load state. In these cases, the aforementioned Playwright functions will never return. And while these functions do support timeouts, the default timeout is 0, which equates to an unlimited timeout.

Thankfully, Playwright does allow you to override which event these functions wait for. For example, the following call only waits until the DOMContentLoaded event, which fires immediately after the HTML document is parsed:

await page.goto("https://bestbuy.com", {
  waitUntil: "domcontentloaded",
});

Note that at the DOMContentLoaded stage, the page may not have finished loading all assets. While Playwright locators automatically wait for the targeted element to stabilize, any custom logic will have to incorporate explicit waiting logic. For example, if your goal is to extract the name of a button on the page, you should explicitly wait for the button to exist before performing the extraction:

await page.goto("https://google.com", {
  waitUntil: "domcontentloaded"
})
await page.locator("button").scrollIntoViewIfNeeded({
  timeout: 3000,
})
const name = await page.evaluate(() => {
  return document.querySelector("button").getAttribute("name)
})

In the above example, the scrollIntoViewIfNeeded method ensures that the button has fully loaded before the custom evaluate code begins.

Hanging locator.evaluate calls

Playwright supports evaluating JavaScript code in the browser with the locator.evaluate() function, which supports a timeout option. For example, the following code retrieves the tag name of an element on the page:

const locator = page.locator(".button")
const tagName = await locator.evaluate((elm) => elm.tagName, { timeout: 1000 })

Many engineers will assume that the timeout parameter controls the execution time for the function being evaluated. Confusingly, this is not the case.

In actuality, the timeout is the maximum amount of time Playwright will wait for the element to be available on the page. Even if your function does something incredibly expensive, Playwright will never terminate it!

For example, the following code that attempts to wait until an input element has a value will never terminate and likely cause your headless browser to crash:

const locator = page.locator(".password");
const passwordVal = await locator.evaluate(
  (elm) => {
    while (!elm.value) {
      console.log("element is still empty");
    }
    return elm.value;
  },
  {
    timeout: 1000,
  },
);

Because of this limitation, at Momentic we've learned to use evaluate() incredibly sparingly and closely monitor the speed of each call we do perform with our observability platform. Any time a call takes more than a second, we immediately alert on it to prevent test performance degradation.

Canceled navigations

Consider the following pseudo-code that attempts to sign in and then navigate to the dashboard sub-page:

await passwordInput.fill("password");
await signInButton.click();
await page.goto("/dashboard");

While this code seems correct at a glance, it actually contains a hidden race condition: if the redirection triggered by clicking the sign-in button happens too slowly, the redirection will actually be cancelled by the final page.navigate() call.

This is problematic since a complete redirection is usually required to obtain correct authentication cookies. Therefore, in all likelihood, the dashboard page will not consider the user logged in. Some browsers even explicitly crash with an ERR_ABORTED or ERR_CANCELLED code when a navigation is cancelled.

Why does this happen? It turns out that in-progress browser navigations are immediately interrupted by page.goto or page.reload calls. While Playwright attempts to wait for navigations immediately triggered by interactive commands to complete (e.g. clicking on an anchor tag), any JavaScript-triggered redirections can run into this race condition.

An easy but potentially flaky way to address this issue is to wait for the navigation to complete by explicitly waiting until some new text appears on the page. For example:

await signInButton.click();
await page.waitForText("Welcome");
await page.goto("/dashboard");

Alternatively, at Momentic, we built an in-house page load management system that automatically detects when new page navigations occur, and pauses all other Playwright operations until those navigations are complete. This avoids users having to explicitly craft waiting conditions in most cases.

Conclusion

Playwright is an amazing piece of open source software. However, due to its wide feature set and complexity, writing effective, bug-free, and maintainable Playwright code can be a challenge!

The four pitfalls outlined in this article are just some of the most frustrating quirks we encountered, investigated, and debugged over thousands of hours of operating Playwright at scale. We hope that this information saves you some tears and sweat!

P.S: If you are looking for a way to achieve quality without worrying about any of this, we built Momentic so that anyone can author complex E2E tests without a single line of code.