Prompting 101
Writing assertions
Guide to asserting on complex application state with Momentic’s AI
Momentic’s AI is trained to understand HTML, the accessibility tree, and screenshots. Use a combination of these three modalities to verify assertions. Read more here.
Here are some poorly crafted assertions that are likely to be flaky, misunderstood, or incorrect:
Best practices
- Keep assertions short and specific. Break down complex assertions into smaller, manageable parts. This helps the AI understand your intentions. We recommend using no more than 20 words.
- Provide useful context. To avoid misinterpretations, include information such as a specific part of the page the AI should focus on or the element you are looking for. Explain any non-standard terminology.
- Avoid ambiguity and subjectivity: Minimize “grey area” when formulating your assertion - the answer to your assertion should be clearly true or false given the page state. If your assertion is highly nuanced, explicitly state the criteria you want the AI to use. For example, you can say “the textbox contains the exact value ‘heading1’” rather than “textbox has heading”.
Examples
Here are some good examples of how to write assertions using AI check:| Category | Example |
|---|---|
| Presence (or absence) of text | A congratulations message is shown in the modal |
| Presence (or absence) of elements | The page shows 3 search results |
| Logical statements | The publish date of this article is more than 30 days ago |
| Page state | There is no error message present on the page |
| Color | The buy button is blue |
| Layout | The ‘XL’ button is the largest button in the submission form |
| Images & video | There is a LinkedIn icon on the page |
| Category | Bad example | Alternative |
|---|---|---|
| Malformed assertion | Search results | There are search results displayed on the main body |
| Ambiguous assertion | The order of the results is correct | The search results are sorted by date |
| Multiple focuses in a single assertion | The logo is blue, the page shows 4 results, and the last result is more than 30 days old | Split into 3 separate assertions |
| Assertion contains an action | Wait for the results to appear | Use a wait step if necessary, and then assert that the results have appeared |
| Assertion about imperceivable details | The color of line divider is #ff5733 | The color of the divider looks red |
| Subjective assertion | The page looks well designed | No text in the table overflows its cell |
| Assertion depends on changes over time | The video plays a 10-second advertisement for an insurance product | N/A |
Pre-trained behavior
The assertion agent is pre-trained to follow specific thought patterns to ensure AI safety, testing consistency, and ease of interaction. Momentic has carefully tuned the following behaviors:- Minor typos are tolerated: The agent ignores minor typos. For example, if you write “the username is
john@@gmail.com” but the actual username isjohn@gmail.com, the AI will pass the assertion since the test author likely typed an extra@. If your test genuinely relies on a small, single-character difference, explicitly state this and use single quotes around the text value (e.g. “the username is exactly ‘john@@gmail.com’). If you are generating test data, we recommend creating long, distinct values using JavaScript functions likecrypto.randomUUID()orfaker.word.words(3). - Visual criteria are verified using the screenshot: The assertion agent is required to verify any visual criteria mentioned by analyzing the screenshot. It is not allowed to infer the visual state using HTML attributes or CSS properties. If a described element is not visible in the screenshot (e.g. because it is hidden, covered, or out of the current viewport), the assertion will be evaluated as false. Examples of visual criteria include color, size, position, and shape.
- Counts are treated as exact requirements: If the assertion states that a certain number of objects are present, the agent will evaluate the assertion as false if the actual number of objects does not exactly match the expected number. This includes the case where the actual count is greater than the expected count.
- Limited real-world knowledge: The agent does not have access to up-to-date real-world knowledge such as the news or the current date. In addition, the agent is also trained to focus on the current page state. You can pass the agent dynamic context using
{{}}expressions, but we recommend using a JavaScript step to evaluate assertions that do not depend on the page state (e.g. mathematical calculations).
Data visualizations
You can also assert on SVG or canvas-based data visualizations rendered by libraries such as D3, Hicharts, Recharts, and more. To improve accuracy, use the following guidelines:- Maximize or expand your graph so that it takes up a majority of the screen. If your charting library does not have this functionality, you can try setting
document.body.style.zoommanually using JavaScript. - Provide clear labels and units for all axes. Include grid lines when possible.
- For interactive visualizations, use Click or Hover steps to show tooltips and labels before making assertions.
Troubleshooting
- Assertions with actions: The assertion agent cannot execute actions (e.g., “expand the dropdown”, “wait for the page to load”) and will not be able to understand such queries. Instead, split the action and assertion into separate steps.
- Assertions over time: The AI does not support assertions that evaluate changes over time (e.g., “after 5 seconds, a popup appears”, “the table has shifted upwards”). Instead, use a Wait step to pause the test and then check for the expected post-condition.
- Assertions on small visual details: Due to resolution downscaling performed by AI providers, the AI may not be able to perceive small visual details such as exact hex codes, colors of indicator dots, colors of thin font faces, and shapes of SVGs smaller than 12px. We recommend keeping assertions general enough to avoid these edge cases.
- Complex math: The AI may struggle with complex mathematical calculations (e.g., “the average slope of the graph is 31.4”). Instead, simplify the assertion to a more straightforward comparison (e.g., “the graph shows an upward trend”).
- Complex conditionals: The AI may not handle complex conditional logic well (e.g., “if the user is logged in, check that their profile photo is visible; otherwise, check that there is a Google icon on the page”). Instead, break it down into simpler assertions that do not rely on conditional logic.