The Test Drama: Cypress vs Playwright - Control Your Tests (Part 2): TAGS & TEST FILTERS
#cypress#qa#playwright#automation
Sebastian Clavijo Suero

Sebastian Clavijo Suero @sebastianclavijo

About: Staff QA Engineer/SDET | Engineer in Computer Science | Cypress and Playwright Independent Contributor | Blogger | Accessibility — Creator of WICK-A11Y, CYPRESS-AJV-SCHEMA-VALIDATOR, and PW-API-PLUGIN

Location:
Portland, OR
Joined:
Apr 27, 2024

The Test Drama: Cypress vs Playwright - Control Your Tests (Part 2): TAGS & TEST FILTERS

Publish Date: Apr 24
6 2

Subtitle.

(Cover image from pexels.com by Gu Ko)

ACT 1: EXPOSITION

This is the second part of the article on controlling the execution of your tests in the Cypress and Playwright test frameworks (or Playwright and Cypress, if you prefer 😉). In this second one, we will focus on two features: Tags and Test Filters.

In addition to the out-of-the-box functionality provided by each framework, we will review and evaluate some interesting plugins that address certain gaps related to these features.

As mentioned in the first article The Test Drama: Cypress vs Playwright - Control Your Tests (Part 1): ANNOTATIONS & GROUP TESTS, I recommend reading this second installment in its entirety to gain a comprehensive understanding of these features in significant detail and depth.

ACT 2: CONFRONTATION

Let's explore these tools in detail to see how they offer complete control over our test framework, whether you're using Cypress or Playwright.

🏷️ TAGS

In test frameworks like Cypress and Playwright, tags are used for organizing and managing tests more efficiently. They help in categorizing tests based on certain criteria, such as the type of test, priority, or functionality.

These are some of the common uses:

  • Test Selection: Tags allow you to run specific subsets of tests based on certain criteria. For instance, you can run only "smoke" tests, "regression" tests, or tests related to a particular feature by specifying the relevant tag.

  • Filtering: You can filter tests during execution to include or exclude certain tags. This is useful in CI/CD pipelines, where different environments or stages might require different sets of tests.

  • Organization: Tags help in organizing tests into categories, making it easier to understand the scope and coverage of the test suite.

  • Reporting: Tags can be used to generate detailed reports that highlight specific areas of the software being tested, which can help in tracking test coverage and identifying gaps.

Overall, effectively using tags can significantly enhance the flexibility, readability, and maintainability of your testing framework. Therefore, we will learn how to use them in both Cypress and Playwright.

 

CYPRESS TAGS

Tags are not natively supported by the Cypress test framework, which might be surprising but is indeed the case.

To address this gap, there are two grep plugins you can use in your Cypress framework:

To provide some context: some time ago, Gleb created a grep plugin named cypress-grep. This plugin was submitted to Cypress.io, which renamed it to @cypress/grep, making it the official version. However, to introduce improvements, Gleb forked the newly named @cypress/grep plugin into a separate project called @bahmutov/cy-grep. He has since maintained his new plugin in the new repository.

Personally, I prefer Gleb's current version of the plugin and use it in my personal and organizational projects for several reasons:

  1. Gleb updates his plugin very frequently and keeps the dependencies up-to-date, since he relies on it heavily in his projects.
  2. He resolves issues quickly when reported. For example, I reported a problem with .only annotations coexisting with certain configurations just a few days ago, and he fixed it in a matter of hours!
  3. Gleb is the original creator of both plugins and is widely recognized as the Cypress plugin master, a well known fact in the Cypress community.

Although the weekly downloads of the @cypress/grep plugin on npm are approximately ten times greater than those of the @bahmutov/cy-grep plugin, I would like to share additional information from the GitHub repositories of both plugins that supports my opinion as of the date this article was written.

@cypress/grep source code was last updated 2 years ago:

Image description

@bahmutov/cy-grep source code was last updated 1 day ago:

Image description

@cypress/grep pull requests closed in the past few months (1):

Image description

@bahmutov/cy-grep pull requests closed in the past few months (108):

Image description

The configuration and use of both grep plugins are essentially the same. Hence, whatever we discuss in this article can be easily adapted for either one.

Another very cool feature of @bahmutov/cy-grep, which is not supported by the @cypress/grep plugin, is that when running your tests in 'open' mode (with the browser open), you can perform grep operations directly from the browser's DevTools console using the Cypress.grep() command.

Image description

But wait! There is more...

In "open" mode, you can also run just the failed tests from the DevTools console using Cypress.grepFailed().

Alright, it's settled... we'll use the @bahmutov/cy-grep plugin as we move forward in this article. 🤘

 

Plugin @bahmutov/cy-grep (by Gleb Bahmutov)

You can find this Cypress plugin in:

You can locate precise instructions on how to install the plugin in your Cypress framework in the plugin's README.md documentation.

It consists of four main steps:

1.Install the plugin as a devDependency in the project root using the terminal:

npm i -D @bahmutov/cy-grep
Enter fullscreen mode Exit fullscreen mode

2.Register the module in the e2e.js support file by adding the following lines:

// cypress/support/e2e.js

// load and register the grep feature using "require" function
// https://github.com/bahmutov/cy-grep
const registerCypressGrep = require('@bahmutov/cy-grep')
registerCypressGrep()
...
Enter fullscreen mode Exit fullscreen mode

3.Load and register the module in the cypress.config.js file within setupNodeEvents:

// cypress.config.js

const { defineConfig } = require("cypress");

module.exports = defineConfig({
  ...
  e2e: {
    setupNodeEvents(on, config) {
      require('@bahmutov/cy-grep/src/plugin')(config);
      // IMPORTANT: return the config object
      return config;
    },
  },
});
Enter fullscreen mode Exit fullscreen mode

4.Configure the environment variables grepFilterSpecs and grepOmitFiltered to true in your cypress.config.js file (recommended - we will discuss these variables later on):

// cypress.config.js

const { defineConfig } = require("cypress");

module.exports = defineConfig({
  env: { grepFilterSpecs: true, grepOmitFiltered: true },
  ...
});
Enter fullscreen mode Exit fullscreen mode

Tags can be any word representing the nature of the tests (like smoke). However, it is very common (I would even say recommended) to use the character @ at the beginning of the tag name for clarity (like @smoke).

Tags can be created at the group test level (describe or context) and the test level (it) by passing an object with a tags property as the function's second parameter. You can assign either a single tag (as a string) or multiple tags (as an array of strings) to the same test group or test.

This example below illustrates tags at both the suite and test levels, including specifications for either single or multiple tags:

// test-tags.cy.js

// The test suite 'User Registration' has two tags: '@user-reg' and '@smoke'.
describe('User Registration', { tags: ['@user-reg', '@smoke'] }, () => {

    // The first test tagged with '@happy' to represent a happy path scenario.
    it('should register a new user successfully', { tags: '@happy' }, () => {
        // Logic for testing successful user registration.
    });

    // The second test tagged with '@unhappy' to represent an unhappy path scenario.
    it('should not register with an already existing email', { tags: '@unhappy' }, () => {
        // Logic for testing registration with a duplicate email.
    });

});
Enter fullscreen mode Exit fullscreen mode

Then, you can select which test suites or tests to run using tags by passing them in the environment variable grepTags when executing the Cypress run command in the CLI: npx cypress run --env grepTags=... (we will will explore this in detail in the Cypress Test Filters section).

 

PLAYWRIGHT TAGS

Tags are natively supported by the Playwright test framework. And believe me... this is very very handy! 🤲

A tag in Playwright must start with the character @. You set tags using the tag property in the details parameter when defining a test. If you want to specify only one tag, you can provide it as a string. If you would like to tag a test with more than one tag, you can do so by providing an array of string tags.

import { test, expect } from '@playwright/test';

test('Sample test with 1 tag', {
    tag: '@smoke',
}, async ({ page }) => {
    // ...
});

test('Sample test with 2 tags', {
    tag: ['@smoke', '@regression'],
}, async ({ page }) => {
    // ...
});
Enter fullscreen mode Exit fullscreen mode

Tags also can be created at the group test level (test.describe):

// test-tags.spec.ts

import { test, expect } from '@playwright/test';

// The test suite 'Sample group with tag' has one tag: '@query'.
test.describe('Sample group with tag', {
    tag: '@query',
}, () => {
    // The first test does not have any tag
    test('Sample test one', async ({ page }) => {
        // ...
    });

    // The second test has 2 tags '@smoke' and '@regression'
    test('Sample test two with 2 tags', {
        tag: ['@smoke', '@regression'],
    }, async ({ page }) => {
        // ...
    });
});
Enter fullscreen mode Exit fullscreen mode

And that's it—no more drama or anything else needed! 🙌

Then you can select which tests or group tests to run in the terminal using the --grep option followed by the tags. We will cover the details in the Playwright Test Filters section.

 

⚗️ TEST FILTERS

Filtering tests by tags, spec files and test titles allows precise control over which tests to execute. This feature enhances efficiency by enabling QA Engineers to target specific tests relevant to their current work, facilitating faster and more focused testing cycles.

 

CYPRESS TEST FILTERS

In Cypress, you can natively select which spec files to execute by using the --spec command when running tests from the terminal.

However, Cypress does not support out-of-the-box filtering tests by tags or test titles. To achieve this, you'll need to use plugins provided by the Cypress community:

 

Filter by Spec files

Cypress provides built-in support for filtering tests by specifying spec files through the --spec parameter in the command line.

You can execute specific spec files by specifying their names, or run all tests in a folder that match a glob pattern. It is highly recommended to use double quotes when specifying the files to run, and the path can be either absolute or relative to the current working directory.

To run a specific test file:

$ npx cypress run --spec "cypress/e2e/my-spec.cy.js"
Enter fullscreen mode Exit fullscreen mode

You can specify multiple test files separated by a comma:

$ npx cypress run --spec "cypress/e2e/examples/actions.cy.js,cypress/e2e/examples/files.cy.js"
Enter fullscreen mode Exit fullscreen mode

To run all tests in a folder that match a glob pattern:

$ npx cypress run --spec "cypress/e2e/login/**/*"
Enter fullscreen mode Exit fullscreen mode

You can combine all of this with the --project parameter, but to be honest, I find it more confusing, so I won’t spend time on it here. If you’d like to see an example of this parameter, refer to the Cypress official documentation.

 

Filter by Test Title (with @bahmutov/cy-grep)

Cypress does not support filtering by test title out-of-the-box. To accomplish this, you would need to use the cy-grep plugin.

To filter tests by title, use the --env grep=... option in the terminal. If the title string you want to filter includes spaces, make sure to enclose it in quotes:

# Run all tests with "footer" in their title
$ npx cypress run --env grep=footer
# Run all tests with "footer links" in their title
$ npx cypress run --env grep="footer links"
Enter fullscreen mode Exit fullscreen mode

Title filters are also applied to test suite blocks (describe and context). If a suite matches the filter, all tests within that block will be executed:

# Run any tests in the blocks including "footer"
$ npx cypress run --env grep=footer
Enter fullscreen mode Exit fullscreen mode 
describe('test footer', () => {
  it('this will run', () => {})

  it('this will also run', () => {})
})
Enter fullscreen mode Exit fullscreen mode

Or Substring Matching

You can provide multiple title substrings to match by separating them with a ; character. Each substring will be trimmed automatically:

# Run all tests with "footer links" or "header links" in their title
$ npx cypress run --env grep="footer links; header links"
Enter fullscreen mode Exit fullscreen mode

Invert Filter

You can invert a filter by placing the - character in front of the string to match:

# Run all tests WITHOUT "footer links" in their title
$ npx cypress run --env grep="-footer links"
# Run tests with "footer", but without "links" in the titles
$ npx cypress run --env grep="footer; -links"
Enter fullscreen mode Exit fullscreen mode

In "open" mode, you can also run test by title using the command Cypress.grep() directly in the browser's DevTools console:

// Filter tests by title substring
Cypress.grep('header links')
Enter fullscreen mode Exit fullscreen mode

Image description

Additionally, you can run the filtered tests multiple times:

// Run filtered tests 10 times
Cypress.grep('header links', null, 10)
Enter fullscreen mode Exit fullscreen mode

 

Filter by Tags (with @bahmutov/cy-grep)

As previously mentioned, Cypress does not natively support filtering tests by tags, however this can be achieved using the cy-grep plugin.

You can choose which tests to run or exclude by using tags with the --env grepTags=... option. By including commas in the grepTags environment variable, you can separate multiple tags.

If a specific tag is not found in the specs, then you will get a warning in the terminal:

$ npx cypress run --env grepTags=@wrong-tag
cy-grep: could not find the tag "@wrong-tag" in any of the specs
Enter fullscreen mode Exit fullscreen mode

And Tags
Use the character + to indicate that both tags must be present in a suite or test for it to run. You can specify the tag string with or without quotes:

# Run suites and tests that have both tags @smoke and @regression
$ npx cypress run --env grepTags="@smoke+@regression"

# This one is equivalent
$ npx cypress run --env grepTags=@smoke+@regression
Enter fullscreen mode Exit fullscreen mode

Or Tags
When you want to run suites or tests that match one tag or another, separate them with a space character. In this case, you must always provide the tag string within quotes because of the spaces:

# Run suites and tests that have the tag @visual or the tag @regression
$ npx cypress run --env grepTags="@visual @regression"
Enter fullscreen mode Exit fullscreen mode

Inverted Tags
You can skip running certain suites and tests that include a specific tag by using the invert option, which involves prefixing the tag with the character -:

# Do not run any suites or tests that have the tag @quarantine
$ npx cypress run --env grepTags=-@quarantine

# Run suites and tests with the tag @smoke but without the tag @quarantine
$ npx cypress run --env grepTags=@smoke+-@quarantine
Enter fullscreen mode Exit fullscreen mode

Not Tags
You can exclude tests with a specific tag from running, even if they have a tag that should run, by using the "not" option. Simply prefix the tag with -- to skip it:

# Run suites or tests with the tag @visual or @regression but exclude those with the tag @quarantine
# (note that @quarantine is prefixed with -- due to spaces)
$ npx cypress run --env grepTags="@visual @regression --@quarantine"

# This command is equivalent to the previous one
$ npx cypress run --env grepTags='@visual+-@quarantine @regression+-@quarantine'
Enter fullscreen mode Exit fullscreen mode

Grep untagged tests

To run tests without any tags that are also within suites without tags, set the environment variable grepUntagged to true when run in the CLI command:

$ npx cypress run --env grepUntagged=true
Enter fullscreen mode Exit fullscreen mode

grepFilterSpecs and grepOmitFiltered

Previously, we recommended setting up these two environment variables when installing the @bahmutov/cy-grep plugin. Now, let's explore their intended use.

By default, when using the @bahmutov/cy-grep plugin, all specifications are run, and all internal filters (such as title-based filters) are applied. This process can result in unnecessary time being spent.

To pre-filter specs, you can set the environment variable grepFilterSpecs to true. Note that grepFilterSpecs works only with positive "greps" and is not compatible with inverted ones.

Additionally, the plugin defaults to marking all filtered tests as pending by using it.skip(). To completely omit these tests from the output, set the environment variable grepOmitFiltered to true.

If these environment variables were not set as configuration options during plugin installation, you can specify their values for the run directly in the CLI command:

# Filter all spec files, and run the suites and tests with the tag "@smoke"
# and do not display the filtered tests in the terminal
$ npx cypress run --env grepTags=@smoke,grepFilterSpecs=true,grepOmitFiltered=true
Enter fullscreen mode Exit fullscreen mode

Let's examine what the terminal will display when it encounters the spec file test-groups.cy.js that finds no suite or test matching the specified tags @user-reg+@smoke, under various configurations of grepFilterSpecs and grepOmitFiltered.

  • With the command npx cypress run --env grepTags=@user-reg+@smoke,grepFilterSpecs=false,grepOmitFiltered=false, all specifications are executed, filters (such as filters by title) are applied, and any filtered tests are marked as skipped:

Image description

  • With the command npx cypress run --env grepTags=@user-reg+@smoke,grepFilterSpecs=true,grepOmitFiltered=false, the output may look the same; however, specifications are NOT executed (significantly reducing runtime), filters are NOT applied (further saving time), and filtered tests are marked as skipped.

Image description

  • With the command npx cypress run --env grepTags=@user-reg+@smoke,grepFilterSpecs=true,grepOmitFiltered=true, specifications are NOT executed, filters are NOT applied, and filtered tests are omitted from the output:

Image description

With this plugin, you can also run tests by tags using the Cypress.grep() command in the browser's DevTools console when running Cypress in 'open' mode.

// Run filtered tests by tags (with tag @smoke or @regression)
Cypress.grep(null, '@smoke @regression')
Enter fullscreen mode Exit fullscreen mode

Image description

And also run by test title, tags and multiple times:

// Run tests with title containing "footer" and tag @regression 10 times
Cypress.grep('footer', '@regression', 10)
Enter fullscreen mode Exit fullscreen mode

 

⚠️ NOTE: Cypress supports a --tag parameter when running tests in the terminal or CI pipeline. However, this --tag parameter is not used to filter tests by specific tags. Instead, it is used to add tags to the recorded run, making it easier to identify them when displayed in Cypress Cloud.

cypress run --record --tag "regression,nightly"`

Image description

 

Plugin cypress-cli-select (by Dennis Bergevin)

Recently, Dennis Bergevin released a plugin called cypress-cli-select, which allows you to interactively select and run specific tests or groups of tests directly from your terminal. It uses the @bahmutov/cy-grep plugin as its core engine.

To install the plugin, add cypress-cli-select and @bahmutov/cy-grep as devDependencies:

$ npm install --save-dev cypress-cli-select

$ npm install --save-dev @bahmutov/cy-grep
Enter fullscreen mode Exit fullscreen mode

Instead of manually typing out spec file paths or using complex command line arguments, cypress-cli-select visually presents a list of your test files, suites, individual test titles, or tags, enabling you to select which ones to execute.

Just type in the terminal:

$ npx cypress-cli-select run
Enter fullscreen mode Exit fullscreen mode

And "select" you go!

Image description

Using the --choose-spec-pattern option, you can run selected specs in a specific order. However, in this case, you will not have the option to run test titles or tags.

$ npx cypress-cli-select run --choose-spec-pattern
Enter fullscreen mode Exit fullscreen mode

This is a very cool and useful plugin, especially when you are still developing and debugging your tests. Give it a try — it’s worth it!

You can learn everything about this plugin in its documentation. Additionally, there’s an intro video by Gleb that explains how to use it.

 

PLAYWRIGHT TEST FILTERS

Playwright offers robust native support for filtering tests, allowing you to target specific spec files, utilize tags, or focus on test titles, giving you greater flexibility and precision in test execution.

 

Filter by Spec files

You can run specific test files in the CLI by simply providing their file names. There's no need to include the full path, as Playwright will search all the directories within the tests folder that match the specified name.

Additionally, you can execute all test files within specific directories by passing the names of these directories (no need to pass the full path). If the directories contain nested subdirectories, Playwright will recursively include and execute all the tests within them.

# Run two specific test files provided
$ npx playwright test login.spec.ts logout.spec.ts

# Run all tests within the two specified folders
$ npx playwright test tests/contacts/ tests/profile/
Enter fullscreen mode Exit fullscreen mode

If you want to run tests that contain specific keywords in the file name, simply pass those keywords to the CLI.

# Runs tests that contain the keywords 'login' or 'logout'
$ npx playwright test login logout
Enter fullscreen mode Exit fullscreen mode

 

Filter by Test Title

To execute a test with a specific title, simply use the -g flag followed by the substring, enclosed in quotes, that must match the test title.

# Runs all the tests that contain the string "validate contacts" in their title
$ npx playwright test -g "validate contacts"
Enter fullscreen mode Exit fullscreen mode

That's it! Quick, easy, and hassle free! 🍰

 

Filter by Tags

You can run specific test suites (describe) or tests using the --grep option followed by the tags. Make sure the tags are enclosed in quotes.

# Runs tests with the tag "@query"
$ npx playwright test --grep "@query"
Enter fullscreen mode Exit fullscreen mode

If a specific tag is not found in the specs, then you will get a warning in the terminal:

$ npx playwright test --grep "@wrong-tag"
Error: No tests found
Enter fullscreen mode Exit fullscreen mode

And Tags

To run suites and tests that include both tags (AND condition), you will need to use regex lookaheads.

Regex lookaheads are a type of zero-width assertion that allow you to check whether a specific pattern exists (positive lookahead) or does not exist (negative lookahead) ahead in the input string without consuming characters. Lookaheads do not form part of the final match; instead, they assert a condition that must or must not be true.

Types of Lookaheads:

  • Positive Lookahead: (?=...)
    Ensures that the specified condition is present ahead in the string.

    • Example: a(?=b) finds an a only if it is immediately followed by b.

      Match: "ab", No match: "ac"

  • Negative Lookahead: (?!...)
    Ensures that the specified condition is not present ahead in the string.

    • Example: a(?!b) finds an a only if it is not followed by b.

      Match: "ac", No match: "ab"

# Runs tests with the tag "@api" and tag "@sanity"
$ npx playwright test --grep "(?=.*@api)(?=.*@sanity)"
Enter fullscreen mode Exit fullscreen mode

Image description

Well... if you ask me, I’d have to say that creating a simple AND condition is considerably more complex compared to the cy-grep plugin we discussed for Cypress. 😒

Or Tags

When you want to run suites and tests that match one tag *OR another, separate the tags using the | character.

# Runs tests with the tag "@regression" or tag "@sanity"
$ npx playwright test --grep "@regression|@sanity"
Enter fullscreen mode Exit fullscreen mode

Image description

Inverted Tags

You can skip running specific suites and tests that include a particular tag by using the --grep-invert option.

# Skip the tests that includes the tag "@sanity"
$ npx playwright test --grep-invert "@sanity"
Enter fullscreen mode Exit fullscreen mode

Image description

You can also filter tests in the configuration file via testConfig.grep and testProject.grep.

 

Plugin playwright-cli-select (by Dennis Bergevin)

And since Dennis didn’t want Playwright to be left behind, he created a playwright-cli-select plugin (the counterpart of his cypress-cli-select).

Because Playwright natively supports filtering tests by spec file, test title, and tags, it relies solely on the Playwright core, making its installation even faster than the Cypress version of the plugin.

$ npm install --save-dev playwright-cli-select
Enter fullscreen mode Exit fullscreen mode

To run the plugin, simply:

$ npx playwright-cli-select run
Enter fullscreen mode Exit fullscreen mode

And you get an amazing CLI user interface to select the filters 'on the go':

Image description

ACT3: RESOLUTION

Alright, how can I put this in the clearest way possible about what everyone’s been silently thinking as they read through this article?

Why are not tags and filters by title supported in Cypress as part of its core functionality? Why must users rely on a third-party plugin developed by members of the Cypress community to handle such a basic feature?

Both major grep plugins for Cypress were developed by the same individual, Gleb Bahmutov. Notably, the one featured in the official plugin directory hasn't seen updates in quite some time. It raises the question—what would the level of support for this feature be without Gleb's contributions?

I have to admit, tagging and test filtering in Playwright feels far more convenient and user-friendly. Yes, I’m not a huge fan of how conditional AND filtering by test names is handled, but overall, in my opinion, everything else tips the balance heavily in favor of Playwright.

Ok, I said it—someone had to, right?! 🤐

One last thing, though... If you want to rerun the last failed tests, Playwright supports this natively with the --last-failed flag in the CLI.

$ npx playwright test --last-failed
Enter fullscreen mode Exit fullscreen mode

And what about Cypress? Well, what do you think? 🎲
Bingo! You’ll need a plugin!

For this, you can use:

  • The aforementioned @bahmutov/cy-grep plugin by Gleb Bahmutov and invoke the command Cypress.grepFailed() from the DevTools Console.

Image description

  • Or the cypress-plugin-last-failed created by Dennis Bergevin, which works in both run mode and open mode. It also supports CI/CD pipelines.

Image description

If you want to experiment with all the examples discussed in this article, you can clone the repository sclavijosuero/cypress-vs-playwright-frameworks. 🧪 🧫

I'd love to hear from you! Please don't forget to follow me, leave a comment, or a reaction if you found this article useful or insightful. ❤️ 🦄 🤯 🙌 🔥

You can also connect with me on my new YouTube channel: https://www.youtube.com/@SebastianClavijoSuero

If you'd like to support my work, consider buying me a coffee or contributing to a training session, so I can keep learning and sharing cool stuff with all of you.
Thank you for your support!
Buy Me A Coffee

Comments 2 total

Add comment