.png)
Share this post
The significance of end-to-end tests in bug detection pre-deployment. It details Cypress usage and introduces Cucumber for more readable tests. It advises a balanced approach, combining these tests with other levels for optimal efficiency
End-to-end tests help you catch bugs before you deploy to production, but why should you care about bugs in production?
Bugs often require costly rework and decrease customer satisfaction. In some extreme cases, features from your site can be rendered completely unusable. The question you might be more interested in is how can I spot bugs long before they happen in production?
If you’re already familiar with end to end tests as a concept, you can skip the introduction and go straight to the code examples in Cypress or Cucumber.
If you are new to the topic, keep reading and learn about the power of end-to-end tests!
I’m sure you’ve heard of the test pyramid, the idea that you should test applications by different levels of specificity:
End-to-end tests just make sense: usually backend and frontend environments are developed without much synchronization, sometimes even by different teams. Suppose your backend routes are well tested, as well as your frontend components. How can you make sure that it’s going to work well when it all comes together? Some critical features require you to replicate the user flow to make sure everything is working as intended.
Let’s look at an example.
Imagine you have a website for pizza deliveries. Early in the user flow you ask the user if they are vegetarian, and if so, your site will only show vegetarian options in the menu.
The menu listing is provided by your backend but at some point, the API interface contract changed, and the data sent from the back end makes your ‘vegetarian filter’ fail. Your back end logic seems correct, you tested it. Your frontend component worked as intended, you tested it as well. But since you mocked the data coming from your backend in those tests, you couldn’t catch this error.
End-to-end tests allow you to see the whole flow, and test communication across your application layers, and as such, prevent bugs from the one above to reach your customers.
So now you’re asking yourself, how can I harness all this power? Surely it’s a pain to test this complex scenarios!
Well, with Cypress and Cucumber, you could write tests in 5 minutes that your product owner can understand at a glance. Let’s see how to do it.
Cypress is a JavaScript based E2E testing framework. Unlike Protractor, another popular E2E testing framework for Angular exclusively, it isn’t Selenium based, which requires you to install a Web driver to interact with your browser. Protractor, Selenium and many other frameworks can help you write end-to-end tests. You can get an idea of the variety by reading this article.
I am going to focus on Cypress, because it comes with everything you need to run end-to-end tests, it’s easy to set up, and very well documented. It’s also an open source project.
To show you how easy it is to use, let’s look at an example of a Cypress implementation of the pizza menu scenario described above. I’ll assume you have npm installed. If that's not the case, you can find the commands that suit you in the official documentation.
First, you need to install Cypress.
npm install cypress --save-dev
Then, to run cypress, use npx cypress open
A window similar to this one will open. Cypress comes with a variety of well documented examples to get you started with end-to-end tests ready to be run.
I’ve created my own end-to-end test file : veggie.spec.js. Let's take a look at it to learn the basics of Cypress.
You’ll recognize the “describe/it” syntax common to testing in JavaScript.
The structure of your tests will also be familiar to you, starting with your Setup, or “Given” statement…
describe('veggie-menu-path', () => {
it('Visits the home page',() => {
// Cypress starts out with a blank slate for each test
// so we must tell it to visit our website with the
// `cy.visit()` command. cy.visit('http://localhost:3000')
})
})
…followed by your test and assertion
it('asks if user is vegetarian', () => {
// We use the `cy.get()` command to get all elements
// that match the selector.
//
// Then, we use `should` to assert that there are
// two matched items
cy.get('.is-vegetarian').should('contain', 'Are you a vegetarian')
})
In this example, we use the get method to select an element in your Web page based on its CSS class. We then make the assertion that we want to test: that when we visit the landing page of our site, we ask the user if he is vegetarian.
This test has two issues.
First, our selector depends on the CSS class, which is highly susceptible to change.
To fix this, you could add a data-cy attribute to the HTML element you want to select, that way you can find such element independently of its styling.
cy.get('[data-cy=is-vegetarian]')
Of course, your data-cy might also change but since it is deeply tied to your tests, changing it means you are changing your tests. It isn’t as dangerous or frequent as changing your styling or implementation.
You could alternatively bypass the selection via attribute and find directly the text that you want to see on your page.
cy.contains('Are you a vegetarian')
Check out Cypress’s best practices for more on the topic.
Second, we only test for an element to be present in the DOM, which doesn’t guarantee that it is visible.
To improve your assertion, you could use 'be.visible'. Your complete assertion now looks like this.
cy.get('[data-cy=is-vegetarian]').contains('vegetarian').should('be.visible')
Let’s see it in action !
This is the code for the whole ‘vegetarian option’ scenario
As you can see, I modified the first test to add a click action to simulate a user path and get to the real meat of the scenario (no pun intended): checking whether we only see vegetarian options in the menu.
The assertion shown here is fairly poor: I know that my menu has 4 pizzas total, and 2 of them are vegetarian. So if the menu has only two elements, then I’m all set, right?
Not really, not only could I add or remove elements from my menu which would require changing the test every time, I might also be hiding another bug: what if only the two first pizzas from the menu are showing, whether they are vegetarian or not ?
Let’s make our test more robust with the following assertion and break it down.
const menuItems = cy.get('[data-cy=menu]').children()
menuItems.each((item) => {
cy.wrap(item).find('[data-cy=item-logo]').should('have.attr','alt').should('include','vegetarian');
})
That is technically all you need to write end-to-end tests. Open cypress and launch your test to see it in action.
But as we could see in the last example, it can get quite complex and the meaning of your test can get muddled in the implementation of it. What you want out of your tests is for them to be readable and not spend long creating new scenarios. And that’s what Cucumber is here for.
Cucumber is a framework that allows you to write and automate scenarios based on a functional specification that describes user behavior: this is called BDD, or Behavior-Driven-Development.
It does so by defining a '.feature' file written in Gherkin syntax, coupled to 'step definitions', which is just code that implements each of the Gherkin statements.
End-to-end tests are a perfect example of BDD in action, which is why Cucumber is such a good addition: it allows us to focus on the user instead of the technical aspect. Let’s get started!
Cucumber supports multiple languages, but we’ll look at an example on JavaScript.
To allow Cypress to integrate Cucumber features; we’ll install the cypress-cucumber-preprocessor npm package:
npm install --save-dev cypress-cucumber-preprocessor
Then follow the steps in the Getting started section of the package documentation to add cucumber as a Cypress plugin and make your final configurations. Now it’s time to write our test.
Create a '.feature' file on your cypress/integration folder. In this file, you'll declare your feature and the scenarios you want to test. Then describe the scenario using the Gherkin syntax.
Feature: Cypress Pizzas Menu Selection
Scenario: Vegetarian only Menu
When I visit the landing page
Then I should see "Are you a vegetarian"
Define your step definitions inside a folder with the same name as your '.feature' file. In my case, it looks like this.
import { When, Then } from 'cypress-cucumber-preprocessor/steps';
When('I visit the landing page', function () {
cy.visit('<http://localhost:3000>')
});
Then('I should see {string}', function (expectedText) {
cy.contains(expectedText).should('be.visible')
});
Notice how I am reusing the Cypress statements we created in the previous version of the test, only wrapped in When and Then Cucumber methods.
This is the final version of the test.
.feature file
Feature: Cypress Pizzas Menu Selection
Scenario: Vegetarian Only Menu
When I visit the landing page
Then I should see "Are you a vegetarian"
When I click on the button "Yes"
And I click on the button "See the menu"
Then I should see only vegetarian options
.js file
On the one hand, you have your scenario specification, focusing solely on the functional side of things; on the other, the technical implementation using best practices.
Now let’s watch it in action!
And that’s it! Have fun making your own end-to-end scenarios with Cypress and Cucumber.
You might be tempted to start using end-to-end tests for everything, given how simple and practical they are. But beware, end-to-end tests can be costly to fix. I’ll give you an example from my personal experience.
I worked in a project where we ran end-to-end tests on our CI. Each scenario ran for 2 to 5 minutes, and used to break very often.
Slowly but surely we started improving our tests. We wrote them using best practices, used Danger.js to notify us whenever we changed a translation key, since wording changes can make your tests fail, and focused on a limited amount of scenarios we deemed critical.
Remember the test pyramid. Define your critical scenarios and complement existing unit and integration tests with end-to-end. Every other scenario and feature should already be covered by other tests.
If you want to know about a concrete case study of end-to-end tests in the QA process, check out this article.
And if you are interested in diving deeper into the world of testing, take a look at this article, on the subject of mutation testing.
Don’t hesitate to contact me if you have any questions about the subject!
