STOP! Automating everything for "regression" is a waste of time

Publish Date: Mar 12 '19

29 10

Before you automate

Do your analysis, identify your biggest risks and choose how you can mitigate those risks. This is abstract, so what kind of risks might you identify?

Allowing invalid input crashes the application or allows corrupt data to be recorded
Users can no longer do something they could before
Users can do something now they were prevented from doing before, that they should not be allowed to
Usability is compromised so it's now more difficult or confusing to use the application
Application on longer performs as required
Application no longer projects privacy or security

There are many more risks you could identify for your own applications, but militantly it boils down to this:
The application on longer behaves how it is expected, preventing the customer or business getting the desired value.

Be selective

Given you have identified your key risks, ask yourself, will automating your UI give you confidence you have mitigated these risks?

If yes - stop reading and go make that automation happen!

If, like me, the answer is almost always "not completely", think carefully before you sink too much time into UI and front-end automation. And start asking the question "so how do I mitigate the risks identified?".

At some point, there is no single answer, but stick with it. Work with your business and/or client you will get there.

So what options might you have?

API Automation combined with UI automation
Chaos or "Monkey" testing
Exploratory testing, supported by tools and scripts executed "on demand"
Data-driven automation

If you come up with some good ideas on this, let me know!

If it doesn't bring you joy, throw it away

So - you already have an automated regression suite? Is it giving you value? is it bringing you joy?

Remember, it's important not to let your regression tests to go stale. Make sure they are still giving you value, and they are not getting flaky. If they are taking a long time to run, are they still worth it?

Don't be afraid to stop running regression tests if they no longer bring you joy! For joy, read "confidence to make changes".

Comments 10 total

Jesse PhillipsMar 13, 2019
This is so often misunderstood. Daily I'm trying to eliminate swaths of tests to get the most out of the time testing.

When I work with others on the automation being written I ask what is this testing for and the answer is often specific to the incident and not the issue. Which prevents a generic test to catch other case.
- Ben DowenMar 13, 2019
  
  I would love to know, do these incident specific tests catch regressions?
  - Jesse PhillipsMar 13, 2019
    I would love to know that too.
    
    We haven't had good review of failure analysis and have spent so much time updating these "incident" tests I don't really believe the claim that they have found bugs.
    
    But I do have bias because I like to evaluate risk of an issue reoccurring, severity if it happens, cost to maintain the test due to routine changes of a release.
- Alan BarrMar 13, 2019
  That's been the rub. The people doing the work typically are not looking at the big picture and likely find it difficult to think that way.
Alan BarrMar 13, 2019
Love the idea of using spark joy for one's tests. I have definitely seen teams not delete tests due to fear.
- Ben DowenMar 13, 2019
  
  This actually goes for all regression tests not just automated ones. :)
- f00dog-x300Mar 15, 2019
  I've seen this as well, but in some ways I can understand a bit. Some people don't want to delete tests because they are afraid that product managers will change their minds in the last minute. However, if done right with version control, I don't think this should even be a problem.
f00dog-x300Mar 15, 2019

I really like this article. People can spend long days, weekends, and nights writing up every single test in the book. However, when a production issue hits and it was not caught by an extensive regression test people will start wondering if automation is worth it or not. When that starts to creep in then it's all downhill from there.
Hemanth YamjalaSep 18, 2019

Benjamin, I can understand where you are coming from. 100% test automation is anyway more harmful than 100% manual testing. You atleast still have control over what is being tested and how it is being tested. But, I believe that one should not skip regression testing entirely. Automated regression testing can be applied to some of the more static components, and the dynamic one can be saved for manual discretion. You might like to check this out - cigniti.com/blog/alert-need-regres...
- Ben DowenSep 18, 2019
  
  Absolutely! I have nothing against targetted automated regression. But I wanted to be clear that the goal of "Fully automated regression" can be dangerous and expensive. Automation is not trivial to "Right size". Any some automation is usually better than none.

Add comment

Ben Dowen @dowenb