Introduction
In our previous post, we gave an overview of how test automation gets a bad rap. Many organizations try, struggle, and ultimately fail at adopting a successful test automation strategy, and as a result come to the wrong conclusion that test automation doesn’t work, when in fact, it is just not effective in the way they are trying to use it – similar to a lot I hear and see with Agile oddly enough. In this second part of a multipart blog series, I want to discuss expectations, and how to properly set them. One major reason I have seen test automation fail is due to unreasonable expectations or expecting the wrong things from automation. So, with that, let’s dive into some test automation misconceptions.
Test Automation will Replace Manual Testing
I’ve seen/heard hundreds of variations of this, and all of them worry me some. “Automation will steal testers’ jobs”, “Automation will replace manual testing“, “We’re going to automate 100% of our testing”, and so on. As I mentioned before, I have worked at a LOT of different places, and not once have I seen this be the case. Even with automation in place, there are manual testers. I have spoken at a bunch of conferences, and I have asked thousands of testers, “Have you seen automation replace testers?” I have never heard that automated tests themselves have put testers out of a job. The only thing I’ve heard is that testers who are unwilling/unable to automate have been replaced by those who are/can.
There are two main issues that come along with this expectation. The first, which we’ll dive into a bit more later on, is that if you’re trying to automate everything, you’re probably automating things incorrectly. Automation does not directly replace manual testing. These activities are different in shape and size, and an automated script shouldn’t directly replace a manual test. If it does, you’re probably doing something wrong. The second, which is just as damaging, if not more so, is the pressure and goals from management. I have literally heard the CEO of a company tell me, “I cannot wait until you are done with the automation, so that I can fire all my testers.” Ignoring how happy he looked when he said that (it was a not a safe environment to work in), that just isn’t going to happen.
You Can’t Automate Everything
Even if you managed to automate all of your testing (which won’t happen by the way), you still need testers around to manage those automated tests. The application is constantly changing and growing, and as a result, you’ll need to update your tests as the application grows and add new tests as well. On top of that, it’s not usually cost effective to automate all of your testing. Non-functional testing aspects like usability, layout, accessibility and more, are incredibly difficult and expensive to automate all aspects of. Typically, it’s much cheaper to automate part, and manually examine the rest. Stayed tuned for another post on how to choose what to automate, and what not to.
There are definitely some organizations that are out there that claim 100% test automation, but I can tell you, the reality is, there is still manual testing occurring. Places like Google and Netflix tout a fully automated testing environment, but that’s not really a fair claim. Ignoring the fact that those are huge organizations and comparing yourself to them is setting you up for failure, while they may not see/perform any manual testing, it doesn’t mean there isn’t any. Google, for example, let’s their end users do some of their testing. Call it beta testing if you will, but Google has such a rapid and seamless DevOps pipeline built out, that an error caught in production is easy to fix and doesn’t cost them much. They automate what is important, and what isn’t, let the end users ‘test out’. It works very well for their model, but it’s an important distinction to note: they understand their quality risk profile, and allow their software to be vetted and tested by others.
I have asked thousands of people what their automation level is, and I usually hear anywhere from zero to about fifty percent. I once heard someone tell me they had 100% automation coverage, and I thought that was amazing. After talking to them some more, I found out that they still did manual testing: they do exploratory testing on some parts of some of their applications. I thought it was an interesting disconnect, but important to note. Just keep that in mind when you hear these claims about 100% automation, it doesn’t mean that no manual testing is going on.
Clarity of Benefits Is Important
What ends up happening, as a result of this misconception, is that managers and organizations do not get the value they were expecting out of automation. They were expecting major cost cutting in personnel, but instead that has not happened. So even if you have succeeded in writing a fantastic framework, and automating all the things, you’ve failed. Instead, you need to set the expectations properly. No, we’re not going to fire all the testers, instead we’ll use their time more effectively. Because what you will find is that when you don’t need to test X all of the time because it is automated, you can finally get around to testing Y. We only have so much time, and since I have yet to meet a tester who has had enough time to test everything they want, automation can enable them to free up some time to get that overlooked work done. That in turn produces greater test coverage, which can lead to a better (and hopefully lower) risk profile when it is time to release.
A side effect of being able to expand coverage, is that more bugs should be able to be found sooner. This in turn saves money (it is significantly cheaper to fix a bug found in QA than in Production). This added automation can also decrease development time, whether it is by speeding up CI, fewer botched deploys to QA, or even just a shorter regression cycle; it is an immediate productivity boost for everyone. While these things will take time to achieve, there are benefits, just not the ones your boss might be is expecting. So it’s important to make sure to outline the real benefits, especially if you hear people expressing the wrong expectation. (Yes, I told that CEO he was wrong, and that wasn’t the point of automation. He blew me off, saying it was a joke, but I heard him continue to say it to other people – it was a rough contract).
Next Time
As I mentioned above, this is the second part of a multi-post blog series. Next up, I plan to expand on the point I started above, about how automation should look to directly replace manual testing. While you want your automation to cover the same things you are testing manually, taking a manual test case, and scripting it simply won’t have the desired effect. It will be brittle, miss things, and most importantly, not be as effective as the manual test case. Unfortunately, this strategy is most testers go to when writing automating tests (no shame, most don’t know any better, it was definitely mine when I started out). Next post I will discuss why this is a bad strategy, and how to better develop automated tests and scripts to address the problems.
If there is a particular area you’re struggling with, please leave a comment below, and I’ll either try to address it there, or add it onto my list for topics to write on. Good luck, and happy testing!
