Software test automation snake oil

Humans are able to evaluate hundreds of problem patterns, some of which can only be specified in purely subjective terms. Many others are complex, ambiguous, and volatile.

Therefore, we can only automate very narrow spectra of testing, such as searching for technical bugs i. What is more important is that testing is not only about finding bugs. As the Testing Manifesto from Growing Agile summarises very illustratively and to the point, testing is about getting to understand the product and the problem s it tries to solve and finding areas where the product or the underlying process can be improved.

It is about preventing bugs, rather than finding bugs and building the best system by iteratively questioning each and every aspect and underlying assumption, rather than breaking the system. A good tester is a highly skilled professional, constantly communicating with customers, stakeholders and developers.

So talking about automated testing is abstruse to the point of being comical. Test automation on the other hand is the automated execution of predefined tests. A test in that context is a sequence of predefined actions interspersed with evaluations, that James Bach calls checks. These checks are manually defined algorithmic decision rules that are evaluated on specific and predefined observation points of a software product.

And herein lies the problem. If, for instance, you define an automated test of a website, you might define a check that ascertains a specific text e. When executing that test, this is exactly what is checked—and only this. So if your website looks like shown in the picture, your test still passes, making you think everything is ok.

A human on the other hand recognises with a single glimpse that something has gone awry. But if test automation is so limited, why do we do it in the first place? Li, J. Ma, R. Conradi, J. Ji, and C. Dossani and N. View at: Google Scholar M. Hartman, M. Katara, and A. Persson and N. Auguston, J. Michael, and M. Cavarra, J. Davies, T. Jeron, L. Mournier, A. Hartman, and S. Vieira, J. Leduc, R. Subramanyan, and J. Xiaochun, Z. Bo, L.

Juefeng, and G. Yu and G. Blackburn, R. Busser, and A. View at: Google Scholar P. Santos-Neto, R. Resende, and C. View at: Google Scholar G. View at: Google Scholar A. Strauss and J. In most cases although it differs from place to place developers write code against a set of requirements. But if the GUI keep changing constantly from one release to another without any requirement for that change then, there is a problem in the process. You may have the best tool and the best for your environment architecture is in place and you will still have problems with your automation because of a faulty process.

I think there is a need to educate QA managers about the benefits and limitations of automation. There is a need to separate the facts from the fictions. But here is the problem, in most cases consultants are brought in to fix problems of prior attempts instead of initial setup. At this point the managers have already learned painfully the pitfalls of automation. In order to avoid this painful experience I would recommend most automation engineers will agree with me to spend more time up front doing research about the styles and techniques of automation and find an architecture that fits the environment.

There is no doubt that automation adds a great value to overall QA process but, short of knowledge and understanding about automation and lack of planning can also cause a nightmare. Site Hosted NuvoDev. Main navigation. Breadcrumb Home. Software Test Automation - Myths and Facts. Conclusion I think there is a need to educate QA managers about the benefits and limitations of automation. About us! Who are we? Privacy Policy Disclaimer. Software Test Management.