Automation seems to be the de facto standard in testing nowadays, where we should be spending most of our efforts. So, why are we still getting critical bugs in production after working hard to automate all our tests? Is test automation failing us?
In this talk Diego Molina will tell you his story about having all his tests automated simplified the software release process. In this talk, Juan Luis Fernández Serrano will implement performance concepts and how he applies them in his day to day testing with what he calls "domestication".
Pushing a bug fix to our customers is faster because our quality gates are automated. Shipping a new version to production is safer. If that is all true, why are we still getting outages in commonly used features in our software?
On a Friday morning, Diego fixed a bug, tested manually, added automated tests, and shipped the fix to production. On Monday morning, a customer opened a ticket showing that the fix had created a new bug. A bug that was even worse because it was blocking customers from using an important feature in the website.
Diego felt really bad and did not understand how that happened. What did he miss? Everything was automated and tested! Together with his team, a quick hotfix solved the issue and customers were able to use the website properly again. But even if a fix was in place, Diego didn’t really understand why it was fixing the issue. In short, the root cause of the bug had not been addressed.
Diego took a step back to understand the situation. After reading the documentation and talking to colleagues he understood what the problem was: he never spent time understanding the system architecture and how the application works. These are basic concepts in testing, no matter if it is automated or manual. By prioritizing writing code over creating a good test plan, Diego created a bigger bug than the one he wanted to fix.
A production outage showed Diego that writing automated tests before thinking about what to test is a recipe for failure. After realizing this, doing root cause analysis, and creating a test plan, he outlined tasks to prevent this situation from happening again.
This talk shows that testing concepts are more important than ever. In a time where tools promise to do everything, we focus less on what and how to test, and more on using tools to test rather than employing testing concepts. It outlines how a test strategy can leverage a continuous testing setup. More importantly, it is ok to fail, but it is not ok to keep failing for the same reason. In addition:
- How to view failures as a learning opportunity by identifying the problem and doing root cause analysis
- How to start from a problem statement and develop each of the Test Automation steps to solve the problem
- Build a Continuous Testing setup as a result of solving the correct problem in Test Automation
This talk is aimed towards Testers (Manual & Automated), Developers, Test Leads and Managers interested in understanding test automation strategies, best practices, and continuous integration.