Test Automation: Good, Bad and Ugly

The modern approach to software quality and software development life cycle requires that business guys, developers and testers understand that the long manual test phase, although often still necessary, must be reduced to a minimum and replaced by test automation. Working in continuous delivery and continuous integration environment requires us to create automated tests that run on demand, checking our application integration and it’s core functionality correctness. However, there are still many problems with designing and writing automated tests, resulting in their costly maintenance or abandonment in favor of a return to manual processes.

goodbadugly

In this article I will focus on describing common good practices of test automation. This post is more than an overview than complete reference guide. Broader aspects, such as the Page Object pattern or Page Factory will be described in detail in a separate article on this blog. Although most practices apply for every types of automated tests, this article refers to independent functional and acceptance testing projects. Example used in article can be cloned from my github.

Story telling

Tests are the best project documentation. They are always ‘alive’ and up to date with all the changes in the project, because otherwise usually we will not be able to perform application build. We should also pay attention to their readability – test classes should include a ’story telling’, readable code, similar to the common, human language. Below you can see an example of the logically correct test, which readability, however, leaves much to be desired:

This test definitely needs to be refactored. Proper test should have all the logic implementation hidden behind helper methods, and the test class should contain only the test case essence to illustrate the flow. Example below (test only, without helper methods):

Given When Then

Test code fragmentation into Given When Then sections is one of the most common practices in writing of tests. The Given block should contain initial conditions, which are not a part of test scenario, but are required to perform it. In this section you can implement, for example, database connection inicialization, or login in to online store. When block should contain test case implementation. In our case, it may be an attempt to make a transaction on the database or to buy an item at the online shop. Then block should contain result from When block. It would include assertion for database transaction denied, or charging credit card from our online shopping. Let’s add Given When Then block to the test from the previous example:

Naming Convention

One of the best ways to increase readability of your tests is to adopt a naming convention defining what should be the positive test result. This allows you to find out test case only by the method name. As you can see in our example with Facebook logging in test, test method name shouldNotLoginWithWrongPassword() immediately tells us about the test case context and expected result. Of course naming convention with should, isn’t the only one possible. There are many approaches, and you can also create your own convention. However, it is important to use descriptive names. The same applies to the test class names – the class name should correspond, for example, to the name of application functionality under test.

Before and After

Many tests contains repeated steps. This can be either opening the browser, downloading authorization token or just initialization of the objects. Most modern testing frameworks have features that allows to mark methods to be perform before / after each test or before / after each test class. In case of one of the most popular test framework in java world – junit, these are methods annotated with @Before /@After (the method is executed before / after each @Test method) and @BeforeClass / @AfterClass (the method is executed before / after each test class). Another important thing is to make test cases independent of each other and to “clean up” after each tests. Test results should be identical if you fire up a single test and if you run all the tests in the project. For example, in case of Webdriver, initializing web driver object in @BeforeClass method would be bad practice, since each test should be operated in independent environment, and therefore it should be rather initialized in @Before method, and “killed” in @After method (new browser instance is opened at the beginning of each test and closed when test is done). Here is the example of using junit annotations in our test:

Externalize Constants

The most common argument against automated functional tests is their high cost of maintenance. Test implementation is mostly based on the shared API or the structure of html website and while their changing, a single test require constant updating to a new contract.

Good practice that allows to reduce maintenance cost is externalizing as many constants as much as is required. A very good example are webdriver tests, where html selectors are one of the most frequently-changing elements of the page. Here is an example of externalizing html selectors of our facebook login test:

Summary

Proper and effective test automation can be a difficult task. However, there is a group of techniques and practices, which noticeably reduces the maintenance costs of tests. Well implemented automated functional test plan is a great improvement in the software development lifecycle and should be taken into account in the development process of every team.