Flaky Tests

Flaky tests are tests that could pass or fail for the same code. This is a problem because the failure of a flaky test doesn’t always indicate a problem with the code, but you can’t just ignore the test because you could be ignoring a bug.

In the the blog post “Where do our flaky tests come from?”, Jeff Listfield, a Senior Software Engineer at Google, talks about the the potential causes of flaky tests and what can be done to avoid creating flaky tests. He demonstrates the correlation between the objective size of the test (binary size, memory usage) and the likelihood for it to be flaky. He also shows a correlation between certain tools and a higher rate of flaky tests, however, the reason for this is because larger tests are more commonly written using those tools. The tools themselves only contribute a small amount to the likelihood of a flaky test being created. When writing tests you should think about what code that you are testing and what a minimal test would look like in order to minimize the likelihood of creating a flaky test.

I chose this topic because effective use of test cases is very important in software development because it allows you to make sure that you’ve addressed and tested all of the product requirements, allows future testers to run your test cases when needed, and also makes it possible to build automated scripts to run as many tests as possible. By writing your test cases out, you also won’t need to constantly repeat the process and remember what values you’re testing every time as they’ll already contain all necessary variables, allowing you to maintain consistency in your tests.

This blog post in particular was interesting because it uses data gathered from real tests used in Google’s continuous integration system to show a cause of flaky tests and how to avoid them. Before reading this blog I didn’t realize how important writing test cases really was, but seeing just how many automated tests that Google used (4.2 million!) which lead to me doing more research on their importance. It also reminded me that the best solution is usually the simplest, in that you should remember to keep your test cases as simple as possible to avoid creating a flaky test and giving you a headache in the future trying to figure out what’s wrong.

Source: https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: