Preemptive Parallelism

If you work with tests at the pointy end of the test automation pyramid — be they browser-driven, end-to-end API tests, or any sort of test that involves the entirety of your stack, you’ll no doubt be aware that tests implemented at this level can take a long time to run. Even a relatively small suite of Selenium or similar browser-based tests can quickly expand to take half an hour or more, and if your requirements expand to cover multiple browsers, this then expands accordingly.

The large amount of time spent executing tests at this level can lead to a proportionally large increase in feedback time and this can be expensive in terms of context switching for members of the team, time spent re-testing bug-fixes and so on. Ideally we want our tests to provide feedback to the team as quickly as possible whilst also remaining reliable. Quick execution means quick feedback, and quick feedback is desirable.

When a test suite grows beyond what the team feels is an excessive execution time, there are a number of possible avenues to reduce feedback time whilst not reducing coverage. One that is often applied is parallel execution of tests. Most test frameworks have the possibility to run tests in parallel across multiple threads — for example cucumber-js provides a native parallel mode, TestNG offers a number of options for parallel execution, newcomer Playwright offers a parallel mode and a number of options therein. Whilst there are plenty of other ways to streamline your test suite and reduce execution time, assuming your framework supports it, running tests in parallel is often seen as having a large impact for little effort.

It is worth noting that in this context we are only using parallel execution as a means to reduce feedback time. Parallel execution of tests can also potentially uncover bugs or issues in the application under test, but if these sorts of issues are likely to be common in your application, it is likely to be worthwhile introducing parallelism in some of your lower-level test suites in order to discover these issues earlier.

At the top of the pyramid, however, introducing parallel execution to an existing test suite can be a substantial piece of work — it can introduce a number of unexpected and complicated bugs in a codebase that was designed to run in a single-threaded environment. Tests that previously relied on shared state, existing data or tests which make assumptions about the state of the underlying application can fail intermittently, and the underlying issues can be hard to pin down. Furthermore parallel execution can expose previously unforeseen issues in libraries, your CI setup, or third party service providers (such as cloud-based browser testing providers) — for example you may find with browser-based tests running in the cloud that your provider will sometimes queue tests as instances are created/destroyed, causing previously unseen timeout issues.

In my experience, as parallel execution is often only thought about by the time the test execution time is at an unacceptably high level (or is about to be, for example new requirements to test across multiple browsers), the team then need to slow or even halt new development whilst the existing codebase is transitioned to run in parallel. In addition, if a limitation is exposed with a third party service, this may necessitate further rework.

To try and avoid this situation, ideally we should adopt parallel execution as soon as possible when developing a set of tests that we expect to be long-running -ideally at the very beginning. This has a number of advantages:

• If we run our tests in parallel from the beginning, we are already taking steps to reduce feedback time from the beginning, even if we only have a handful of tests to start with.
• We can be confident our codebase and all our supporting infrastructure is parallel-friendly, avoiding any costly changes or rework in this area later on.
• Finally, if our test execution time reaches a level at which we feel it is not providing fast enough feedback to the team, we can simply increase the number of tests running in parallel to reduce it to a more acceptable level with minimal rework.

tl;dr: End-to-end tests almost always end up having comparatively long run times. Start thinking about (and implementing) parallelism early — ideally before you write any tests at all — to avoid painful and costly rework later on.

View this article on our Medium Publication.