There are some concepts that people pay extraordinary attention to while developing, but they tend to go under the radar for testing. I believe this is the case for concurrency and parallelism. In this post I’d like to review them and discuss their importance in the testing area.
Running your tests in parallel:
This means execute several tests at the same time in different machines. Sometimes this is run in an external location (for example in “the cloud” – which is basically a group of physical computers placed on several undisclosed locations)
Could you do run tests at the same time in the same machine? Technically, if you have multiple processors, then they might run at the same time for your tests, but it’s not a given. Sometimes, for machines with several processors, you would need to specify their exact use while writing your code. If that’s the case, our code would get too complicated and machine dependent, which wouldn’t be ideal.
Concurrency: what is it and why would you need it?
You may be thinking: “hold on your lynx, I’ve run tests in parallel in my same machine and they worked as a charm”. To what I reply: how are you so sure that they were actually executing in parallel and it’s not an “illusion” from your machine? Computers tend to be very good at giving the impression that things are done at the same time: you can move the mouse and watch a video and one thing does not interfere the other. But that does not mean that is happening this way, the processor just switches between them so quickly or eyes don’t realise about it.
Concurrency happens when something appears to run at the same time as something else. This is what you could be experiencing if you run tests in “parallel” in your machine.
Even thought this concept might seem far from the testing world, as many applications are developed with this at their core, there are many errors that could be caused by it. Therefore, we should always be aware and try things that could potentially cause them. When some error seems hard to reproduce and almost random, suspect there could be some concurrency issue happening (or parallel, if more machines are involved such as servers).
The importance of independent tests
In order to achieve parallelism, you need to make sure that the tests are not going to interfere each other. Let’s see an example below:
Test 1: “Item can be added to shopping cart” : adds a item to shopping cart. Verify cart has a item.
Test 2: “Item can be removed from shopping cart”: adds a item to the shopping cart and removes it. Verify the cart is empty.
Can you figure out what issues could be caused by these two tests if they were to be executed at the same time? If the shopping cart is shared across the tests, the timing of execution could make one or both tests fail randomly. It is very important that tests provide an unique value and that value is the same under the same circumstances and every time the test is executed.
For better tests, we might check if the value is in the cart or not, rather than if there are more values in the cart. Also, consider what would happen if two tests of the same type execute in parallel (two people pressed the run button). For example, consider the following execution of the two tests before called twice (1.1 and 1.2 are two execution of the same test 1 from above):
Test 1.1: adds item.
Test 1.2: adds item.
Test 1.1: verifies it’s in the cart. Removes it.
Test 1.2: verifies is in the cart (there were two of them, so it is) and removes it.
Unless there are issues on remove (which should have its own test) everything should work as before. However, what about our other test (test 2) executed twice:
Test 2.1: adds item to cart.
Test 2.2: adds item to cart.
Test 2.1: removes item.
Test 2.1: Check if the item was removed. Fails!! (There is still one item there)
Test 2.2: Continues and passes.
A solution could be to verify the number of items before removing, but even so, you are not guaranteed that another test would not add another one right after you checked. For this, you might want to add unique values of items per test execution (the items might be stored on a database under a unique ID)
If uniqueness couldn’t be implemented this way, maybe you could have a method to get items from a list based on your IP and execution time, so items would be picked at random and it would be more secure (although this still does not guaranteed independence).
Another way of implementing uniqueness is to use different users per execution (so your shopping cart is unique to your test case), but you might have a restriction on your number of test users and you might need a way of orchestrating them across test executions.
Finally, you could use some sort of orchestration on the test execution to make sure nobody else executes this test at the same time as you. The right solution depends highly on the implementation of your app and your testing platform.
Lastly, you should be careful with methods like “initialise” and “cleanup” to arrange circumstances like this as well. These methods should be used for common repeated operations among tests such as variable initialising, not to perform tasks as to deleting all values from the shopping cart.
Dependencies on testing
Sometimes we might need to deal with dependencies on testing. I’m not talking about: “always execute test 1 before test 2”. That’s probably a really bad idea that you should avoid. But what if you need to communicate two tests? Imagine you are testing a chat platform and need to test the following feature:
Friend 1: opens chat. says hi
Friend 2: should see a new chat with hi. Sends received
Friend 1: sees received.
In this case, you should orchestrate everything from within a test, and make sure that if you run the same test twice, it wouldn’t interfere with itself. There are many ways in which you could do all of this, but that’s… well… another story…