We need to stop using white/black box testing categorisation

First things first: happy new year lynx! I wish to start this year with a somehow sensitive topic, and I want to apologise in advance if I say anything wrong, mind that it is said from my heart and my current knowledge.

According to “Biased” by Jennifer Eberhardt: “Categorization -grouping like things together – is not some abhorrent feature of the human brain, a process that some people engage in and other do not. Rather, it is a universal function of the brain that allows us to organize and manage the overload of stimuli that constantly bombard us”.

In testing, we also have categories that help us learn and understand the types of tests, such as functional vs non-functional. However, white vs black box categorisation is, in my honest opinion, inaccurate and inadequate. It is also the one that’s being more widely used, particularly for beginners. I believe we should stop using these terms, and in this article I will give my reasoning and a better alternative.

Note: For clarity purposes I will still use these terms until the end of the article.

Promoting stereotypes

You might think that white box tests are not better or worse than black box tests and therefore, it is OK to use those names. However, the idea behind it is that with white box testing you can see the code whilst you can’t with black box testing. With this categorisation we are promoting the stereotype that white is something clear, usually associated with purity, while black is opaque, usually associated with immorality.

If you really think about it, you would agree that this definitions make no sense.

Why can “you see the code” with white box test? You shouldn’t be able to see the contents of a white box at all. Consider this: most ceilings are painted white, can you see the sky or your neighbors throughout your ceilings? How can we normalize the fact that a white box is somehow see-through?

Here are three boxes with three different colours. I still can’t see inside any of them, can you? (Image credit)

Unit tests category chameleon

If I were to ask you in which category unit tests fall into, I bet most of you would say white box, because “you can see the code.” However, if you think in terms of TDD, you would write the tests BEFORE writing the code and therefore you are not seeing it, which by definition shouldn’t technically be white box.

What’s the actual difference between unit tests, positioned at the base of the testing pyramid, and the other tests?

The person who writes them writes also the code? But that could be said of any of the others, especially when shifting left.

The biggest difference is that these look for code structure rather than feature logic. For example, you don’t care about the number of products shown in the shopping cart, but that this particular piece of code can handle the right exceptions and have the right outputs.

Switching to clear vs opaque

In my opinion, just replacing the wording here, although it’s a step, it would not be enough. Whilst it is true that it would be a fast solution to avoid prolonging stereotypes and keep our systems and classifications intact and ourselves a bit more comfortable, I feel the stereotype would somehow still hold: people will first think white vs black and then change to these other names.

Furthermore, I still have more issues with this categorisation, which I’m going to continue arguing before giving what I think it’s the best alternative.

Grey box testing

You can tell when a category doesn’t make sense when it needs a lot of exceptions and an extra group to gather anything that can’t be placed in the others. Introducing grey box testing: when you… partially? sometimes?… see the code.

Grey box testing at the left… (Image credit)

As before, grey is another colour, if you use it in a box, you won’t see it’s interior. Not even partially. What should we use instead? Scratched box? Partially opaque box? It does not quite make sense. Which tests exactly would land in this category?

Test/QA people rarely work with white box

There are a lot of concepts behind white box testing that are really interesting but are usually incorporated in automatic code checking tools and rarely used daily by tests/QA people.

Cyclomatic complexity is rarely calculated manually, although sometimes it’s called out in code reviews. Most times this is part of code optimisation or happens behind other methods.

Statement testing, edge testing, branch testing, loop testing, Branch and Relational Operator testing and data flow testing are methods that are rarely planned, examined and automated in a frequent manner. Many of these terms are still unknown or unused by others on our industry today.

Most of the rest of the test pyramid have to do with what’s currently called black (or grey) box testing.

Structural vs logical

I think a clearer classification could be structural vs logical. In fact, in order to document myself for this article, I have been recently reviewing my notes from university and in them, it is said that these are alternative names these test categories are known for. Yet people rarely refer to them. Why not using them, when they are more suitable and less damaging?

The definition would be whether you are testing the structure of the code (number of branches, proper exception handling…) or the logic behind the feature (the shopping cart should have 0 products after a purchase).

Here is a tree: we can classify it by number of branches or colour of the trunk, but also by its function: does it give fruit or has flowers? Are they the right ones? (Image credit)

What about back-end testing / integration testing or database testing? Exactly the same: if you are testing the database for structure, then that would be structural, if you are testing it for logic, logical. We can reach more granularity with this definition, although we usually tend to test these for logic rather than structure.

This classification opens up to think about more tools to validate code. It also leaves unit test as a structural test with no room for confusion or need for extra categories.


I hope you understand that my point in this article is not to remove a category, but to enclose it in more appropriate terms and definition. If leaders such as Github have started a major shift in their branches to remove old coding terms, why can we not do this when we already have more appropriate names?

I really hope this post get to places such as ISTQB® and we start changing our terminology and explanations, as so many people look to them as the standard in the testing area.

As Miguel Ruiz said in “The four agreements” you should “be impeccable with your word”. Wouldn’t it be a great start of the year if we apply it to this test categorisation, using structural and logical instead of white box and black box?

Do you agree? Disagree? Let me know in the comments below. I could tell you more about structural testing and give you ideas to improve your coding with it, but that’s… well…another story.

The trust between test-dev

In many companies where development and testing are two separate efforts, I’ve noticed some testers trust too much in the developers. Even in companies where development is highly integrated with testing, this could result in missing tests, some of which could be crucial for finding important bugs in the app under development.

One of the clear examples of this is when it comes to CI/CD. Usually, an admin or devops set this up. The involvement of a test expert in this area is particularly important, as they have the skills to plan the right automated tests to validate the builds. I have seen pipelines where there is no testing at all, or a group of insufficient or irrelevant tests.

For some people in the quality team, the developers are their “source of truth”. I’ve fallen into this same error, with other colleagues with more years of experience than me: Whatever they say should be the right way – they know better. Comparing your knowledge to other’s in terms of “more-less” rather than “different” is a recipe for disaster…

Let me tell you the story of a time I trusted too much in an experienced dev: there was an obvious bug in staging environment. I’ve logged it. Someone else logged a duplicated. We were late for release. I said I couldn’t sign up with that bug there.

Butterfly, Insect, Colorful, Moth, Moth Cerura Vinula
From https://pixabay.com/images/id-814568/ – “Oh, hi there! I’m here because the dev lead says it is fine”

The developer assigned to the feature, called the PM and me to his office and both tried to reason me to sign up. I explicitly called out the risk. He said the bug was unlikely to be hit and he didn’t have time to fix it. I trusted him – I was only a junior while he was senior and longer in the company. My instincts told me that the seniority did not make him a fortune teller, but even so I said: “OK, I’ll sign up as long as you get my back if anything happens.” He reassured me and agreed, he said he would take all the responsibility if anything was to happen. Nothing was written down about it.

When users hit the issue in production, both PM and developer immediately blamed me. By writing. Both of them. My manager blamed me. They all said I missed finding a clear bug. When I explained everyone what happened, providing logs of the defect, which was still in open state and the duplicated, where everyone involved was informed. My managed called me to his office. He told me I shouldn’t have signed up and my behavior was not professional as we “shouldn’t point fingers to others”. In other words, I should have taken the blame the others put on me without even trying to clarify it. He didn’t listen to me and it affected my performance review, which was just a week later.

He told me I got two really bad “anonymous” reviews (hard to guess who did them, right?) and the rest, whilst good, were irrelevant.

At the start of that same review period, I found a fix to a very important defect on another team that was costing the company a lot of money and the manager of that team said that if a developer fixed it, they would be immediately promoted. I was the one that provided the fix. It did not seem to matter.

Even the worse of the situations can become a learning, whether it happened to you or someone else. What I learnt from this one:

  • Follow your instincts, if you think something is important stick with it. Avoid trusting blindly other team members based on their experience or how well you get on with them. Know your value and what you can give to the company: I decided to trust a dev based solely on his experience after having figured out an issue that many other experienced devs could not figure out.
  • Getting to an agreement is good, but it is better if you do so by writing, so everyone can remember and review it later on.
  • If you need to release a feature with a defect, that is OK, but plan for the worst and think about a course of action to assure the least disturbance to the users if they hit it.
  • Finally, if you get a manager that does not support you, consider moving away. Leaders and managers should be there to help you grow and learn, to challenge you and inspire you, never to belittle you. You should always feel they make an effort to understand and protect you, at least, that’s what I try to do when I lead/coach others.
Arm, Hand, Desk, Notebook, Pen, Writing, Write, To Do
From: https://pixabay.com/images/id-1284248/ – Plan for the worst and write it all down.

I hope this story I can help you out if you find yourself in a similar situation. Remember that every obstacle can become a learning.

As a final note: having trust is very important, given the right dialogues and plans of action. The opposite (not trusting the developers) is also not ideal. When there is not enough trust, walls are created between teams and things tend to get tested multiple times, but that’s well..another story.

30 days of programming languages – Java vs C#

I am starting to collect a long list of programming languages that I have worked with, but in reality, when I need to switch from one to another, it takes me a bit of time to get used to them again, specially after a while.

A couple of months back, I’ve challenged myself to practice a language each month, by solving a daily problem, to keep myself constantly updated on them. Since challenges seem to be fashionable, you can call this the #30DaysOfProgrammingLanguages challenge.

To do this, I started working with leetcode, which happens to have monthly challenges, so the choice of those problems was already conveniently made for me. These problems don’t take long to complete, and if they do it means I need to practice that concept, which in turns means that it’s a good thing to go over them.

I did April’s challenge with C# and May’s with Java, and here is what I’ve learnt:

Initial assumptions

Before starting my feeling was that both languages are close enough and I was going to need to change very little in between them, maybe some grammar. On the other hand, I thought that one of these coding challenges per month was not going to be enough to review all the details of the languages and to really get them I was going to need to build some app.


Casing differences:

Java prefers classes in uppercase, and methods in lowercase, the opposite as C#.

Length, length(), size(), count:

I am still wondering why do we need different grammar for the same concept. As far as I remember:

  • For C#, arrays use Length, strings Length and Lists and most other classes under Icollection, Count.
  • Similarly, for Java, arrays use length, string length() and the others usually size().

When in doubt, remember the rule of lowercase in Java, upper in C#.

Values and references:

For problems such as Validate Binary Search Tree, sometimes is convenient to send “null” for the first iteration of the methods. In Java, this means passing the object instead of the data type (so you are getting reference instead of value). In C#, it is possible to do this too, using nullable value types (“int?“), but I actually struggled many times with this problem in C# because this was not introduced until C# 8.0 (.NET core 3.0). It is possible to solve it without passing null, it’s just faster and cleaner to use that.

The following are two equivalent function definitions, for C# first, and Java second.

private bool isValid(TreeNode node, int? left, int? right);
private boolean isValid(TreeNode root, Integer lower, Integer higher);

Dictionaries and maps:

This is probably what needs biggest attention. I honestly find dictionaries much easier to work with in C# than in Java. For example, access of an element is done with [], in Java you need the get() method, which is already more typing and more confusing when getting inside other statements that use parentheses.

Iterating through a dictionary (in my experience, if you need to do this, keep in mind that there is probably a better way of doing that exercise than the one you are thinking of), is done by KeyValuePair in C# and entrySet() in Java.

On initializing, you need to use objects for Java while you C# would use data types when possible, which means you need to compare with equals rather than ==. Check below’s example for this last point, first code is written in C# and second the Java equivalent.

Dictionary<Char, Int32> myMap = new Dictionary<Char, Int32>();
 // Initialization omitted. 'a' and 'b' have the same integer.
 myMap['a']==myMap['b']; // -->works
 myMap['a'].equals(myMap['b']); // --> Compiler error: 'int' does not contain a definition for 'equals' and no accessible extension method 'equals' accepting a first argument of type 'int' could be found
 HashMap<Character, Integer> myMap = new HashMap<>();
  // Initialization omitted. 'a' and 'b' have the same integer.
 myMap.get('a') == myMap.get('b'); //-> sometimes would return false when true
 myMap.get('a').equals(myMap.get('b')); //-> works

Note as well how the use of generic are slightly different for both of them. Since most times it might not cause an issue, because the way the heap works, it might cause issues that are hard to reproduce. This was one of the problems that helped me reproduce the issue (in particular when there was a case with a very long string with at least two characters with the same frequency).

Interfaces for the data structures:

Interfaces in C# conventionally start with “I”, while in Java they might have a different names. One more example follows, C# code firstly and Java equivalent after.

IList<T> myList = new List<>();
List<T> myList = new ArrayList<>(); // There is no "List" class

Below is a generic diagram of equivalence of collection classes in Java vs C#, where left and blue represent Java conventions and green and right C#. Note the classes below are for the C# generic collection, but you might find keywords that would match Java’s in C# which are not from the generic collection (you can’t use <T> with them). The use of these are not recommended and I have not added them in the diagram’s C# side.

There are also some concurrent classes that are not included.

Diagram made with Creately. Concrete classes that inherit from Map/Dictionary are purposely not added. Zoom in might be required but worth it.

Let me know if you want another article comparing class by class and zooming in on each group, or if you find any errata on the diagram.

Rediscovering switches:

One great thing about leetcode is that once you have finished the challenge, it shows you a graph comparing your solution’s performance with others. That gives you the opportunity of checking how others did it and improving your skills. By doing this, I realized that I needed to refresh my knowledge about switch statements in c#.

Things I did not get to review

There were also some things I did not happen to find and I wish I would have reviewed:

  • Regular expressions
  • Lambda expressions
  • File manipulation
  • Tuples
  • Concurrency and async programming
  • Iterators over data structures (more in depth)
  • linq

Another issue to mention is that most times you can get away changing the objects that were passed by parameters in the problems, without penalty from leetcode, and this could lead to problems in a bigger code base.

It will be nice to have challenges that highlight these in a git repository or leetcode collection and add some reviewing these concepts.

You can see other differences here.


I actually completed several of these problems per day, and some of the days I would switch back and forth (especially if I find something interesting), but the point is to keep constantly updated with the languages and I was looking for something that I could do daily without getting burnt out. Even so, I did find many things I did not remember of the languages and I refreshed more than the syntax. I think under daily busy situations, one problem a day is still achievable.

If I was to go back to some versions of say, Java, I would encounter different issues than the aforementioned. Keep in mind these are the ones I noticed at the time of writing this article.

Overall, it was a great experience and review of concepts. I am looking forward to keep refreshing other languages and see how it affects me in the long run. So far, I feel I’m slower when writing code because I keep forgetting the right grammar and particulars of each language, hopefully this will help me getting faster while keeping me up to date across them all. I will keep comparing other languages and updating my experience, but that’s well..another story.

Concurrency and parallelism in testing

There are some concepts that people pay extraordinary attention to while developing, but they tend to go under the radar for testing. I believe this is the case for concurrency and parallelism. In this post I’d like to review them and discuss their importance in the testing area.

Running your tests in parallel:

This means execute several tests at the same time in different machines. Sometimes this is run in an external location (for example in “the cloud” – which is basically a group of physical computers placed on several undisclosed locations)

Could you do run tests at the same time in the same machine? Technically, if you have multiple processors, then they might run at the same time for your tests, but it’s not a given. Sometimes, for machines with several processors, you would need to specify their exact use while writing your code. If that’s the case, our code would get too complicated and machine dependent, which wouldn’t be ideal.

Photo by Manuel Geissinger from Pexels – This is a server room, which is basically a room full or computers without screens. When the use of these are allowed externally and interconnected globally, this is called ‘the cloud’

Concurrency: what is it and why would you need it?

You may be thinking: “hold on your lynx, I’ve run tests in parallel in my same machine and they worked as a charm”. To what I reply: how are you so sure that they were actually executing in parallel and it’s not an “illusion” from your machine? Computers tend to be very good at giving the impression that things are done at the same time: you can move the mouse and watch a video and one thing does not interfere the other. But that does not mean that is happening this way, the processor just switches between them so quickly or eyes don’t realise about it.

Concurrency happens when something appears to run at the same time as something else. This is what you could be experiencing if you run tests in “parallel” in your machine.

Even thought this concept might seem far from the testing world, as many applications are developed with this at their core, there are many errors that could be caused by it. Therefore, we should always be aware and try things that could potentially cause them. When some error seems hard to reproduce and almost random, suspect there could be some concurrency issue happening (or parallel, if more machines are involved such as servers).

The importance of independent tests

In order to achieve parallelism, you need to make sure that the tests are not going to interfere each other. Let’s see an example below:

Test 1: “Item can be added to shopping cart” : adds a item to shopping cart. Verify cart has a item.

Test 2: “Item can be removed from shopping cart”: adds a item to the shopping cart and removes it. Verify the cart is empty.

Can you figure out what issues could be caused by these two tests if they were to be executed at the same time? If the shopping cart is shared across the tests, the timing of execution could make one or both tests fail randomly. It is very important that tests provide an unique value and that value is the same under the same circumstances and every time the test is executed.

For better tests, we might check if the value is in the cart or not, rather than if there are more values in the cart. Also, consider what would happen if two tests of the same type execute in parallel (two people pressed the run button). For example, consider the following execution of the two tests before called twice (1.1 and 1.2 are two execution of the same test 1 from above):

Test 1.1: adds item.

Test 1.2: adds item.

Test 1.1: verifies it’s in the cart. Removes it.

Test 1.2: verifies is in the cart (there were two of them, so it is) and removes it.

Unless there are issues on remove (which should have its own test) everything should work as before. However, what about our other test (test 2) executed twice:

Test 2.1: adds item to cart.

Test 2.2: adds item to cart.

Test 2.1: removes item.

Test 2.1: Check if the item was removed. Fails!! (There is still one item there)

Test 2.2: Continues and passes.

From https://pixabay.com/images/id-4974313/ – This seems to be the most wanted shopping item of 2020, and we’ve just added too many of them to the cart.

A solution could be to verify the number of items before removing, but even so, you are not guaranteed that another test would not add another one right after you checked. For this, you might want to add unique values of items per test execution (the items might be stored on a database under a unique ID)

If uniqueness couldn’t be implemented this way, maybe you could have a method to get items from a list based on your IP and execution time, so items would be picked at random and it would be more secure (although this still does not guaranteed independence).

Another way of implementing uniqueness is to use different users per execution (so your shopping cart is unique to your test case), but you might have a restriction on your number of test users and you might need a way of orchestrating them across test executions.

Finally, you could use some sort of orchestration on the test execution to make sure nobody else executes this test at the same time as you. The right solution depends highly on the implementation of your app and your testing platform.

Lastly, you should be careful with methods like “initialise” and “cleanup” to arrange circumstances like this as well. These methods should be used for common repeated operations among tests such as variable initialising, not to perform tasks as to deleting all values from the shopping cart.

Dependencies on testing

Sometimes we might need to deal with dependencies on testing. I’m not talking about: “always execute test 1 before test 2”. That’s probably a really bad idea that you should avoid. But what if you need to communicate two tests? Imagine you are testing a chat platform and need to test the following feature:

Friend 1: opens chat. says hi

Friend 2: should see a new chat with hi. Sends received

Friend 1: sees received.

Photo by Roman Pohorecki from Pexels

In this case, you should orchestrate everything from within a test, and make sure that if you run the same test twice, it wouldn’t interfere with itself. There are many ways in which you could do all of this, but that’s… well… another story…

The HONEST (anti)guide to WFH

With the lock down that many countries are having these days and the need to work remotely, I’ve been noticing an increasing number of posts about how to work from home. I think many of the advises are quite obvious, but some of us tend to gravitate towards a “different” experience when we start doing it. So, here is my satiric guide to how to NOT work from home:

Free-Photos / 9091 images – Example of a perfectly fine home ‘desk’
  1. Turn off your alarm. No commuting means extra time, and the office has flexible starting time, what could possibly go wrong?
  2. Wake up, go to get some breakfast, put your phone to charge and realize you had a meeting 10 minutes ago. Which genius decided to set up a meeting at 9am when everyone usually starts arriving at 9.30m?
  3. Put on a shirt. Don’t worry about the bottom part (it’s an online call and you’ll be sitting down) If you are female improvise some make up without washing your face (no time for that).
  4. Take your laptop and sit in the living room. Everyone is still asleep, surely they wont disturb you.
  5. Try to pretend you’ve been in the call all this time when it’s your time to speak.
  6. Your favorite relative appears in the background and turn the TV on. Turn microphone and camera off while taking the laptop to a more private room.
  7. Come back to the call and apologize for having “connectivity issues”. Turn the camera off and on a few times to prove your point. Finish your talk.
  8. Your relative walks behind you and starts speaking to you without noticing you are in a call. Shut them up.
  9. Realize your microphone was on. Turn off.
  10. The call continues. Realize that the camera is pointing down and you were still wearing your pajamas. Turn it up and hope nobody noticed it. They have.
  11. Finally the meeting is off. Go to speak with your relative who is now very angry and offended because you shut them up and try to explain the situation. They don’t quite understand it.
  12. Go back to your improvised desk. Start checking your slack and email.
  13. Realize is time for lunch. Go to have lunch.
  14. Go back to work. Notice everyone started pinging you while you were having lunch.
  15. Reply to your coworkers, even the ones that are sending you blogs about how to work from home and other jokes.
  16. Go get a snack.
  17. Go back to work. Start reading blogs about how to work from home.
  18. Go to get a drink, the snack made you thirsty.
  19. Go back to work. Remember you were working and close the blogs. Do a bit of work.
  20. Go to the toilet (badly needed now because of the drink you had before).
  21. Go back to work. Realize the day is almost done and you have not done anything useful. Start doing some work.
  22. Your relative comes over. You chat for a bit because you feel guilty about before.
  23. Go back to work. Read the latest news about the global pandemic.
  24. Your neck hurts and back hurt because your desk and chair are not ergonomically set and you have the habit of sitting down in weird positions. Go lie down.
  25. Go back to work but immediately pick up a personal phone call from someone asking how everyone is getting on.
  26. Go back to work, it’s time to turn off the laptop but since you’ve done nothing and you are at home you can make up staying a bit longer…
  27. It’s 9pm. Realize you’ve done nothing really useful today, but tomorrow… tomorrow you’ll be super productive! All you need is to do is to set up some daily milestones and read one more blog about how to work from home.

I hope some of these made you laugh a little and was worth your quarantine’s time. These days I am working on a side project, but that’s…well..another story.

Test architecture in a nutshell

Picture from ulleo / 3990 images – a plant in a nutshell, literally

In my career I’ve gotten a lot of questions about test architecture: How to start testing when nothing has been done for years? How many test projects should there be? How many different type of testing do we need? Who should be in charge of testing in percentages? How to escalate testing? How to deal with big applications? How to deal with existing tests that are incomprehensible? What tools should we use? How many browsers should we test (are there more than two!?)?

First things first: Each application is different, each company is structured in a different way and with different resources and budget and each one has a different maturity state (which does not mean they should aspire to go to the next state, maybe they are right were they should be). There is not a ‘one size fits all’ solution and that’s why I recommend you check it with an expert. But here are some of my previous experiences, in case you find them useful.

On getting started

The first things I ask when I start a conversation about starting testing for a company are:

  1. What process are they following? (Agile, waterfall…)
  2. What technology are you using? (For development, any existing tests, code sharing, code control, task management…)
  3. What’s the relationship between the developers and testers? Do they sit together? Share code? Do common code reviews?
  4. What size is the company?
  5. Do developers write unit tests?
  6. How many people are dedicated (or plan to be) exclusively to tests?

Do they ring a bell? Here are some of the questions I suggested you could ask your interviewer if you are interviewing for a job. Of course, depending on the answers to these questions, I would suggest a different set of tools and practices (or if I am looking for a job, I could tell if I would be able to grow professionally and help the company given the circumstances and what to expect of the position)

Some more that are truly valuable are:

  1. What does your app or company do?
  2. How many clients do you have/plan to have?
  3. What’s the current vs potential growth?
Image from StartupStockPhotos / 121 images – all architecture need a study of the company first

In my experience, when someone asks for help getting started with testing, there are three status the company could be at, and therefore three things might be needed:

  1. Convincing upper management of the need to test: Sometimes you need to be able to explain to high management of the need for testing so you can get started with it. Their main fears are usually time and budget. Addressing those fears are not always easy and depend on each case, but the most important question is “Why do we test?” I leave this one for you for now, but if you are able to explain it, this point would be very easy to deal with.
  2. Convincing other team parties: Other times, they know testing is needed but they don’t know where to start (hint: unit test and TDD) When you don’t have a test team in place yet, I usually suggest the developers start getting into test. Then it would be nice to have someone with test experience to orchestrate these tests, to educate the team about testing and to start creating a test system for end to end and other sort of testing, or even doing manual checks. However, from when you get someone to when you have it all up and running, it might take a while, that’s why it is good that the developers start spending time in unit testing, as ideally they would keep doing this afterwards anyways.
  3. Getting a full framework up and running: In this case, the company have some unit testing and/or manual testing but they need help creating an automation framework and identifying other testing needs. Maybe they are also feeling insecure about the current tests and want to make sure they are not missing anything or want to improve their system (see the next two points).

On keeping it going

Sometimes people are struggling with their current test systems. They might be wondering if they are doing their best as it is, or if the tests they have provide actual value. The questions you need to ask yourself are:

  1. Is there any repetition that could be reduced? (how many current tests and what type of tests do you have? If they run over 15 minutes, this should likely be improved)
  2. Is what we are testing really important? Sometimes we test things that don’t really provide any value but it’s taking time and budget to implement and keep.
  3. Are there any tests that are not up to us? For example, downloading something from a browser, unless your app IS a browser, it’s probably something you should not test.
  4. Are we executing out of date tests? Keeping the system clean is highly recommended, if a feature comes back, you should be able to retrieve those tests too.
  5. Could we use any tools for tracking the issues to understand where we need or don’t need tests?
  6. Do we have the right tests for the right moment? Integrating tests with CI/CD in different branches means you need to decide what to run and check and when.

On escalating

Horizontal scaling means that you add more machines to your testing. For example using the cloud to test in different systems (more web browsers, mobile vs machine, different OS…)

Vertical scaling means that you scale by adding more type of tests (security, accessibility, performance…)

Identifying the type of tests you need and what systems should be covered would be part of an architecture design.

From Free-Photos / 9091 images – Don’t forget to have a conversation with the team before and after designing the test architecture. Everyone should be on board

On moving on

You’ve set up one or more test frameworks. Your team (developers/SDETS…) are managing the new tests which are carefully planned and being executed as part of the CI/CD structure. The analytics are set up and alerting of any problems or improvements… Could you ever be done with testing?

Most people in quality would straight away jump to the easy answer: “you are never done with it”. However, I feel like bringing up the argument: I believe sometimes you are. The same way as you might be done with development too. You will never be 100% happy about it, but sometimes you’ve done your best and maintenance it’s all there is left, at least for that project.

What do you do when you are developing and you are done with that project? You move on to the next. Of course, someone should stay to fix potential issues and provide support, but you wont need such a big of a team or the same sort of skills for this. I argue here, that the same could be said for testing. Development and testing come hand to hand, when there are less changes in development there would be less changes in testing.

Does that mean nobody should look into it anymore? No, that means you need someone with other set of skills to do that. Not worse or better, just different. Some people enjoy being at the beginning of the projects (I love doing research and creating new stuff) but other people prefer maintaining them. And this does not need to be after all it’s escalated, maybe your company is in the place it should be, and investing in more testing might not be necessary, even if you have the budget for it. The problem is, of course, discerning when that is the case, for which I highly recommend talking to a professional.

From JACLOU-DL / 5460 images – look at this cat: this cat is moving on, why can’t you?

On hiring the right test manager/architect

As I mentioned before, having a test expert helping figuring out the maturity of the company and what is needed to improve the quality of the product is very important. What do we call this expert? Test manager and test architect should not be the same position, although, sometimes companies use the descriptions a bit freely.

Test manager: A test manager makes sure the tests are done correctly (being them from test or dev team) and good practices are implemented. The position requires deep knowledge about testing and people skills.

Test architect: The architect designs the frameworks and tools to be used by testing. The position requires deep technical knowledge and experience planning and coding things from the start (also deep knowledge about testing).

Not all companies need both of the positions, and sometimes both jobs would be done by the same person. But being called whatever it’s called, someone has to make sure the system to do the testing is in place and someone has to make sure there are good practices and the right people understand how and what to look for in the quality area.

Therefore, in order to look for the right skills, you need to understand what you need. I have been asked many times for help defining test roles (I talk about the roles I’ve been in here), interviews and even salaries. In fact, there was a time I had to tell my interviewers how to interview me (and, funny story, somehow I ended up not getting the job!) But that’s, well, another story…

Making friends with the impostors inside

Disclaimer: These are only my experiences and this should not at any case be taken as a substitute for any sort of medical advice.

Photo by fotografierende from Pexels

First things first: It is extremely difficult for me to talk about feeling like an impostor. I’ve read about the power of words and how when you say “don’t think about a red hammer”, what do you imagine? A red hammer. So, if anybody is to say: “I feel like an impostor”, whoever is listening is going to imagine you as one. I’ve read about this syndrome before and I couldn’t help myself from thinking: “hmm… if you are feeling like one… maybe you are?” And maybe I am, let me explain…

When people talk about the “impostor’s syndrome” they usually talk about feeling like an impostor. I think it’s more a feeling of not being good enough or feeling different than the others, or both, and that we have multiple of these “impostors” inside.

So, yes, I’ve felt “not good enough” and I’ve felt “not fitting the mold”, and I was probably right every time: I wasn’t good enough for my standards and to myself and I was different from the rest, but everyone is unique in some form.

An impostor on Test

When you are an SDET (Software developer engineer in Test) you definitely are not fitting the mold: Developers tend to shy away from these positions and testers need development skills for them. I remember trying to convince people applying for Microsoft to join the SDET teams instead of SDE teams, because they felt they would be paid less or have less importance.

Photo by mentatdgt from Pexels (not necessarily an impostor)

Microsoft decided to have a separate team for making tools and tasks related to testing (as explained in this book, at least the version I read years ago). This should work out the same way as there are specialized developers for front end or back end. Many other companies followed this approach. But there are so many different companies that work in different structures and call different positions different names, that sometimes concepts get mixed up. And when you are a SDET you get offered all sort of positions, and usually, yes, less salary than a SDE. After many years, Microsoft decided to move everyone to Software Engineers and split the tasks evenly between the teams (which I discussed on previous articles talking about test roles and the future of testing).

On a different company, I’ve worked with SDE’s that have only worked with manual QA’s before, so they could not understand that I was a developer too. These are some of the things I had to hear:

  • “You are lucky, SDETs now are being paid almost as much as SDEs” – I’m soooo relieved that I won’t have to decide between selling my computer science degrees or moving to a position that takes away one letter from my title
  • “I am not doubting you have used it but… you can’t know the same about X technology that someone that uses it daily” – Do you REALLY use it daily? I heard tons of front end engineers talking about back end stuff and nobody is doubting their knowledge based on their lack of daily use
  • “Testing is not part of a developer’s job” – Not sure if you saying this is a bad or a good thing…
  • “We would not mind you to be in our code reviews, but… it would slow us down, better to restrict them to only two people (dev architect and dev lead) reviewing the entire team” – Ignoring the bottle neck you are creating, finding issues during code review would make the process FASTER… and if it is slowing it down you should definitely worry about the quality of the code being written
  • “It’s great that you checked the developers unit tests in your previous company.. But, were they so bad that they needed a tester checking that?” – What they were was actually good team players that split the work evenly and accepted advise
  • “It’s nice of you to be interested on being added to the design meeting, but it is really technical, you will get bored” – You know what? Just take notes afterwards, so I don’t bother you with my “lack of knowledge” (and I save myself from another meeting in which you are discussing technicalities about implementation to assess dominance)
  • “So..your university…was it technical? I mean, did you do technical stuff or was it theoretical but was giving out technical degrees?” – Well, to be honest, I was once walking down the street and I found a degree title with my name on it…(rolling eyes, the average of time for people to finish my degree in my university was twice the expected)
  • “Well, but there are different things in development, for example, design patterns…” – OOh design patterns, really? it sounds so fancy!! Please keep talking… I learnt this stuff on the second year of university: they are just repetitive ways of doing something that people with experience realized about and shared so other people with less experience can brag about… sorry use them too.
From knowyourmeme.com (my face on the last comment)

Of course, at the time, I didn’t think quickly enough these responses. I mostly stared blankly at the person without quite believing what I was hearing. It takes a while to build on responses, but what good would it have made anyways? Sometimes, people just talk because they like hearing their own voices. In Plato’s words “Wise men speak because they have something to say; Fools because they have to say something”.

Sometimes, it does not really matter what you can tell them, they will be stuck on their own beliefs and versions. Sometimes people need to feel, to see, to experience… in order to change their minds. And I believe I did my job with that, since at some point they started coming to me for advise and trusted me much more. But it took a while.

A quick note on degrees: I think having a degree does not necessarily give you better skills. I think degrees are good for having the bases and guidelines. But, no, you don’t need to have a computer science degree to be an SDET, but neither you do for an SDE position (I’ve known plenty of SDE without any sort of certification). However, you have to be really applied and be able to demonstrate skills to do the job; having a degree, experience or some projects you can show are some ways for proving this.

How do I deal with my impostors

Well, as the title of this post states it: I made friends with my impostor’s feelings. I have not overcome them. They are always here, part of me, waiting to get my attention, but I now acknowledge them and learn from them.

Picture by DigiPD from Pixabay (an impostor kitty)

I may not be taking the right steps but here is what I have told my impostor in the past :

  1. Use it on your favor. People that underestimate you are the ones that get most surprised when you get to show your skills, if you feel this is the case, maybe it is a good thing.
  2. Let people be wrong. Haters gonna hate, you don’t need to proof yourself to anybody but you. This is really difficult sometimes, but I learnt to care a bit less about it.
  3. Be kinder to yourself. Most of the times, this is all in your head.
  4. Use it to improve, learn more, grow… I think feeling you are not good enough is a great incentive to get better. Think of the words of Mark Twain: “The secret of getting ahead is getting started.”
  5. Be kinder to others. When someone makes you feel bad you realize you might be making others feel bad too, it can make you more empathic. Use it to reflect about your own acts and also try to understand why the other person is acting like that towards you…maybe they are the ones with insecurities?
  6. Realize that nobody was born knowing everything. Even when they do know about one thing that does not make them automatically know everything else there is to know. A lot of people love flexing. A lot of people have actually not an idea of what they are doing. In the end, everyone is constantly learning, that’s what life is about.
From https://knowyourmeme.com/memes/i-have-no-idea-what-im-doing. I like to imagine this dog with the face of the people that seems incredibly talented or never wrong.

At times, the impostor can get louder, for example when a deadline is coming closer or when public speaking. It does not matter how many times I do this I still can get nervous and anxious about a talk.

A especially bad time for feeling as an impostor is when you get laid off, it does not matter the reason for it, you always feel that if you really were an “A player” it should not happen to you. Also interviewing… this is especially troublesome, as your value might be calculated based in a company’s needs and not your actually qualities: When you need a pencil, a pen will never be as good.

Finally, when you get to join a company the impostor might be calling out the expectations people could have of you.

Those times that the impostors are louder, are when a special effort is required, by listening, understanding and talking to them, but every time it gets a little bit easier. I personally have many other examples when I felt like other impostors, but those are…well…another story.

How to test a time machine

I have recently watched a video from PBS space time that got me thinking: if we were to have a time machine, how would we test it? I’ve seen a lot of “how would you test X” type of questions in interviews, but I don’t think I’ve ever seen this one before (I am not trying to give you ideas for interview questions!)

It’s not rocket science… it’s rocking testing science!

I couldn’t help but compare it firstly to a spaceflight, so I started wondering: how do they test spacecrafts? And what better one than Apollo 11 to start with? If only we had a time machine for getting its source code… Yes, I have been looking at Assembly code trying to make sense of potential testing routines, like this one… and… guess what I found there?

This section of the code is checking the Gimbal lock of the accelerometers! Do you remember the concept from my last post? Maybe I just have a case of Baader-Meinhof, but I do feel Gimbal lock is an important concept to learn, so check it out.

Testing in Assembly was not as ‘easy’ as nowadays (for example, macros does not seem to be a thing that I could find in the Apollo11 programming syntax). Do not expect a page object model or a library with tests or testing functions. Nor common methods for before and after tests. Actually, don’t expect any sort of OOP, to start with.

In my search I could find some files with tests on them, but they are mostly for stressing the hardware by sending signals to the different devices and recovering from bad statuses. Also, spacecrafts might need to check correctness of bits to make sure there are no catastrophic arithmetic errors.

from @wikipedia

Time traveling tests

Imagine we have covered the unit and integration tests for both hardware and most of the software for our machine that could potentially match any currently existing ones. What are the specific cases we should cover for our time machine? The logical cases to think of:

1) The machine should not kill the traveler: We should insert some devices to measure that the cabin of the machine is livable. Also, keep in mind for the rest of the tests, that usually spaceships are first tested with robots inside, rather than humans. (Security test)

2) There must be some ways of safe interrupting the process (risk assessment/testing): In case anything wrong happens, what’s the risk involved for the passenger? (Security test)

3) As the machine goes forward or backwards, the traveler state stays while the surrounding world changes: We could measure this by introducing some other object before that we know it decays and checking how long time it has passed in the machine. (Usability test)

4) There is a way to return: I assume we will want this, so we should test it. (Security test?)

5) Performance tests: how many times it can go backwards and forward in time? (Performance but also, regression test?)

6) What is the minimum time that can travel? (test it) If the time machine requires a rapid negative speed (as the video above suggested as one of the possibilities), I imagine traveling in time as playing billiard or golf. Let me explain: When you try to position a ball on a particular hole, you need to give it the right original impulse (not too much, not too little) but also the right angle depending on the starting point. We are likely to travel space as well as time, so it might be particularly difficult to stop on a particular moment or place. (Which may explain why nobody made it to the time travelers party) (Usability test)

Photo by Tomaz Barcellos from Pexels

7) Test boundaries scenarios: go back to start of the earth (or universe) and forwards to its end. What happens if just before or just after? Does time always exist? Can we travel to a point there is no time? Technically these should be tests we would like to do, but I think they are probably not doable, realistically speaking (can I use this expression in this context?)

8) Try to change a small thing in the past…does it change the future? If a change makes a parallel universe, then we should not ever recover the machine. If we test this, we should look for an event we can easily undo (like maybe turn a light on/off?) (Exploratory test)

9) Time traveler meeting same time traveler. Test paradoxes. (Exploratory test)

11) Test placing a box where the machine is positioning before a trip. Then take the box away, position the machine on that same spot and test traveling to before we positioned the box. Does the box o machine break? (Integration test)

12) Test traveling to when the box is still in place; do the machine or the box break? (Integration test)

13) Try to travel twice to the same exact time and place. (Integration test)

14) Could the machine travel between different universes? (If we have a way of doing so, which might be more likely if we are using wormholes for the trips than negative speed) (Integration test)

From futurism.com (and an interesting article)

15) Is there a maximum/minimum size of the machine? If so, test them. (Boundary test)

16) Can everybody use the machine or only qualified people? (Accessibility test)


Maybe we have already invented a time machine but kept it secret because it is dangerous. Or maybe we have gone back in time to stop ourselves from inventing it, creating a parallel universe and as a result, our universe is the one in which such machine was never invented (time version of the Fermi paradox)

Whichever the answers to these questions, I hope you’ve found this exercise fun and enjoyed reading this post as much as I enjoyed writing it. Let me know if you can come up with more tests we could do to the time machine (if we were to have one). I will resume my serious posts soon, they will be… well… another story.

How do I get into…? guide

One of the most common question I get when people reach out to me about virtual reality (VR) is: how could I get started? Even thought I already have written an article about this, but maybe I should talk about my own experience instead so you can get more specific examples about it. If you are not into VR, please bear with me, as what I’m about to tell you can help you getting into any topics you wish to get into.

My Story

My first experience with a VR device was during a hackathon in Microsoft, when one of the interns brought his Oculus Rift. Back then it was a very expensive device, so it was very interesting to be able to play around with one. But I found that it would still have some issues to solve, starting for adding hands gestures.

As life goes, sometimes you get stuck in what you are doing at work and don’t get the time to investigate interesting new things. In my case, I bought a house and there was a lot of stress related to this. It was not until years later that I actually got the chance to try again another device, this time it was over mobile on a meetup called “tech for good” in Dublin. In this meetup, they were using VR mobile devices to provide social impact. It was my first experience with phone VR and I thought: Ok, now this is something that anybody can use and get, therefore it is something that is going to need testing.

After that, another hackathon (this time an open Nasa hackathon) got my interest in VR and AR back. I highly recommend this hackathon as I made really good friends there and we had so much fun building a AR/VR experience to navigate a satellite. My team (who won the local People’s choice award in Dublin) created an application that simulate a satellite around the orbit (on AR) and translate to see the view from that satellite (VR). If you are interested, here is our project

When I found myself having more time, I started looking for information about VR. I found a Udacity course on VR and decided to take it on. Back when I started it, the course covered many topics, although they made the decision of separating the courses in different specialties, which makes much more sense. If you are interested in seeing some of the projects I made during this course, check my Github account.

After that, I got interested in open source projects on AR and wanted to start doing testing there… However, life got in the way again when I moved to China. It’s still on my to-do list.

I was lucky enough to start working for Netease Games in China right after, so I had then enough flexibility and hardware access to do some more research in VR including some automated testing with Google Cardboard, which it should be now integrated in Airtest project (I know, not many people are using Google Cardboard anymore but, hey, you need to start somewhere.. the other research is still ongoing)

I also was lucky to have the opportunity to attend to the second Sonar in Hong Kong, which is a music and technology festival, and it showcased some cool new technologies and devices in VR (including aroma experiences and snow-surfing)

Besides that, I started to think of plans and ways of testing VR applications too (as Netease was working in some projects like nostos, which I had the opportunity to try myself and really enjoyed it).

Around that time, I gave a talk in Selenium conference in India gathering all this gained knowledge (which I talked about on this post). In order to prepare for this talk I played around and created my own ‘conference simulator’ just to get prepared for it.

Another thing I do frequently to gain knowledge in VR is to watch playthroughs and online reviews, as you can learn a lot from watching others play and it could be very good to understand your potential users if you are working on a game. I also have read some books on the matter (shout out to packtpub which gives away free IT books everyday!)

Have you found a pattern?

I know you have, if you are a reader of this blog you surely are a clever Lynx by now, but just in case, I have highlighted it in bold letters: Attending to (and after a while starting) hackathons, meetups, talks and festivals, watching or reading online related content and books, and playing around in courses, open source projects, at work and on your own projects will get you into anything you are interested to get into.

It sure sounds like a lot of things to do, but the tools are already around you, and I’m talking about years worth of experience here. Just take one thing at a time and you will too become an expert of that thing you are into. The biggest difficulty is to pick which one to take at any given time and to be honest about yourself and how much time you can spend on that. (I regret a ton not having put more efforts on the AR open source project when I had the chance)

Of course, if you are not really into it, then it would sound like a lot of work, in which case it’s probably better to save yourself time and pick something else. I like to think of it as a test of passion, or on the words of ‘Randy Pausch’ from his talk ‘Achieving Your Childhood Dreams’: “brick walls”. (By the way, this is one of the best motivational talks I’ve ever watch, and I actually re-watch it at least once a year to keep me motivated. Also, it mentions VR too 🙂 )

As you would imagine, this is not the only subject I spent time with or I gave my attention to, another big one for me is artificial intelligence, but that’s…well…another story.

Hacking social media

I know, I still owe you some stories, but I am now inspired to talk about something else. Besides, today’s issue is easier to put it into words. I don’t need to sit down and think carefully on a way of explaining some technical concept such as artificial intelligence while not sounding boring. But, just so you know, I am still working on the other stories.

I would like to show you how dangerous social media could become and on one hand highlight the need of asking the right questions when a new technology comes along in order to set proper tests and barriers (lynx are curious animals, aren’t we?).

On the other hand, highlight the importance to take breaks from it and think about yourself and things that would make YOU happy instead of thinking of things that ‘would make other people think that you are happy’ and therefore approve of you. I hope you enjoy reading this article.

Let’s think about it: the most viewed YouTube channels from independent creators are from people below their 30’s (or just on them), many started them 5-9 years ago. They have been getting a lot of pressure from fans and companies that would like a piece of their influence. Celebrities with less direct exposure to their fans have done crazy things in the past because of social pressure. Yet these influencers are not invited (that I know of) to Davos or famous lists of most influential people, besides demonstrating incredible marketing strategies, knowledge of new technologies, having charisma and being very intelligent (more than they let to be seen in some cases)

Social media affects society, not only these influencers, but many people are actually feeling depress or harm themselves because of social media. It is also a potential source for propaganda of all types and an a source for advertisement of all sorts by using the platforms ‘algorithms‘ in their favor.

There is a very important point to consider, which is its the potential for hacking (if you are interested on this, there is more information here and here). So, imagine that someone could actually go there and decide what you are going to see… how could this affect you?

Techniques and prevention:

Let’s imagine a platform that accepts comments and likes (let’s forget about dislikes). How could someone socially hack it?

1) Removing the likes: We would need to intercede the information that this platform is showing to the user and eliminate the likes the user will see. Maybe we should only eliminate them partially, so the user is not suspicious of not seeing any likes at all. For this, we could have a pondered random variable that would eliminate or not eliminate each like. How would you feel if all of the sudden nothing you write gets any likes? Prevention: From test side, make sure the like system works properly and cannot be done by anonymous sources. Make sure accounts are real. Make sure the user sees only real data. From user perspective, when you see something that you like, mention it in person, start a conversation about it instead of just clicking a button.

2) Liking specific posts: This is a bit fancier. Based on above, we could have some sort of AI algorithm that could classify the posts. Then we can decide which comments are going to show as liked for the user. How would you feel if only some type of your posts would get liked? Would that change your way of writing? Prevention: From test side, make sure all information is shown to the user. From user side, find your audience and focus on them. Also, consider talking with this people directly too (or in conferences). Try to keep honest to your goals for writing and who you want to reach.

3) Filtering comments: This would require some form of classification as with the previous point. Instead of targeting the likes, we would target the comments but the idea would be the same. Eliminate from view those that are not ‘interesting’. What would you think if you only receive a certain types of comments? Prevention: From test side, make sure all information is shown to the user. Maybe have a conversation about the feature itself and allow users to hide all comments. From user side, as above.

4) Creating comments: We could create new comments with AI. You might think the user would realize about this, but if done carefully they might not even notice this or confront the person making that comment. Besides, the social media platform might allow for not-logged in comments. This adds to the feeling of the previous one. Prevention: Have a conversation about blocking anonymous comments or disabling them. From user side, if you see a strange comment from someone that does not add up, clarify this with the person. It can also help with misunderstandings. Option 2, disable or stop reading comments.

5) Changing the advertisement around the website to a convenient one for propaganda or for harm (only for some types of social media). Prevention: Most of sites have a way of deciding what advertisements you are more interested of. Also, try cleaning cookies regularly or use private browsers and VPN services.

6) Extracting automatically interesting information for malicious purposes. Prevention: Be careful with what you post, don’t use information that is available for anybody as your passwords or security questions or pictures with personal data (such passports or train tickets). If you really want to share pictures of a trip, try uploading them after the trip and enjoy while in there!

7) Connecting certain types of people. I am not 100% sure how this could be used for malicious purposes but surely someone would find a way. Making sure you can block people is also very important.

8) Taking things out of context. Prevention: It’s very hard to delete something from the internet once it is out there, but some platforms allow it. Have you ever read your old posts? It is a good idea to do some clean-up every so often. Also, if this happens to you, keep a track of the entire context. Maybe have a system in which you can remove what you have written before it goes online, take some hours before posting to make sure you want to post that.

Why am I talking about social hacking social media in a test blog? Well, because, if you happen to be working on developing a social media project, you should make sure that these attacks are not possible and think about how the user could feel for other features to come.

(Please take some time to go throughout the articles I liked above to know more)

Thoughts about being addicted to being connected:

When was the last time you did something good for someone but did not tell anybody about it? When you do something good for someone and post about it, how do you know you are doing it because of the other person and not to have a better image of yourself in front of others?

When was the last time that you went for a trip and didn’t share the pictures with anybody? What about not taking any pictures at all? There is something truly special about having a memory that is just yours to have.


Social media has evolved quickly in very short time, and we need to consider a lot of new things, more so if we are in the development team of one of these platforms. We should really stop and agree about what is ethical and not with the particular platforms and maybe even list the set of contraindications as if we do with addictive substances. For example, would it be ethical for some platforms (maybe for young people?) to change what the users see in order to protect them from the bad critic? Consider this could, in theory, save some lives, but, in contrast, it would take away some potential good feedback disguised as bad comments. Maybe this is a feature you want people to turn on and off? If not, maybe you should list in your contraindications that it could be an issue. And I don’t mean terms and conditions, it’s not about saving your behind if anything happens, it’s about actually alerting the user about what could be experienced. Terms and conditions are…well.. another story.


Some sources if you are interested in this area that helped me control my internet usage:

Book: “How to break up with your phone” by Catherine Price.

Watch: Crash courses on navigating digital information