First things first: happy new year lynx! I wish to start this year with a somehow sensitive topic, and I want to apologise in advance if I say anything wrong, mind that it is said from my heart and my current knowledge.
According to “Biased” by Jennifer Eberhardt: “Categorization -grouping like things together – is not some abhorrent feature of the human brain, a process that some people engage in and other do not. Rather, it is a universal function of the brain that allows us to organize and manage the overload of stimuli that constantly bombard us”.
In testing, we also have categories that help us learn and understand the types of tests, such as functional vs non-functional. However, white vs black box categorisation is, in my honest opinion, inaccurate and inadequate. It is also the one that’s being more widely used, particularly for beginners. I believe we should stop using these terms, and in this article I will give my reasoning and a better alternative.
Note: For clarity purposes I will still use these terms until the end of the article.
You might think that white box tests are not better or worse than black box tests and therefore, it is OK to use those names. However, the idea behind it is that with white box testing you can see the code whilst you can’t with black box testing. With this categorisation we are promoting the stereotype that white is something clear, usually associated with purity, while black is opaque, usually associated with immorality.
If you really think about it, you would agree that this definitions make no sense.
Why can “you see the code” with white box test? You shouldn’t be able to see the contents of a white box at all. Consider this: most ceilings are painted white, can you see the sky or your neighbors throughout your ceilings? How can we normalize the fact that a white box is somehow see-through?
Unit tests category chameleon
If I were to ask you in which category unit tests fall into, I bet most of you would say white box, because “you can see the code.” However, if you think in terms of TDD, you would write the tests BEFORE writing the code and therefore you are not seeing it, which by definition shouldn’t technically be white box.
What’s the actual difference between unit tests, positioned at the base of the testing pyramid, and the other tests?
The person who writes them writes also the code? But that could be said of any of the others, especially when shifting left.
The biggest difference is that these look for code structure rather than feature logic. For example, you don’t care about the number of products shown in the shopping cart, but that this particular piece of code can handle the right exceptions and have the right outputs.
Switching to clear vs opaque
In my opinion, just replacing the wording here, although it’s a step, it would not be enough. Whilst it is true that it would be a fast solution to avoid prolonging stereotypes and keep our systems and classifications intact and ourselves a bit more comfortable, I feel the stereotype would somehow still hold: people will first think white vs black and then change to these other names.
Furthermore, I still have more issues with this categorisation, which I’m going to continue arguing before giving what I think it’s the best alternative.
Grey box testing
You can tell when a category doesn’t make sense when it needs a lot of exceptions and an extra group to gather anything that can’t be placed in the others. Introducing grey box testing: when you… partially? sometimes?… see the code.
As before, grey is another colour, if you use it in a box, you won’t see it’s interior. Not even partially. What should we use instead? Scratched box? Partially opaque box? It does not quite make sense. Which tests exactly would land in this category?
Test/QA people rarely work with white box
There are a lot of concepts behind white box testing that are really interesting but are usually incorporated in automatic code checking tools and rarely used daily by tests/QA people.
Cyclomatic complexity is rarely calculated manually, although sometimes it’s called out in code reviews. Most times this is part of code optimisation or happens behind other methods.
Statement testing, edge testing, branch testing, loop testing, Branch and Relational Operator testing and data flow testing are methods that are rarely planned, examined and automated in a frequent manner. Many of these terms are still unknown or unused by others on our industry today.
Most of the rest of the test pyramid have to do with what’s currently called black (or grey) box testing.
Structural vs logical
I think a clearer classification could be structural vs logical. In fact, in order to document myself for this article, I have been recently reviewing my notes from university and in them, it is said that these are alternative names these test categories are known for. Yet people rarely refer to them. Why not using them, when they are more suitable and less damaging?
The definition would be whether you are testing the structure of the code (number of branches, proper exception handling…) or the logic behind the feature (the shopping cart should have 0 products after a purchase).
What about back-end testing / integration testing or database testing? Exactly the same: if you are testing the database for structure, then that would be structural, if you are testing it for logic, logical. We can reach more granularity with this definition, although we usually tend to test these for logic rather than structure.
This classification opens up to think about more tools to validate code. It also leaves unit test as a structural test with no room for confusion or need for extra categories.
I hope you understand that my point in this article is not to remove a category, but to enclose it in more appropriate terms and definition. If leaders such as Github have started a major shift in their branches to remove old coding terms, why can we not do this when we already have more appropriate names?
I really hope this post get to places such as ISTQB® and we start changing our terminology and explanations, as so many people look to them as the standard in the testing area.
As Miguel Ruiz said in “The four agreements” you should “be impeccable with your word”. Wouldn’t it be a great start of the year if we apply it to this test categorisation, using structural and logical instead of white box and black box?
Do you agree? Disagree? Let me know in the comments below. I could tell you more about structural testing and give you ideas to improve your coding with it, but that’s… well…another story.
In many companies where development and testing are two separate efforts, I’ve noticed some testers trust too much in the developers. Even in companies where development is highly integrated with testing, this could result in missing tests, some of which could be crucial for finding important bugs in the app under development.
One of the clear examples of this is when it comes to CI/CD. Usually, an admin or devops set this up. The involvement of a test expert in this area is particularly important, as they have the skills to plan the right automated tests to validate the builds. I have seen pipelines where there is no testing at all, or a group of insufficient or irrelevant tests.
For some people in the quality team, the developers are their “source of truth”. I’ve fallen into this same error, with other colleagues with more years of experience than me: Whatever they say should be the right way – they know better. Comparing your knowledge to other’s in terms of “more-less” rather than “different” is a recipe for disaster…
Let me tell you the story of a time I trusted too much in an experienced dev: there was an obvious bug in staging environment. I’ve logged it. Someone else logged a duplicated. We were late for release. I said I couldn’t sign up with that bug there.
The developer assigned to the feature, called the PM and me to his office and both tried to reason me to sign up. I explicitly called out the risk. He said the bug was unlikely to be hit and he didn’t have time to fix it. I trusted him – I was only a junior while he was senior and longer in the company. My instincts told me that the seniority did not make him a fortune teller, but even so I said: “OK, I’ll sign up as long as you get my back if anything happens.” He reassured me and agreed, he said he would take all the responsibility if anything was to happen. Nothing was written down about it.
When users hit the issue in production, both PM and developer immediately blamed me. By writing. Both of them. My manager blamed me. They all said I missed finding a clear bug. When I explained everyone what happened, providing logs of the defect, which was still in open state and the duplicated, where everyone involved was informed. My managed called me to his office. He told me I shouldn’t have signed up and my behavior was not professional as we “shouldn’t point fingers to others”. In other words, I should have taken the blame the others put on me without even trying to clarify it. He didn’t listen to me and it affected my performance review, which was just a week later.
He told me I got two really bad “anonymous” reviews (hard to guess who did them, right?) and the rest, whilst good, were irrelevant.
At the start of that same review period, I found a fix to a very important defect on another team that was costing the company a lot of money and the manager of that team said that if a developer fixed it, they would be immediately promoted. I was the one that provided the fix. It did not seem to matter.
Even the worse of the situations can become a learning, whether it happened to you or someone else. What I learnt from this one:
Follow your instincts, if you think something is important stick with it. Avoid trusting blindly other team members based on their experience or how well you get on with them. Know your value and what you can give to the company: I decided to trust a dev based solely on his experience after having figured out an issue that many other experienced devs could not figure out.
Getting to an agreement is good, but it is better if you do so by writing, so everyone can remember and review it later on.
If you need to release a feature with a defect, that is OK, but plan for the worst and think about a course of action to assure the least disturbance to the users if they hit it.
Finally, if you get a manager that does not support you, consider moving away. Leaders and managers should be there to help you grow and learn, to challenge you and inspire you, never to belittle you. You should always feel they make an effort to understand and protect you, at least, that’s what I try to do when I lead/coach others.
I hope this story I can help you out if you find yourself in a similar situation. Remember that every obstacle can become a learning.
As a final note: having trust is very important, given the right dialogues and plans of action. The opposite (not trusting the developers) is also not ideal. When there is not enough trust, walls are created between teams and things tend to get tested multiple times, but that’s well..another story.
As part of my talk in Heisenbug, I showed two examples of automation in virtual reality: Unium and Poco(Airtest). I feel the explanations weren’t very deep, so I decided to use this format to expand on them.
1) Preparing the APP to test
In the examples below I’m going to be using a VR app that I created as part of the VR foundations Udacity course. This app was done using Google VR Cardboard(which is currently sort of deprecated, but I believe it’s the best first step into VR development)
You can find the code here. It was build with Unity version 2017.1.0p4 and Google Cardboard version gvr-unity-sdk-1.0.3. I’m specifying this because, sometimes Unity is funny when switching versions and things go deprecated fast in Google VR, so if you use other versions it might not work straight away.
This application is a maze game. The user has to find a key and then a house to unlock the door with it. There are also coins scattered around the maze and the goal is to find them. Inside the house, there would be a board with the number of found keys.
Unit testing the application with test runner we can check some things. For example, we can check that the door ‘lock’ variable is on or off. However, we don’t see the real result as the user would, which could cover issues such as the lock variable does not cause anything to happen in the scene or the sound or particle system are not pleasant or the user cannot reach the objects or they sizes feel wrong…
To test this, you would have to actually go around the maze, get all the coins, get the key, go back to the door and check everything this way. The path is always the same and manual testing could get very time consuming. That’s the reason of looking for solution for automation of user behaviour, as we would see with Unium.
2) Installing and configuring Unium
Although there is a Github project available, I recommend you do this from the asset store in Unity.
Then we search for Unium and we download the asset.
A folder will appear then in the Assets’s folder of the project. Then we add a new empty object in the Scene of the project and we add the script of the folder from our Assets. What this script does is opening a local web server, with port 8342 by default (this can be changed), that we can use to interact with the application, retrieving data from it and sending data to it. If you are curious about how this could work, check out my post about API’s. This creates communication capabilities between the server and our Unity program.
3) Using Unium
Now we can do calls directly in a browser as if we were accessing a website, such as:
This would set the camera to a rotation of 180 in the x axe, 0 in the y axe and 0 in the z axe.
Movement of the camera is crucial for VR apps, and we can see three issues already with these functions. The first one is that for VR applications we usually move by clicking objects called “waypoints” or with handset functions. With Unium we could use waypoints as objects and click them as such:
Note that I have all waypoints under another object “Waypoints”, the exact waypoint to click is called “Waypoint_7” and then the code that has the click method is called “Waypoint”. The call to click on the door the is easier because we only have one door object and we are not reusing names:
If you are not sure of the exact name of the object you can use wildcards and queries.
The second issue is the use of EulerAngles. I was able to see the rotation with quaternions (using “rotation” instead of “eulerAngles” at the end of the call) but I was not able to set it up for some reason (maybe now there is support for it or I was doing something wrong at the time of the testing)
The last issue is the rotation of the camera: users would create a movement when rotating towards a point or in a direction, but the rotation done with the call is done in one setting. This would not really emulate the user behaviour but with Unium we could set up a script and emulate such behaviour or use the web socket capabilities to run a query with a chosen frequency.
Unium has many scripting languages which can be used to automate these calls, for example nUnit (which is the one we would use in C# for unit test). Here is an example of use of code to do this:
Unium is a powerful tool for automation of Unity projects. VR automation is possible with it, even emulating user’s behaviour. However, this sort of automation is not fully black boxed, as you need to add this code into the application and possible use some methods that are existing in the application. Ideally, a fully black boxed automation would simulate the behaviour of the controllers directly.
What’s the difference between automating any game VS automating a VR Game? The main difference is that instead of a movement of a player, you need to automate the camera movement and take into account the handsets (if present).
Sometimes, the transform (walking) is usually done by clicking waypoints and this could be done easily with Unium. Rotation can be emulated by rotating the camera as demonstrated before.
The next step would be to to automate the handsets. However, if we do this directly by calling functions on the controllers of the handset, we might enter the territory of testing that the functions by the controller’s provider are working fine. Instead of this, we might just want to make sure that our functions work fine, so maybe we should call the functions that we have created to control our application when the controller selects and clicks around. But that’s… well… another story…
In my career I’ve gotten a lot of questions about test architecture: How to start testing when nothing has been done for years? How many test projects should there be? How many different type of testing do we need? Who should be in charge of testing in percentages? How to escalate testing? How to deal with big applications? How to deal with existing tests that are incomprehensible? What tools should we use? How many browsers should we test (are there more than two!?)?
First things first: Each application is different, each company is structured in a different way and with different resources and budget and each one has a different maturity state (which does not mean they should aspire to go to the next state, maybe they are right were they should be). There is not a ‘one size fits all’ solution and that’s why I recommend you check it with an expert. But here are some of my previous experiences, in case you find them useful.
On getting started
The first things I ask when I start a conversation about starting testing for a company are:
What process are they following? (Agile, waterfall…)
What technology are you using? (For development, any existing tests, code sharing, code control, task management…)
What’s the relationship between the developers and testers? Do they sit together? Share code? Do common code reviews?
What size is the company?
Do developers write unit tests?
How many people are dedicated (or plan to be) exclusively to tests?
Do they ring a bell? Here are some of the questions I suggested you could ask your interviewer if you are interviewing for a job. Of course, depending on the answers to these questions, I would suggest a different set of tools and practices (or if I am looking for a job, I could tell if I would be able to grow professionally and help the company given the circumstances and what to expect of the position)
Some more that are truly valuable are:
What does your app or company do?
How many clients do you have/plan to have?
What’s the current vs potential growth?
In my experience, when someone asks for help getting started with testing, there are three status the company could be at, and therefore three things might be needed:
Convincing upper management of the need to test: Sometimes you need to be able to explain to high management of the need for testing so you can get started with it. Their main fears are usually time and budget. Addressing those fears are not always easy and depend on each case, but the most important question is “Why do we test?” I leave this one for you for now, but if you are able to explain it, this point would be very easy to deal with.
Convincing other team parties: Other times, they know testing is needed but they don’t know where to start (hint: unit test and TDD) When you don’t have a test team in place yet, I usually suggest the developers start getting into test. Then it would be nice to have someone with test experience to orchestrate these tests, to educate the team about testing and to start creating a test system for end to end and other sort of testing, or even doing manual checks. However, from when you get someone to when you have it all up and running, it might take a while, that’s why it is good that the developers start spending time in unit testing, as ideally they would keep doing this afterwards anyways.
Getting a full framework up and running: In this case, the company have some unit testing and/or manual testing but they need help creating an automation framework and identifying other testing needs. Maybe they are also feeling insecure about the current tests and want to make sure they are not missing anything or want to improve their system (see the next two points).
On keeping it going
Sometimes people are struggling with their current test systems. They might be wondering if they are doing their best as it is, or if the tests they have provide actual value. The questions you need to ask yourself are:
Is there any repetition that could be reduced? (how many current tests and what type of tests do you have? If they run over 15 minutes, this should likely be improved)
Is what we are testing really important? Sometimes we test things that don’t really provide any value but it’s taking time and budget to implement and keep.
Are there any tests that are not up to us? For example, downloading something from a browser, unless your app IS a browser, it’s probably something you should not test.
Are we executing out of date tests? Keeping the system clean is highly recommended, if a feature comes back, you should be able to retrieve those tests too.
Could we use any tools for tracking the issues to understand where we need or don’t need tests?
Do we have the right tests for the right moment? Integrating tests with CI/CD in different branches means you need to decide what to run and check and when.
Horizontal scaling means that you add more machines to your testing. For example using the cloud to test in different systems (more web browsers, mobile vs machine, different OS…)
Vertical scaling means that you scale by adding more type of tests (security, accessibility, performance…)
Identifying the type of tests you need and what systems should be covered would be part of an architecture design.
On moving on
You’ve set up one or more test frameworks. Your team (developers/SDETS…) are managing the new tests which are carefully planned and being executed as part of the CI/CD structure. The analytics are set up and alerting of any problems or improvements… Could you ever be done with testing?
Most people in quality would straight away jump to the easy answer: “you are never done with it”. However, I feel like bringing up the argument: I believe sometimes you are. The same way as you might be done with development too. You will never be 100% happy about it, but sometimes you’ve done your best and maintenance it’s all there is left, at least for that project.
What do you do when you are developing and you are done with that project? You move on to the next. Of course, someone should stay to fix potential issues and provide support, but you wont need such a big of a team or the same sort of skills for this. I argue here, that the same could be said for testing. Development and testing come hand to hand, when there are less changes in development there would be less changes in testing.
Does that mean nobody should look into it anymore? No, that means you need someone with other set of skills to do that. Not worse or better, just different. Some people enjoy being at the beginning of the projects (I love doing research and creating new stuff) but other people prefer maintaining them. And this does not need to be after all it’s escalated, maybe your company is in the place it should be, and investing in more testing might not be necessary, even if you have the budget for it. The problem is, of course, discerning when that is the case, for which I highly recommend talking to a professional.
On hiring the right test manager/architect
As I mentioned before, having a test expert helping figuring out the maturity of the company and what is needed to improve the quality of the product is very important. What do we call this expert? Test manager and test architect should not be the same position, although, sometimes companies use the descriptions a bit freely.
Test manager: A test manager makes sure the tests are done correctly (being them from test or dev team) and good practices are implemented. The position requires deep knowledge about testing and people skills.
Test architect: The architect designs the frameworks and tools to be used by testing. The position requires deep technical knowledge and experience planning and coding things from the start (also deep knowledge about testing).
Not all companies need both of the positions, and sometimes both jobs would be done by the same person. But being called whatever it’s called, someone has to make sure the system to do the testing is in place and someone has to make sure there are good practices and the right people understand how and what to look for in the quality area.
Therefore, in order to look for the right skills, you need to understand what you need. I have been asked many times for help defining test roles (I talk about the roles I’ve been in here), interviews and even salaries. In fact, there was a time I had to tell my interviewers how to interview me (and, funny story, somehow I ended up not getting the job!) But that’s, well, another story…
Disclaimer: These are only my experiences and this should not at any case be taken as a substitute for any sort of medical advice.
First things first: It is extremely difficult for me to talk about feeling like an impostor. I’ve read about the power of words and how when you say “don’t think about a red hammer”, what do you imagine? A red hammer. So, if anybody is to say: “I feel like an impostor”, whoever is listening is going to imagine you as one. I’ve read about this syndrome before and I couldn’t help myself from thinking: “hmm… if you are feeling like one… maybe you are?” And maybe I am, let me explain…
When people talk about the “impostor’s syndrome” they usually talk about feeling like an impostor. I think it’s more a feeling of not being good enough or feeling different than the others, or both, and that we have multiple of these “impostors” inside.
So, yes, I’ve felt “not good enough” and I’ve felt “not fitting the mold”, and I was probably right every time: I wasn’t good enough for my standards and to myself and I was different from the rest, but everyone is unique in some form.
An impostor on Test
When you are an SDET (Software developer engineer in Test) you definitely are not fitting the mold: Developers tend to shy away from these positions and testers need development skills for them. I remember trying to convince people applying for Microsoft to join the SDET teams instead of SDE teams, because they felt they would be paid less or have less importance.
Microsoft decided to have a separate team for making tools and tasks related to testing (as explained in this book, at least the version I read years ago). This should work out the same way as there are specialized developers for front end or back end. Many other companies followed this approach. But there are so many different companies that work in different structures and call different positions different names, that sometimes concepts get mixed up. And when you are a SDET you get offered all sort of positions, and usually, yes, less salary than a SDE. After many years, Microsoft decided to move everyone to Software Engineers and split the tasks evenly between the teams (which I discussed on previous articles talking about test roles and the future of testing).
On a different company, I’ve worked with SDE’s that have only worked with manual QA’s before, so they could not understand that I was a developer too. These are some of the things I had to hear:
“You are lucky, SDETs now are being paid almost as much as SDEs” – I’m soooo relieved that I won’t have to decide between selling my computer science degrees or moving to a position that takes away one letter from my title
“I am not doubting you have used it but… you can’t know the same about X technology that someone that uses it daily” – Do you REALLY use it daily? I heard tons of front end engineers talking about back end stuff and nobody is doubting their knowledge based on their lack of daily use
“Testing is not part of a developer’s job” – Not sure if you saying this is a bad or a good thing…
“We would not mind you to be in our code reviews, but… it would slow us down, better to restrict them to only two people (dev architect and dev lead) reviewing the entire team” – Ignoring the bottle neck you are creating, finding issues during code review would make the process FASTER… and if it is slowing it down you should definitely worry about the quality of the code being written
“It’s great that you checked the developers unit tests in your previous company.. But, were they so bad that they needed a tester checking that?” – What they were was actually good team players that split the work evenly and accepted advise
“It’s nice of you to be interested on being added to the design meeting, but it is really technical, you will get bored” – You know what? Just take notes afterwards, so I don’t bother you with my “lack of knowledge” (and I save myself from another meeting in which you are discussing technicalities about implementation to assess dominance)
“So..your university…was it technical? I mean, did you do technical stuff or was it theoretical but was giving out technical degrees?” – Well, to be honest, I was once walking down the street and I found a degree title with my name on it…(rolling eyes, the average of time for people to finish my degree in my university was twice the expected)
“Well, but there are different things in development, for example, design patterns…” – OOh design patterns, really? it sounds so fancy!! Please keep talking… I learnt this stuff on the second year of university: they are just repetitive ways of doing something that people with experience realized about and shared so other people with less experience can brag about… sorry use them too.
Of course, at the time, I didn’t think quickly enough these responses. I mostly stared blankly at the person without quite believing what I was hearing. It takes a while to build on responses, but what good would it have made anyways? Sometimes, people just talk because they like hearing their own voices. In Plato’s words “Wise men speak because they have something to say; Fools because they have to say something”.
Sometimes, it does not really matter what you can tell them, they will be stuck on their own beliefs and versions. Sometimes people need to feel, to see, to experience… in order to change their minds. And I believe I did my job with that, since at some point they started coming to me for advise and trusted me much more. But it took a while.
A quick note on degrees: I think having a degree does not necessarily give you better skills. I think degrees are good for having the bases and guidelines. But, no, you don’t need to have a computer science degree to be an SDET, but neither you do for an SDE position (I’ve known plenty of SDE without any sort of certification). However, you have to be really applied and be able to demonstrate skills to do the job; having a degree, experience or some projects you can show are some ways for proving this.
How do I deal with my impostors
Well, as the title of this post states it: I made friends with my impostor’s feelings. I have not overcome them. They are always here, part of me, waiting to get my attention, but I now acknowledge them and learn from them.
I may not be taking the right steps but here is what I have told my impostor in the past :
Use it on your favor. People that underestimate you are the ones that get most surprised when you get to show your skills, if you feel this is the case, maybe it is a good thing.
Let people be wrong. Haters gonna hate, you don’t need to proof yourself to anybody but you. This is really difficult sometimes, but I learnt to care a bit less about it.
Be kinder to yourself. Most of the times, this is all in your head.
Use it to improve, learn more, grow… I think feeling you are not good enough is a great incentive to get better. Think of the words of Mark Twain: “The secret of getting ahead is getting started.”
Be kinder to others. When someone makes you feel bad you realize you might be making others feel bad too, it can make you more empathic. Use it to reflect about your own acts and also try to understand why the other person is acting like that towards you…maybe they are the ones with insecurities?
Realize that nobody was born knowing everything. Even when they do know about one thing that does not make them automatically know everything else there is to know. A lot of people love flexing. A lot of people have actually not an idea of what they are doing. In the end, everyone is constantly learning, that’s what life is about.
At times, the impostor can get louder, for example when a deadline is coming closer or when public speaking. It does not matter how many times I do this I still can get nervous and anxious about a talk.
A especially bad time for feeling as an impostor is when you get laid off, it does not matter the reason for it, you always feel that if you really were an “A player” it should not happen to you. Also interviewing… this is especially troublesome, as your value might be calculated based in a company’s needs and not your actually qualities: When you need a pencil, a pen will never be as good.
Finally, when you get to join a company the impostor might be calling out the expectations people could have of you.
Those times that the impostors are louder, are when a special effort is required, by listening, understanding and talking to them, but every time it gets a little bit easier. I personally have many other examples when I felt like other impostors, but those are…well…another story.
I have recently watched a video from PBS space time that got me thinking: if we were to have a time machine, how would we test it? I’ve seen a lot of “how would you test X” type of questions in interviews, but I don’t think I’ve ever seen this one before (I am not trying to give you ideas for interview questions!)
It’s not rocket science… it’s rocking testing science!
I couldn’t help but compare it firstly to a spaceflight, so I started wondering: how do they test spacecrafts? And what better one than Apollo 11 to start with? If only we had a time machine for getting its source code… Yes, I have been looking at Assembly code trying to make sense of potential testing routines, like this one… and… guess what I found there?
This section of the code is checking the Gimbal lock of the accelerometers! Do you remember the concept from my last post? Maybe I just have a case of Baader-Meinhof, but I do feel Gimbal lock is an important concept to learn, so check it out.
Testing in Assembly was not as ‘easy’ as nowadays (for example, macros does not seem to be a thing that I could find in the Apollo11 programming syntax). Do not expect a page object model or a library with tests or testing functions. Nor common methods for before and after tests. Actually, don’t expect any sort of OOP, to start with.
In my search I could find some files with tests on them, but they are mostly for stressing the hardware by sending signals to the different devices and recovering from bad statuses. Also, spacecrafts might need to check correctness of bits to make sure there are no catastrophic arithmetic errors.
Time traveling tests
Imagine we have covered the unit and integration tests for both hardware and most of the software for our machine that could potentially match any currently existing ones. What are the specific cases we should cover for our time machine? The logical cases to think of:
1) The machine should not kill the traveler: We should insert some devices to measure that the cabin of the machine is livable. Also, keep in mind for the rest of the tests, that usually spaceships are first tested with robots inside, rather than humans. (Security test)
2) There must be some ways of safe interrupting the process (risk assessment/testing): In case anything wrong happens, what’s the risk involved for the passenger? (Security test)
3) As the machine goes forward or backwards, the traveler state stays while the surrounding world changes: We could measure this by introducing some other object before that we know it decays and checking how long time it has passed in the machine. (Usability test)
4) There is a way to return: I assume we will want this, so we should test it. (Security test?)
5) Performance tests: how many times it can go backwards and forward in time? (Performance but also, regression test?)
6) What is the minimum time that can travel? (test it) If the time machine requires a rapid negative speed (as the video above suggested as one of the possibilities), I imagine traveling in time as playing billiard or golf. Let me explain: When you try to position a ball on a particular hole, you need to give it the right original impulse (not too much, not too little) but also the right angle depending on the starting point. We are likely to travel space as well as time, so it might be particularly difficult to stop on a particular moment or place. (Which may explain why nobody made it to the time travelers party) (Usability test)
7) Test boundaries scenarios: go back to start of the earth (or universe) and forwards to its end. What happens if just before or just after? Does time always exist? Can we travel to a point there is no time? Technically these should be tests we would like to do, but I think they are probably not doable, realistically speaking (can I use this expression in this context?)
8) Try to change a small thing in the past…does it change the future? If a change makes a parallel universe, then we should not ever recover the machine. If we test this, we should look for an event we can easily undo (like maybe turn a light on/off?) (Exploratory test)
9) Time traveler meeting same time traveler. Test paradoxes. (Exploratory test)
11) Test placing a box where the machine is positioning before a trip. Then take the box away, position the machine on that same spot and test traveling to before we positioned the box. Does the box o machine break? (Integration test)
12) Test traveling to when the box is still in place; do the machine or the box break? (Integration test)
13) Try to travel twice to the same exact time and place. (Integration test)
14) Could the machine travel between different universes? (If we have a way of doing so, which might be more likely if we are using wormholes for the trips than negative speed) (Integration test)
15) Is there a maximum/minimum size of the machine? If so, test them. (Boundary test)
16) Can everybody use the machine or only qualified people? (Accessibility test)
Maybe we have already invented a time machine but kept it secret because it is dangerous. Or maybe we have gone back in time to stop ourselves from inventing it, creating a parallel universe and as a result, our universe is the one in which such machine was never invented (time version of the Fermi paradox)
Whichever the answers to these questions, I hope you’ve found this exercise fun and enjoyed reading this post as much as I enjoyed writing it. Let me know if you can come up with more tests we could do to the time machine (if we were to have one). I will resume my serious posts soon, they will be… well… another story.
Previously, I’ve written a couple of posts about how to get yourself started on VR in which I promised some stories about testing on this world.
Why do I call it world instead of application? Because the goal of virtual reality is to create realistic synthetic worlds that we can inspect and interact with. There are different ways of interacting in these worlds depending on the device type.
Testing an application in VR would be similar to testing any other application, while we would have to take into account some particularities about the environment. Later on we will see different kinds of testing and think about the additional steps for them in VR, but first let’s see the characteristics of the different types of devices.
Types of devices:
Phone devices (Google Cardboard or DayDream) – allows you to connect your phone (or tablet) on the device to be able to play a VR app in there.
This is possible because most of smartphones nowadays come with gyroscopes: a sensor which uses the Earth’s gravity to determine the orientation.
Some Cardboards (or other plastic versions) can have buttons or a separate trigger for actions on the screen (as it is the case for DayDream), but the click is usually not performed in the object. Instead, it is done anywhere in the screen while the sight is fixated on the object. If the device does not have a button or clicker, the developer have to rely on other information for interaction, such as entering and exiting objects or analyzing the length of time the user was on the object.
Computer connected devices (HTC, Oculus, Samsung VR…) that generally comes with (at least) a headset and handset, have an oled device with high resolution and supporting low persistence embedded in the headset, so you don’t need to connect it to a screen. They detect further movement, as it is not just about the movement of the head but also the movement on the room itself and the hand gestures. This is done differently depending on the device itself.
We have moved from being able to detect user head movement (with reasonable size devices), to use sounds, to use hand gestures… so now, testing VR applications is getting more complicated as it now require the test of multiple inputs. The handset devices usually have menu options as well.
Before going on, I’d like to mention AR. AR is about adding some virtual
elements in the real world, but with AR we do not create the world.
However, AR has a lot in common with VR, starting with the developing
systems. Therefore, the testing of the two platforms would be very
We have talked about the hardware devices in which the VR applications would run, but we should also talk about the software in which the applications are written.
Right now there are two main platforms for developing in VR: Unity and Unreal, and you can also find some VR Web apps. Most of things that are done with Unity use C# to control the program. Unreal feels a bit more drag and drop than unity.
Besides this, if you are considering to work in a VR application, you should also take into account the creation of the 3D objects, which is usually done with tools such as blender or you can find some already created onlin.e
But, what’s different in a VR application for testers?
Tests in VR applications:
VR applications have some specifics that we should be aware of when testing. A good general way of approaching testing on VR would be to think about what could make people uncomfortable or difficult.
For example, sounds could be very important, as they can create very realistic experiences when done appropriately, that make you look where the action is happening or help you find hidden objects.
Let’s explore each of the VR testing types and list the ways we can ensure quality in a virtual world. I am assuming you know what these testing are about and I’m not defining them deeply, but I will be giving examples and talking about the barriers in VR.
It ensures that the customer can use the system appropriately. There are additional measurements when testing in VR such as verifying that the user can see and reach the objects comfortably and these are aligned appropriately.
We are not all built in the same way, so maybe we should have some configuration before the application for the users to be able to interact properly with the objects. For example, the objects around us could be not seen or reached easily by all our users as our arms are not the same length.
You should also check that colors, lighting and scale are realistic and according with the specifications. This could not only affect quality, but change the experience completely. For example, maybe we want a scale to be bigger than the user to give the feeling of shrinking.
It is important to verify that the movement does not cause motion sickness. This is another particularly important concept for VR applications that occurs when what you see does not line up with what you feel, then you start feeling uncomfortable or dizzy. Everyone have a different threshold for this, so it is important to make sure the apps are not going to cause it if used for long time. For example, ensure the motions are slow or placing the users in a cabin area where things around them are static, or maintaining a high frame-rate and avoiding blurry objects.
If there is someone on your team that is particularly sensitive to motion sickness, this person would be the best one to take the tester role for the occasion. In my case, I asked for the help of my mother, who was not used at all to any similar experiences and was very confused about the entire functioning.
Is a subset of usability testing that ensures that the application being tested can be appropriately used by people with disabilities like hearing, color blindness, old age and other disadvantaged groups.
Accessibility is especially important in VR as there are more considerations to make than in other applications such: mobility, hearing, cognition, vision and even olfactory.
For mobility, think about height of the users, hand gestures, range of motion, ability to walk, duck, kneel, balance, speed, orientation…
To ensure the integration of users with hearing issues, inserting subtitles of the dialogs would be a must, and ensuring those are easily readable. The position of the dialogs should be able to tell the user where the sound is coming from. In terms of speech, when a VR experience require this, it would be nice if the user could also provide other terms of visual communication.
There are different degrees of blindness, so this should be something we want to take into account. It is important that the objects have a good contrast and that the user can zoom into them in case they are too far away. Sounds are also a very important part of the experience and it would be ideal that we can move around and interact with the objects based on sound.
I realized on how different the experience could be depending on the user just by asking my mother to help me test one of my apps. She usually wears glasses to read, so from the very beginning she could not see the text as clearly as I did.
I mentioned before that in VR it is possible to interact with object by focusing the camera on them for a period of time. This is a simple alternative to click without the need of hand gestures for people with difficulty using them.
There are many sources online about how to make fully accessible VR experiences, and I am sure you can come up with your own tests.
Its purpose is to ensure that the entire application functions as is should in the real world and meets all requirements and specifications.
To test a VR application, you need to know the appropriated hardware, user targeting and other design details that we would go through with the other type of testing.
Also, in VR everything is 360 degrees in 3 coordinates, so camera movement is crucial in order to automate tests.
Besides, there might be multiple object interaction around us that we would also need to verify, such collisions, visibility, sounds, bouncing…
There are currently some testing tools, some within Unity that could help us automate things in VR but most are thought from a developer’s perspective. That’s one more reason for us to ensure that the developers are writing good unit tests to the functions associated with the functionality and, when possible, with the objects and prefabs. In special, unit test should focus in three main aspects for testing: the code, the object interaction and the scenes. If the application is using some database, or some API to control things that then change in VR, we should still test them as usual. These tests would alleviate the integration tests phase.
Many things are rapidly changing on this area, as many people have understood the need for automation over VR. When I started with unity, I did not know the existence of the testing tools, and tested most of it manually, but there are some automated recording and playback tools around.
Is the process of determining the speed or effectiveness of a system.
In VR the scale, a high number of objects, different materials and textures and number and of lights and shadows can affect the system performance. The performance would vary between devices, so the best thing to do would be to check with the supported ones. This is a bit expensive to do, that’s why some apps would only focus in one platform.
Many of my first apps were running perfectly well in the computer but they would not even start on my phone.
It is important to have a good balance to have an attractive and responsive application. New technologies also make it important to have a good performance, so the experience is realistic and immersive. But sometimes in order to improve performance we have to give up on other things, such as quality of material or lights, which would also make the experience less realistic.
In the case of unity, the profiler tool would give you some idea of the performance, but there are many other tools you can also use. In VR, we need to be careful with the following data: CPU usage, GPU usage, rendering performance, memory usage, audio and physics. For more information on this, you can read this article.
Also, you can check for memory leaks, battery utilization, crash reports, network impact…. and use any other performance tools available on the different devices. Some of these get a screenshot of the performance by time and send it to a database to analyze or set up alerts if anything goes higher for you to get logs and investigate the issue, while others are directly installed on the device and run on demand.
Last but not least, VR applications can be multiplayer (that’s the case of the VRChat) and so we should verify how many users can connect at the same time and still share a pleasant experience.
Ensures that the system cannot be penetrated by any hacking way.
This sort of testing is also important in VR and as the platforms evolve, new attacks could come to live. Potential future threats might be virtual property robbery, especially with the evolution of cryptocurrency and with monetization of applications.
Localization testing: as with any other application, we should make sure that proper translations are available for the different markets and we are using appropriate wordings for them.
Safety testing: There are two main safety concerns with VR (although there might be others you could think of)
1. Can you easily know what’s happening around you?
Immersive applications are the goal of VR, but we are still living in a physical world. Objects around us could be harmful, and not being aware of alarms such as a fire or carbon monoxide could have catastrophic results. Being able to disconnect easily when an emergency occurs is vital on VR applications. We should make sure the user is aware of the immersion by ensuring we provide some reminder to remove objects nearby.
Every time I ask someone to give a test to some app with the mobile device, they start walking and I have to stop them otherwise they might hit something. And this is the mobile device, not the entire immersive experience.
Even I, being quite aware of my surrounding, give a test to some devices that include sound, and I hit my hand with several objects around me. Also, I could not hear what the person that handed over the device was telling me.
2. Virtual assaults in VR:
When you test an application in which users can interact with each-others in VR, the distance allowed between users could make them feel uncomfortable. We should think about this too when testing VR.
Luckily, I haven’t experienced any of these myself but I have read a lot of other people talking about this issue. Even some play through online on VR chat, you can see how people break through the comfort zone of the players.
Testing WITH VR: There are some tools being developed in VR to allow for many different purposes and technologies as emulation of real scenarios. Testing could be one of them, we could, for example, have a VR immersive tool to teach people how to test by examples. I have created a museum explaining the different types of testing, maybe next level could be to have a VR application with issues that the users have to find.
What about the future?
We have started to seen the first wireless headsets and some environment experiences such moving platforms and smelling sensations.
We shall expect the devices to get more and more accurate and complete and therefore there would be more things to test and that we should have into account. We are also expecting the devices to get more affordable with the time, which would increase the market.
I know, I still owe you some stories, but I am now inspired to talk about something else. Besides, today’s issue is easier to put it into words. I don’t need to sit down and think carefully on a way of explaining some technical concept such as artificial intelligence while not sounding boring. But, just so you know, I am still working on the other stories.
I would like to show you how dangerous social media could become and on one hand highlight the need of asking the right questions when a new technology comes along in order to set proper tests and barriers (lynx are curious animals, aren’t we?).
On the other hand, highlight the importance to take breaks from it and think about yourself and things that would make YOU happy instead of thinking of things that ‘would make other people think that you are happy’ and therefore approve of you. I hope you enjoy reading this article.
Let’s think about it: the most viewed YouTube channels from independent creators are from people below their 30’s (or just on them), many started them 5-9 years ago. They have been getting a lot of pressure from fans and companies that would like a piece of their influence. Celebrities with less direct exposure to their fans have done crazy things in the past because of social pressure. Yet these influencers are not invited (that I know of) to Davos or famous lists of most influential people, besides demonstrating incredible marketing strategies, knowledge of new technologies, having charisma and being very intelligent (more than they let to be seen in some cases)
There is a very important point to consider, which is its the potential for hacking (if you are interested on this, there is more information here and here). So, imagine that someone could actually go there and decide what you are going to see… how could this affect you?
Techniques and prevention:
Let’s imagine a platform that accepts comments and likes (let’s forget about dislikes). How could someone socially hack it?
1) Removing the likes: We would need to intercede the information that this platform is showing to the user and eliminate the likes the user will see. Maybe we should only eliminate them partially, so the user is not suspicious of not seeing any likes at all. For this, we could have a pondered random variable that would eliminate or not eliminate each like. How would you feel if all of the sudden nothing you write gets any likes? Prevention: From test side, make sure the like system works properly and cannot be done by anonymous sources. Make sure accounts are real. Make sure the user sees only real data. From user perspective, when you see something that you like, mention it in person, start a conversation about it instead of just clicking a button.
2) Liking specific posts: This is a bit fancier. Based on above, we could have some sort of AI algorithm that could classify the posts. Then we can decide which comments are going to show as liked for the user. How would you feel if only some type of your posts would get liked? Would that change your way of writing? Prevention: From test side, make sure all information is shown to the user. From user side, find your audience and focus on them. Also, consider talking with this people directly too (or in conferences). Try to keep honest to your goals for writing and who you want to reach.
3) Filtering comments: This would require some form of classification as with the previous point. Instead of targeting the likes, we would target the comments but the idea would be the same. Eliminate from view those that are not ‘interesting’. What would you think if you only receive a certain types of comments? Prevention: From test side, make sure all information is shown to the user. Maybe have a conversation about the feature itself and allow users to hide all comments. From user side, as above.
4) Creating comments: We could create new comments with AI. You might think the user would realize about this, but if done carefully they might not even notice this or confront the person making that comment. Besides, the social media platform might allow for not-logged in comments. This adds to the feeling of the previous one. Prevention: Have a conversation about blocking anonymous comments or disabling them. From user side, if you see a strange comment from someone that does not add up, clarify this with the person. It can also help with misunderstandings. Option 2, disable or stop reading comments.
5) Changing the advertisement around the website to a convenient one for propaganda or for harm (only for some types of social media). Prevention: Most of sites have a way of deciding what advertisements you are more interested of. Also, try cleaning cookies regularly or use private browsers and VPN services.
6) Extracting automatically interesting information for malicious purposes. Prevention: Be careful with what you post, don’t use information that is available for anybody as your passwords or security questions or pictures with personal data (such passports or train tickets). If you really want to share pictures of a trip, try uploading them after the trip and enjoy while in there!
7) Connecting certain types of people. I am not 100% sure how this could be used for malicious purposes but surely someone would find a way. Making sure you can block people is also very important.
8) Taking things out of context. Prevention: It’s very hard to delete something from the internet once it is out there, but some platforms allow it. Have you ever read your old posts? It is a good idea to do some clean-up every so often. Also, if this happens to you, keep a track of the entire context. Maybe have a system in which you can remove what you have written before it goes online, take some hours before posting to make sure you want to post that.
Why am I talking about social hacking social media in a test blog? Well, because, if you happen to be working on developing a social media project, you should make sure that these attacks are not possible and think about how the user could feel for other features to come.
(Please take some time to go throughout the articles I liked above to know more)
Thoughts about being addicted to being connected:
When was the last time you did something good for someone but did not tell anybody about it? When you do something good for someone and post about it, how do you know you are doing it because of the other person and not to have a better image of yourself in front of others?
When was the last time that you went for a trip and didn’t share the pictures with anybody? What about not taking any pictures at all? There is something truly special about having a memory that is just yours to have.
Social media has evolved quickly in very short time, and we need to consider a lot of new things, more so if we are in the development team of one of these platforms. We should really stop and agree about what is ethical and not with the particular platforms and maybe even list the set of contraindications as if we do with addictive substances. For example, would it be ethical for some platforms (maybe for young people?) to change what the users see in order to protect them from the bad critic? Consider this could, in theory, save some lives, but, in contrast, it would take away some potential good feedback disguised as bad comments. Maybe this is a feature you want people to turn on and off? If not, maybe you should list in your contraindications that it could be an issue. And I don’t mean terms and conditions, it’s not about saving your behind if anything happens, it’s about actually alerting the user about what could be experienced. Terms and conditions are…well.. another story.
Some sources if you are interested in this area that helped me control my internet usage: