In my career I’ve gotten a lot of questions about test architecture: How to start testing when nothing has been done for years? How many test projects should there be? How many different type of testing do we need? Who should be in charge of testing in percentages? How to escalate testing? How to deal with big applications? How to deal with existing tests that are incomprehensible? What tools should we use? How many browsers should we test (are there more than two!?)?
First things first: Each application is different, each company is structured in a different way and with different resources and budget and each one has a different maturity state (which does not mean they should aspire to go to the next state, maybe they are right were they should be). There is not a ‘one size fits all’ solution and that’s why I recommend you check it with an expert. But here are some of my previous experiences, in case you find them useful.
On getting started
The first things I ask when I start a conversation about starting testing for a company are:
- What process are they following? (Agile, waterfall…)
- What technology are you using? (For development, any existing tests, code sharing, code control, task management…)
- What’s the relationship between the developers and testers? Do they sit together? Share code? Do common code reviews?
- What size is the company?
- Do developers write unit tests?
- How many people are dedicated (or plan to be) exclusively to tests?
Do they ring a bell? Here are some of the questions I suggested you could ask your interviewer if you are interviewing for a job. Of course, depending on the answers to these questions, I would suggest a different set of tools and practices (or if I am looking for a job, I could tell if I would be able to grow professionally and help the company given the circumstances and what to expect of the position)
Some more that are truly valuable are:
- What does your app or company do?
- How many clients do you have/plan to have?
- What’s the current vs potential growth?
In my experience, when someone asks for help getting started with testing, there are three status the company could be at, and therefore three things might be needed:
- Convincing upper management of the need to test: Sometimes you need to be able to explain to high management of the need for testing so you can get started with it. Their main fears are usually time and budget. Addressing those fears are not always easy and depend on each case, but the most important question is “Why do we test?” I leave this one for you for now, but if you are able to explain it, this point would be very easy to deal with.
- Convincing other team parties: Other times, they know testing is needed but they don’t know where to start (hint: unit test and TDD) When you don’t have a test team in place yet, I usually suggest the developers start getting into test. Then it would be nice to have someone with test experience to orchestrate these tests, to educate the team about testing and to start creating a test system for end to end and other sort of testing, or even doing manual checks. However, from when you get someone to when you have it all up and running, it might take a while, that’s why it is good that the developers start spending time in unit testing, as ideally they would keep doing this afterwards anyways.
- Getting a full framework up and running: In this case, the company have some unit testing and/or manual testing but they need help creating an automation framework and identifying other testing needs. Maybe they are also feeling insecure about the current tests and want to make sure they are not missing anything or want to improve their system (see the next two points).
On keeping it going
Sometimes people are struggling with their current test systems. They might be wondering if they are doing their best as it is, or if the tests they have provide actual value. The questions you need to ask yourself are:
- Is there any repetition that could be reduced? (how many current tests and what type of tests do you have? If they run over 15 minutes, this should likely be improved)
- Is what we are testing really important? Sometimes we test things that don’t really provide any value but it’s taking time and budget to implement and keep.
- Are there any tests that are not up to us? For example, downloading something from a browser, unless your app IS a browser, it’s probably something you should not test.
- Are we executing out of date tests? Keeping the system clean is highly recommended, if a feature comes back, you should be able to retrieve those tests too.
- Could we use any tools for tracking the issues to understand where we need or don’t need tests?
- Do we have the right tests for the right moment? Integrating tests with CI/CD in different branches means you need to decide what to run and check and when.
Horizontal scaling means that you add more machines to your testing. For example using the cloud to test in different systems (more web browsers, mobile vs machine, different OS…)
Vertical scaling means that you scale by adding more type of tests (security, accessibility, performance…)
Identifying the type of tests you need and what systems should be covered would be part of an architecture design.
On moving on
You’ve set up one or more test frameworks. Your team (developers/SDETS…) are managing the new tests which are carefully planned and being executed as part of the CI/CD structure. The analytics are set up and alerting of any problems or improvements… Could you ever be done with testing?
Most people in quality would straight away jump to the easy answer: “you are never done with it”. However, I feel like bringing up the argument: I believe sometimes you are. The same way as you might be done with development too. You will never be 100% happy about it, but sometimes you’ve done your best and maintenance it’s all there is left, at least for that project.
What do you do when you are developing and you are done with that project? You move on to the next. Of course, someone should stay to fix potential issues and provide support, but you wont need such a big of a team or the same sort of skills for this. I argue here, that the same could be said for testing. Development and testing come hand to hand, when there are less changes in development there would be less changes in testing.
Does that mean nobody should look into it anymore? No, that means you need someone with other set of skills to do that. Not worse or better, just different. Some people enjoy being at the beginning of the projects (I love doing research and creating new stuff) but other people prefer maintaining them. And this does not need to be after all it’s escalated, maybe your company is in the place it should be, and investing in more testing might not be necessary, even if you have the budget for it. The problem is, of course, discerning when that is the case, for which I highly recommend talking to a professional.
On hiring the right test manager/architect
As I mentioned before, having a test expert helping figuring out the maturity of the company and what is needed to improve the quality of the product is very important. What do we call this expert? Test manager and test architect should not be the same position, although, sometimes companies use the descriptions a bit freely.
Test manager: A test manager makes sure the tests are done correctly (being them from test or dev team) and good practices are implemented. The position requires deep knowledge about testing and people skills.
Test architect: The architect designs the frameworks and tools to be used by testing. The position requires deep technical knowledge and experience planning and coding things from the start (also deep knowledge about testing).
Not all companies need both of the positions, and sometimes both jobs would be done by the same person. But being called whatever it’s called, someone has to make sure the system to do the testing is in place and someone has to make sure there are good practices and the right people understand how and what to look for in the quality area.
Therefore, in order to look for the right skills, you need to understand what you need. I have been asked many times for help defining test roles (I talk about the roles I’ve been in here), interviews and even salaries. In fact, there was a time I had to tell my interviewers how to interview me (and, funny story, somehow I ended up not getting the job!) But that’s, well, another story…