Software development has changed significantly over the last few years. Waterfall models fortunately belong to the past and even large corporations are adapting to agile development methods. Even if this means an increase in profits for the companies, the winners of this development are the end users. “Customer is king” finally applies to software development: companies act fast and changes are implemented immediately.
As promising as it sounds, there are also downsides. In the past, a large-scale test on release day was usually sufficient to uncover the most important software bugs. Nowadays, it is logical to test every commit, but commits happen weekly if not daily. One of the biggest challenges with these rapid software changes is to ensure software quality and especially consistency. Small mistakes can lead to expensive downtime – just imagine a non-functioning shopping cart in an online store preventing customers from shopping. Therefore, software quality should be a top priority in every software and web based company.
For early-stage startups however, the relevance of software testing for functionality does not seem to have arrived yet. I rarely see startups using end-to-end tests before a release. A study by the Blekinge Institute of Technology (Chapter 7.2.4) finds that quality assurance in startups is even “largely absent”. This is understandable considering: the earlier the startup gets feedback, the earlier it can pursue or pivot the idea. And functional tests cost time, which startups don’t have. Therefore, early releases and user tests are usually used to discover bugs. In my experience however, this approach is fundamentally wrong.
The risk of not testing for functionality beforehand
Of course startups should be evaluating hand in hand with their customers in the early stages. However, it must be clearly worked out what the actual purpose of these user tests are: to test the functions for relevance and acceptance by the user (Let’s call this “idea testing”), or just testing for pure functionality.
Mixing these tests generally leads to distortions and here is why:
Imagine an innovative travel planner: Tinder for travel – you swipe to the right when you find an interesting trip. Now the first users express their interest and an early version is released. The founders want to find out which functions are considered good or bad – and whether the idea is well received. In addition, the team has built the functionality to match users with similar interests. Finally, the software is released without broad functional tests, because time equals money and bugs shall be reported by the users.
After the release it’s noticeable that the matching functionality only works correctly on Android, but not on Apple devices, matched users are not saved (Yay, we found a bug). Afterwards the team interviews a few users about their experiences with the app (standard procedure), because the real point is to check if the idea works and is marketable.
Telling someone that their software has a bug is easier than criticising someone’s idea. By pointing out a bug, the user can avoid unpleasant feedback and still meet his or her target of giving feedback. Therefore, the following will happen: all Apple users will report the defective matching function as a bug and furthermore focus on finding more bugs. And this is exactly the problem. The actual purpose of the testing gets out of focus and the startup doesn’t learn if the functionality itself is useful or not. If there were no software bugs at all, the focus would automatically be on the product itself. Startups often try to counteract this effect in problem interviews by saying something like: “Imagine the performance would be better,…” or “Imagine that the bug isn’t there, …” ( I am guilty of this as well and I hear it all the time).
Can those questions even work? The purpose of idea testing is to test the real benefit of a function on the user, not suggesting scenarios that the user should imagine. If the users are asked such questions, they will always answer: “Yes, I would like it better if this would work” – and of course this is true. Without the bug the function would be more useful, but if the user really likes the idea behind it remains open – also because the user did not use the functionality as it was supposed to be.
Therefore, it’s essential that startups work with error-free software in their idea tests. It’s more important to use the basic functions with stable performance and without bugs than to include additional untested features. Most early-stage startups I know do not fully test their software for functionality – also because functional testing is perceived as tedious, unnecessary and very rarely software testers are hired among the first employees. One example: software development professionals working in startups were asked in a study by the University of Cape Town which software development skills are important for startups. “Test Software” was only in 15th place out of 19 possible.
Not testing for functionality almost always distorts the results of idea tests, which is fatal for the survival of the startup. However, even with very few resources, startups can ensure that their software works properly without much knowledge in software QA.
What startups should do
In the following I would like to propose three points which can help startups to develop working software and at the same time make their idea testing more effective:
Make yourself aware of what you want to test
Startups should always distinguish between functional and idea tests – never try to mix both – as promising as it seems. Also functional testing should always be done before the idea test and everyone in the team should be included. It can easily be done manually and does not have to be automated in early stages. Also document what you test, there are a lot of free test management suites out there, in which you can define your most important click paths and schedule them to your employees and co-founders. You don’t need your users to find bugs – do it yourself.
Focus on the essentials
Focus on key functionalities in your application for your idea test and ensure those functionalities are working correctly on every hardware and platform. Do not add untested features to your application as it will twist your results of the idea test.
Be your own customer
At last and most importantly: use our own product. This is nothing new for most entrepreneurs, but I want to highlight this point from a software testing perspective. Most bugs are detected while using actual products, you can define as many test cases as you want, the chance that you miss some combination of functions in your descriptions is very high. By using your own product on a daily basis, you ensure you test the product like your customer uses it and you will find bugs before them.
Key takeaways
Building a startup is a lot of work and resources are limited. One of the biggest reasons why startups fail is the lack of product-market fit. It is therefore understandable if founders go to market with their product as early as possible to show a need. However, it is essential that the products themselves work, otherwise wrong conclusions will be drawn. Therefore, functional testing plays an essential, often neglected role.
About the Author:
Jonas Menesklou | Founder AskYourUI
Jonas Menesklou studied Industrial Engineering (M.Sc.) at the Karlsruhe Institute of Technology with a focus on user-centric security and QA. Furthermore, he is the founder of the startup AskYourUI, which offers a UI test solution. Through his work with startups he knows the pitfalls of early software products and tries to overcome these in his daily work.
Enjoyable read that, thanks for posting.