“We have made a change and it will require at least 1 week to test”.
“We have made a small change and it does not require any testing”.
“We are blocked as we just figured out that we have dependency on the server team”.
We frequently hear above statements in our day to day scrum meetings. Most of the times we end up paying the cost for above in terms of these pain points
- Testing area is too wide and difficult to focus on
- Dependency is uncovered at a very late stage
- Risks are uncovered as defects
- Chaos all around while fixing those defects at last minute
The world around us also keeps changing. In fact it is unavoidable that even the most well-developed software system will need to make changes in order to keep up with the challenging world. Testing is also efficient when we know what to test, how much to test and when to stop the test. We cannot test the entire application.
So it brings a very important question, “Are we testing too much, too little or just right”? It is very difficult to answer unless we know the impact of the change we did. So to know the answer we have to go back to 1996 when Robert S. Arnold and Shawn A. Bohner coined the term “Impact Analysis”. It means “identifying the potential consequences of a change or estimating what needs to be modified to accomplish a change.” In simple words it provides an accurate understanding of the implication of a proposed change.
The idea of Impact Analysis is not to reduce effort but to increase efficiency. It gives us an understanding of areas of the system that may be affected due to the change in the particular section or features of the application. Impact Analysis will help us to determine what to test and based on that we can determine how much to test and when we can end testing execution. Impact Analysis can be done at various levels.
There are 2 types of Impact Analysis, which we can use.
The first type is Traceability Impact Analysis which helps to determine the scope of the change. Change can be software behavior in the requirement phase, System Architecture in the Design phase, Code in the Development phase, or Test Code/ Test Cases in the Testing phase.
The second type is Dependency Impact Analysis which determines the depth of the impact on the system. Systems can be like client/server in Requirement phase, Design documents/architecture diagrams in Design phase, Build system/unit/integration test system in Development phase or Test data/Test Infrastructure in Testing phase.
Skipping Impact Analysis doesn’t change the size of the task. It just turns the size into a surprise. In the product development surprises are rarely good news. Before the team says, “Sure, no testing required” in response to a change request, we should spend time on Impact Analysis.
Let’s go through some examples to understand how it can be applied:
Example 1: Rewrite of a mobile app
This is a feature enhancement on an existing feature. We already had a mobile app. Now we want to rewrite the app with a new UI.
Let’s use Traceability Impact Analysis and try to find out the scope of the change( very high-level scope) in different phases
- Requirement: Client App UI look and feel will change, but server API need not as they will serve the same as the older app
- Design: Components design need to be changed
- Development: Server API remains the same, but the client-side code needs to change.
- Testing: UI related test-cases need change as we are re-writing UI, but business logic related test cases remain the same. Some test case for older UI might go away as well
Lets use Dependency Impact Analysis and try to find out depth of the impact on the system
- Requirement: Client App system will change, but the server remains the same
- Design: Architecture diagram & design documents needs to be updated as we are using new technologies
- Development: Server API code remains the same, but the client-side code needs to change. Build system might need a change if we are using new UI technology
- Testing: No change in test data as business logic remains the same. No changes on the test system as well
Example 2: Software installer does not install in OS with European Language locale on windows.
This is a defect which was found during release and we need to fix it.
Lets use Traceability Impact Analysis and try to find out scope of the change at different phases
- Requirement: No change as the requirement is still the same
- Design: No change as no new requirement is introduced
- Development: Installer code for Windows needs to be changed
- Testing: We need to test installation-related test cases in various locales. We might run automation test cases on the localized OS as well
Lets use Dependency Impact Analysis and try to find out depth of the impact on the system
- Requirement: Client Installer needs to be change
- Design: No change
- Development: No change in developments systems as well
- Testing: We will need new test machines with localized OS
As you see above, it is very easy to call out what is scope and where it needs to be changed. Once we make the change, we will also know what is the risk as well. We have been using Impact Analysis in Autodesk for almost three years now and it has helped to reduce our release work and brought stability in our software. It allows us to identify dependencies at a very early stage. We are moving our focus from actual change to after-effects of the change. It helped us to know the system better day by day. Since we know what is the impact, we can easily document the risk associated with it. We can easily identify the defect cluster and address it more confidently.
For example earlier, during our release, if there is a defect found out during release activities it was a tedious process to go through all the changes and understand which can cause this. But with Impact Analysis we have already covered it since we list out impacted areas, risks, and flows that are changed. We just have to go through that and easily find out concerned change. With respect to dependency identification, we can inform stakeholders at an early stage itself when stories are being discussed that we will need this change from your side.
You may ask what the timeline of this and when we should start the Impact Analysis of features. I will suggest starting as early as possible. It will be best when the story is in the backlog and ready to be groomed. This will give a lot of confidence to the team to find out stakeholders, to determine how much effort goes in. In short, delivering the feature becomes easy.
As a coin has two sides similarly everything has its pros and cons. We went through some pros of Impact Analysis, let’s go through cons as well.
- Impact Analysis can be done by someone who knows the domain and system. It is difficult to do by someone who is new to the system
- It cannot be applied to a scenario where the feature is done from scratch as there is no existing feature to understand the impact analysis
- It takes time to do, so it is always advised to start as early as possible.
I like to say that Impact Analysis is not just a methodology to adopt, I believe it is a thought process. Every change we make in our software day today, we have to understand or analyze the impact of the change we made which can be anything like test code change, software change, design change, or a requirement change. We have to keep thinking about what will be the aftereffect of the change and keep looking for answers to the question “Are we testing too much, too little, or just right”. This does not guarantee that you are going to have 100% defect-free software. In fact, there is no 100 % defect-free software in this world. But if you adapt Impact Analysis, then you can at least make sure that you will not have any major defects. Even if you unearthed defects, it would be very easy to mitigate the risk around them and push towards release. You will have more controlled and less risky releases.
I hope after reading this team will go back and start putting thoughts on “what will be the impact of this change” addition to “what change I have to make”. Most of the time we put so much focus on the change that we forget about the side-effects of the change. I hope my experience with Impact Analysis helps to bring light to this.
Know our Super Writer:
Rajeev Ranjan
Rajeev Ranjan is Senior Software Development Engineer in Test from Autodesk Asia Pte Ltd. He has worked 2+ years in Autodesk and has more than 9 years of experience in the software industry with the main focus on quality and release engineering. He has experience in working in various domain like Banking, Energy, Digital Marketing, E-Commerce. Licensing with companies like ADP, Appdirect, Barclays before joining Autodesk. He always strives for efficient team process and ever-faster software release cycles.
