Introduction:
As a performance tester for last 15 years, I observed how the organizations are shifting towards browser performance testing from protocol (communication mechanism) performance testing to meet today’s accelerated DevOps methodologies. Protocol performance testing is still used more (as also includes non-browser based application) in my experience. In this blog article, I will discuss about why the organizations are inclined towards browser performance testing over protocol performance testing especially in DevOps.
DevOps performance testing:
Let us first understand DevOps performance testing. It combines both shift-left and shift-right performance testing. Shift-left performance testing can be thought of performance testing in dev (development) and should be started early. It is like preventive measure since early stages to avoid any potential issue in production. Whereas, shift-right performance testing can be thought of performance testing in ops (production) for real user experience and feedback. It is like proactive measure to identify the potential performance issue and resolve immediately in production before it actually impacts its end-users.
Traditionally performance testing was only with protocol based. As more and more organizations are starting to use DevOps, they need a solution where performance script can be created quickly and maintained easily. This is mainly due to making performance testing as part of CI/CD (Continuous Integration/Continuous Development) pipeline and knowing the actual end-user experience in browser. Both of these can be easily done via browser performance testing. For example: industry standard commercial tool LoadRunner browser level solution-TruClient protocol and open source tool JMeter browser level solution- Selenium WebDriver sampler.
Browser driven performance testing for today’s modern applications:
Protocol driven performance testing is on server-side communication whereas browser driven performance testing is on client-side communication. Also, protocol based performance testing results are not included browser rendering time. Today’s modern applications are mostly rich internet based where DevOps methodologies are applied. Due to that it is essential to measure browser rendering time in modern applications. Browser level performance testing results includes these rendering times. This is one of the reasons application owners in DevOps are inclined towards browser performance testing.
Browser driven performance testing for component level performance:
Protocol based automated performance test script creation requires substantial amount of time (most of the time is spent on correlation i.e. handling dynamically generated server-side values) and good application knowledge as well as technical knowledge. On the other hand, browser performance test scripts are easy to create. Even maintaining browser performance test scripts are quite easy compared to protocol based due to having less complexity.
Having shift-left approach in DevOps, performance testing should be conducted at component level and at early stages. Component is the lowest unit of an application. An application can be thought of as a group of components. Component performance testing can be conducted certainly as an automated manner as part of CI/CD pipeline process after each build. This will ensure not only early performance feedback but also build failure and performance trends. As browser level performance test scripts can be created easily, takes less time to create and easy maintenance, it perfectly matches with the accelerated delivery concepts. That’s why DevOps application owners are more towards browser based performance testing.
Browser driven performance testing for accurate end-user experience:
Protocol driven performance testing is not conducted on any real browser. Browser rendering time for HTML or JavaScript are not added in protocol driven performance test results. It is more focused on server and network level performance. As a result, protocol driven performance testing don’t provide accurate end-user experience. On the other hand, browser performance testing uses real browser and includes rendering page loading time like HTML, JavaScript etc., in performance test results. This rendering time can definitely impact end-user experience. Overall, browser performance testing provides accurate end-user experience over protocol driven performance testing.
If we look at the DevOps methodology, shift-right talks about real-user experience and feedback. As, browser level performance testing provides accurate end-user experience, DevOps application owners are trending more towards browser based performance testing.
Browser driven performance testing- less scripting effort but more resource cost:
Brower based performance testing is hugely resource intensive. Each browser instance requires its CPU and memory resources resulting enormous resource required for performance testing. So, more powerful load generator machines or a greater number of load generator machines will be required than protocol performance testing for same number of stipulated user performance test execution. This directly increases the overall performance testing cost.
Yes, DevOps owners know that browser performance testing is having above said added performance testing infrastructure cost. However, browser driven performance testing efforts (both creation and maintenance) and time duration are always less than protocol performance testing. This is mainly because of easy and fast performance scripting in browser level performance testing.
Overall, DevOps application owners are inclined towards browser performance testing as it supports both in accelerated delivery and accurate end-user experience.
Conclusion:
In a nutshell, below table will summarise why organizations are now inclined towards browser performance testing over protocol performance testing in DevOps methodologies:
| Parameters | Protocol driven | Browser driven |
| Communication | Server-side | Client-side |
| Performance Testing Lab Infrastructure cost | Less | Higher |
| Performance Script Creation & Maintenance | Tough & Time consuming | Easy & Quick |
| Performance Scripting Efforts & Cost | Higher | Less |
| Browser | Not on Real browser | Real Browser |
| Browser Rendering Time | Not Added | Added |
About the Author:
Arun Dutta | Senior Test Manager
Arun earned a degree in Computer science from Govt. Engg. College, India (college topper). He is having 15 years of managing E2E testing delivery experience in different types of applications. He has a keen interest in reading and writing different technical papers. He has been selected in multiple international conferences; global webinars and his papers have been published in multiple forums and won various awards. Currently, he is working as a Senior Test Manager and Global Subdomain Leader for Atos Expert: Modern Applications-Testing in Atos-Syntel.
