The big day arrived, after months of hard work in designing, coding, testing and publishing; your website is alive and kicking. While building your website you didn’t stop testing it, time after time checked how it will respond to every action the user does. This is the cornerstone of unit testing. Knowing how specific functions will perform in isolation. If you are using the MVC framework, you probably have a test method for the every action method. You have performed all this in an attempt to try and predict how the website will perform with a large number of concurrent users. Now the site is online and the big dilemma is upon you; do you carry on with performing the tests and mostly load testing or do you give it the benefit of the doubt and let it fend for itself(not literary).
First of all we will need to distinguish between load testing and stress testing. These two terms have often been erroneously used interchangeably. Load testing in essence works by populating the enterprise databases with data and attempting to retrieve it while observing the performance metrics. On the other hand stress testing is what actually deals with the number of concurrent users. I will not delve much into the justification of which is which more because at the end of the day I don’t see how we should have one without the other.
Let’s look at various possibilities and situations that might inform your decision to want to Load test or not.
- Website update frequency/magnitude
Depending on the nature of your website, the frequency at which you update your website might differ from others. For those who do it much often continuous integration is at the top of your list but this is a story for another day. Late last year I read an article in which facebook declared that they will be pushing updates twice a day. Such changes are usually in good faith mostly feature upgrades or patching up bugs. Before these upgrades the QA members of any tech team go through constant cycles of continuous testing to ensure that the right product is shipped. On top of the normal UAT performed and possibly golden path test it would be worthy to consider performing a Load test depending on the modules that change and their impact to the business and the website availability.
- Underlying technology change
Your website’s infrastructure will be a defining factor on whether you will need to perform load testing. Let’s consider some examples. You have a site that is purely powered by word press. This means that to some extend you are controlled by changes made by word press and the underlying theme. If there is an upgrade then how does that update affect your websites availability? Say you use MSMQ for your distributed applications and Microsoft have made a change. Do you think that the change will affect how your sub-applications communicate with one another? Suppose you are a document database proponent and hence are already using RavenDB. If there is an upgrade you need to be able to know that your entire site doesn’t have a drop in performance.
Those are changes on the software side. You might also decide that in order to enhance performance separate your App server from your database server. They may not have a dedicated channel to each other or maybe they do. You could also change your hosting company. As long as there is a change then it is clear what will need to be done. Integration tests as a priority but also /load test make sure potential bottleneck are identified.
- Strategic management
Top level managers usually will share the company’s vision with the tech team. They may outline the projected sales or customer registration. With the primary point of contact being the website you need the website to be able to support the projections when they happen. At this point the Load testing is a requirement to determine how the site will behave when such numbers are hit.
The outcome of such a test ensures that the right decisions are made with regard to whether to start patching up loose strings in the current system, start developing a new one or look for a developer if you outsource your software products.
- Cost/Revenue justification
This factor is usually affected by the last 3 points. The question is, is it worth doing the Load testing procedure? In order to answer this question what one needs to think about is what the cost of not doing it will be. Website outages usually cost thousands of dollars worth of revenue loss. It’s not rocket science when you look at the formula here below.
(Revenue per month) X (minutes of downtime)
Such an outage can be caused by a surge in the number of customers for instance accessing your website because of a marketing program.
When you consider the kind of loss that can be experienced then that is a starting point as to whether it’s worthy to do load testing on your live site.
- Perceived availability
With reference to the point above consider how negatively downtime will affect your company’s image. It’s even worse in face of customers vending their anger out in the social media platforms. Imagine that your website accepts Instant payment notification from third party payment providers. If it’s unavailable then the question of how many retries there has to be comes to play. It’s not a very good picture if the company needs business partners. Such outage could be caused by payments coming in hundreds. Therefore here a decision needs to be made based on performing a load test beforehand to know the maximum number of notifications that can be received.
From the points above we can deduce that once a site is live that will not be the end of it. It will be subjected to changes(or at least the underlying infrastructure) and it’s these changes that will mostly determine whether or not to load test your website. The cost aspect is always very thorny when justifying to the managers on why it needs to be done. Such a debate can be easily won by showing what stands to be lost if the site is unavailable as a result a load test that wasn’t done. I also want to acknowledge that a site could face an outage due to factors different from high numbers but while we can do something about numbers then we should hope that those other factors are taken care of their own way.
