Basics of Scenario Designs

15 May

Let’s talk something about scenario design for doing Load testing. Recently I received a mail from one of my old friend, saying that someone from Big 3 companies asked him the following questions about scenario designing for Load Testing.

  • We are running a test with 2000 users for duration of 4 hours. 1 Iteration takes about 10 mins so for 4 hours how many transactions will be achieved.
  • We want to run a test for 8 hours and complete 8 iterations.How do we design a scenario.

Scenario as per me is imitating the end user’s journey via application using scripted approach. So scenario in addition to following the end user’s path also needs to incorporate the browser behavior which end user uses while walking through the business process. Scenario also needs to ensure that it contains the overall picture of the application usage like how many people are using the application at any point of time, are those people going to use it for entire 8 hours or some folks might drop off after sometime.

Sometimes for some of the application, scenario also contains the details of specific environment which the end users are going to use for accessing the applications. For those environment specific parameters needs to be incorporated in the scenarios. Good example for these applications are mobile applications .

However mostly scenarios should contain the below details in it,

  • Ramp Up: Does the application users increase over time, if yes then we need to have things in scenario design.
  • Ramp Down: Does the application usage tapers down towards the end of day/hour. If yes, then we need to have these things in scenario design.
  • Steady State: Does the application users remain constant throughout the time, if yes, then we need to have these things in scenario design.
  • Duration: How long are they going to use it, do end users do the transaction all day or for specific amount of time.
  • Location: Where are the users of the application based? Are they from same city or different countries? If users are distributed across the globe, then it does add another layer of math to evaluate host of other parameter.
  • Volumes: These could be the transaction/hits per second or Throughput or Pages per second or could be any other metric which we are attempting to achieve so that we can come up with some assessment whether the system can meet the business requirement successfully. Scenarios in all cases needs to achieve some predetermined objectives and only then we can that scenario has successful.

Now all these above points are high level points which one needs to take care while designing the scenario. Since we follow the scripted path to achieve our objectives, there are also certain low level parameters in the scripts that impacts scenario design and needs to be taken care, those are

  • Think time given the scripts: These are nothing but the pause time taken by the end users.
  • Pacing given the scripts: Since the scripted approach is much faster than manual approach, it is important that we have some kind of pacing requirement to control the behavior of the scripts. If user are doing 10 similar transaction per business process , then in real time, we assume that they are going to pause for sometime after doing x transaction out of those 10 transaction. He may feel like having a coffee after each transaction to celebrate successful transaction. Pacing time in scripts reduces the load on the servers.
  • Browser emulation: In real time, most users prefer to use browser cache to cache the frequently used data, so it’s important that scripts needs to have similar setup.
  • Network connection: If the users are based in open internet, then it makes more sense to connect to the application directly in the internet with out proxy servers. It’s basically how we connect to the application. Network connections via scripts / proxy servers add another layer of math in scenario designing.
  • Iteration Setting: Sometimes to achieve the business volumes, we might also need to set up some iteration count in the scenario design.
  • User behavior: How are the users using the applications, are they doing login only once or many times during the day. If they are doing login only once, then scripts needs to update this so as to reflect this behavior. In case of authentication is happening via some third party servers , then it makes more sense to check for this behavior ,since most SSO servers are not transactional servers. Once retrieved they all cache the credentials. This is my understanding; however please check this once again with your SSO admins as how they have implemented SSO.
  • Cookies: Sometimes cookies needs to taken care explicitly. Session cookies are good example where custom handling of cookies is required.
  • Headers: Sometimes custom headers need to be handled explicitly to achieve the some technical objectives.

All these high level and low level points needs to communicated and signed off with the relevant stakeholders before commencing the load tests on the application. Some folks also add this information to their test plan and I feel this is best practice.

Given that they are host of factors which one needs to take care while designing the scenario, questions asked to my friend seems to lack most of this information. However I do agree that these questions tests the high level math involved in scenario designing. Now coming back to these questions,

We are running a test with 2000 users for duration of 4 hours. 1 Iteration takes about 10 mins so for 4 hours how many transactions will be achieved.

1 iteration means for one user it takes around 10 mins. For this lets assume that we do not have any think time in the scripts and there is not browser cache involved. So if 1 iteration has around x transactions in it, then its going to achieve x transaction in 10 mins.So in an hour only 4 iterations can be done, so we can achieve only 4x transaction in an hr.So for 4hrs , it becomes total of 16 iteration with 16x transactions. This is only information I can determine from the given questions. However as per me , achieving 16x transaction is also not possible sometimes given that there are host of factors which impacts transactions like if we get errors midway, or if we have high response time ,then its quite a possibility that given numbers cannot be achieved.

We want to run a test for 8 hours and complete 8 iterations.How do we design a scenario.

There are number of ways to achieve this like running the script and noting down the time it takes to complete and then adding rest of the time either as think time or asking the script to sleep. However running this scenario is a challenge in itself for various reasons.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: