Integrity of Performance Testers – How Strong are you really ?

25 Apr

For number of years I have worked in Performance testing projects using various tools like JMeter, LoadRunner, Silk Performer, Web Load and sometimes even basic windows/Unix scripts. Some Projects were proof of concepts, some were fixed price, some were Time and Material type, some were big programs and some were Scrum Programs.

Performance Testing or Engineering bugs are very hard to fix and it consumes hell lot of time for everyone to fix. So people in almost all the cases wants to avoid the fixes by either blaming factor X or factor Y. So what are those X and Y factors? Below are some of those factors which I have come across in projects,

  • Environment is not in good condition. Typical examples people give are bad corrupt configuration of the application. Reasons for this could overnight windows update etc.
  • Environment is not stable. Lot of people keeps on adding code after code. No matter how strong your process is, it is still impossible to control your testing environment.
  • Environment is not powerful enough. Lot of application owners come forward and give this reasons for not meeting the SLA’s. Of course I don’t deny that environment is key component to assess results, but hardly anyone accesses the reason or analyses why SLA’s were not met. There isn’t any real port mortem done to say that its in fact due to environment SLA’s cannot be achieved.
  • Then we have Test Data. Second most commonly given reason for not meeting SLA’s. Whenever any exception is thrown, first people raise the fingers towards test data.Ohh your data is bad, that’s the reason we have high response time or so many failures in the test run.
  • Then the folks who build the application target the performance testers and his management saying that his script is not well crafted or he has not done scripting well. As a proof they produce some key finding from their backend servers and tells the testing team, look we have done x+y transactions in our server logs, whereas your test results shows only x transactions. When this comes out in open, testing team management goes into the damage control mode and tries to get into some compromise with the stakeholders. Hardly anyone analyze as what went wrong or does the metrics which are being compared are in the same scale. Everyone wants to get the quick signoff and move on with feel good results.
  • For some of the application, folks even list the tools as limitations. Good example for this is Winsock protocol in LoadRunner.Its just takes more time to replay compare to manual run.

In some cases whenever the project heats up, just reverse those factors and people will start asking you for those details, so whenever your project for some reasons is not meeting the objective, be ready with all your data, scripts, plan and other stuff as people are going to come and look those things very closely.

Some times in some projects like FP or Large Programs, stakeholders also offer a good rating to the performance testers in case if he provides a signoff with excellent results. All they are looking forward are clean runs. Those are runs where we do not have any errors and has response time which meets the given SLA’s. I have seen lot many engineers taking credit for clean runs in spite of having fatal functional bugs in the applications.

Maintaining and Standing by your results is one of hardest job a performance engineer does especially whenever the objectives are not meeting for various reasons. If for any reasons if you cannot stand by with your finding whether good or bad, then I strongly feel that one has to still walk miles in this profession due to inherent nature of this profession wherein bad news for some stakeholders is good news for other set of stakeholders.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: