Hard to reproduce incidents – 1

11 May

Does Functional testing add value to Performance testing? Do performance testers need to know Functional testing? I would say yes it does add lot of value to performance testing. Performance testing is nothing but testing concurrently the functionality of the application and seeing if performs well under many concurrent users.

Performance testing also uncovers the functionality related issues which cannot be done by functional testing alone. The main reason as why functional testing in itself is inadequate is that it follows the “I “process rather than “WE” process. “I” being the single, means at any point of time, only one person is doing the business process Whereas “WE” means many people doing activities on that functionality.

There are many examples to showcase as why functional testing by itself is insufficient however I would like to point to one specific example where in Functional testing was inadequate.

We had a multi tier application where in authentication was happening via Siteminder and this Siteminder was responsible for providing page level authentication to the users.

For each page, it use to send the SMSession cookie back the client and this cookie was updated for every request. So every time, client was getting the updated cookie from the Siteminder.In short, these cookies were an kind of identity to the Siteminder boxes that right users with right credentials were in fact browsing the site protected by it.

Now let’s assume that we have around 30 http requests for a business process, so client will be getting at the max 30 updated cookies from Siteminder web agent for the single users. Everything works fine with single user in the browser session manually, no issues at all. With the help of browsers we test for Siteminder functionality to ensure that cookies are in fact updated for each request, check if the cookies has all the required attributes like expiry date, length etc as per the specifications. So everything looks good for the user using the application.

However for any reasons, sometimes we start seeing the below page which is nothing but the redirect by IIS toward web agent which in turn sends the request towards policy servers for checking the login credential of the users. Everything looks perfect by design and works excellent for the single user sessions. The page displayed is also perfectly fine as per design.


However during load testing if we continuously start seeing this above page after every click, and then it becomes a problem and a bottleneck to the functionality itself. Now imagine the Google checkout functionality protected by the Siteminder and users after filling the payment details starts seeing this page, Immediately questions will start popping up in the minds of the users as  what will happen to the transactions that has been entered into the page by the user? Does there exist a risk that the transaction might become corrupt? Or they might lose the transaction itself depending on the reasons that is causing this page to pop up in the midway.All these doubts which is caused by this sudden change in the page,might make you lose the confidence of the end users.

Well I believe Functional testing if done correctly and with out of the box thinking strategy can help to mitigate this type of issues along with at least 40% of saving on time lines for Performance testing  or security testing which projects teams start in the later part of the project lifecycle.

This particular Siteminder issue was very interesting one, which I had seen. The reason this issue was interesting was that the above page was not easily reproducible and was happening once in the months while doing load testing.Since Siteminder solution’s are normally implemented across the portfolio of projects,I believe the impact due to this issue would definitely be very high on the bottom lines.

I have done some working on how this error can be reproduced and why we were seeing this error during load testing. Since this issue is logged with CA for their investigation, I will wait for couple of more days for have some more clarity on this. However I assure that I will post my findings on this one.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: