My Thoughts on Out Of Memory Issues

28 May

Recently I came across the post in LinkedIn Discussion (Performance Specialist) group where the poster was asking about how to fix or identify or reproduce out of memory exceptions. Lot of people suggested him a correct process which I agree are in fact some of the steps which we follow while fixing the out of memory errors. However in addition those steps, normally there are many other things which one has to take care while debugging OOM issues. There many steps of analysis like measuring heap size, checking finalizer threads, and reading thread trace etc which needs to be done after taking dumps. Dumps will give you the information required to troubleshoot the issue but will never say where the issue lays precisely, its up to the person who is using that information to find it out as why we are getting out of memory issues after reading and interpreting all the information present in the dump file. So obviously skill sets and basics come into the picture here. I would call it more of Reverse software engineering. We read information backwards towards the written code.

I have come across many such projects wherein I have seen OOM issues across Java and .net platforms and to some extent we have successfully debugged OOM issues in the past. However I would like to point that OOM issues cannot be debugged by the performance engineer alone. These types of issues also require inputs from the Environment folks and Development Folks. Without their involvement, we will never have permanent fix for this for the simple reason that Environment folks are the ones who monitors and controls the environment and development folks are the ones who knows the code better because we know that they are the one who has coded the application. Memory issues manifests either due to bad environments settings or Faulty working code or incorrect scenario settings.

So first let me give you some of my experience where I have come across the OOM exceptions and my thoughts as why we normally see the OOM exceptions,

The first case where I have seen Out of memory exception are those application which are displaying the 100’s of 1000’s of data and out of these user selects one or 2 records to satisfy his business purpose. The core problem with displaying huge amount of data is that, it consumes lot of memory and your platform needs to hold this much amount of data in the RAM. Though by keeping data in memory has some performance improvement, it does have impact on memory management. You are holding the data which is not required by the users and you are keeping that in RAM. In fact this is typical symptom of bad design.

The Second case where I have seen Out of Memory exceptions are when we have lot of unhandled exceptions and these are not handled correctly in the application code. Exceptions consume the memory resources and they are unhealthy to the application performance.

The Third case is where you have large amount of view state or session state. I remember testing some applications wherein size of the view state generated were in the range of 1 to 5mb and in some cases was going up to 10mb.Problem with this approach is that you are passing too much data back and forth across the wire which increase overheads for serialization and deserialization process which in turn consumes lot of memory.

The Fourth case was where the server configuration setting was totally incorrect. Almost every server provides you with the setting to set the physical and virtual memory. Normally there isn’t a fixed setting one can use for their applications and most people initially want to go with default setting which comes with web servers. In most of the cases default setting does the job, however there are cases where default settings will not be of much use, in such cases it makes more sense to do a couple of load tests and determine the optimum memory setting for your application. In addition to memory setting there is also other setting which impacts memory consumptions.

Then the Fifth case was something where in the application was encountering the deadlocks situations among threads and data was getting frequently written by these threads into the memory. Deadlocks can happen for variety of reasons, normally whenever the deadlocks happens in the CLR threads or JDK threads, it increase the possibility of OOM conditions. Sometimes Large objects do not get garbage collected due to these deadlocks and we see that they are not Garbage collected because the finalizer was blocked due to various reasons.

We do have many tools to identify the out of memory conditions and take memory dumps and thread dumps. However the person taking and analyzing dumps also needs to thoroughly skilled in various aspects of memory management, garbage collections methods and processes, thread management and various other technical stuff involved. He also needs to be aware of programming concepts involved in the application. Since the topic on this is very vast, I would not like to into details and would share with you some links which I use whenever I get those OOM issue.

Here are those links, However please do note that these are for .net platform and I am sure they will provide you with excellent insights as what exactly are out of memory exceptions and how to deal with it. For Java platform, I will make that as a separate post as there are some points which differ to some extent compare .net.

http://support.microsoft.com/kb/911716

http://blogs.iis.net/webtopics/archive/2009/05/22/troubleshooting-system-outofmemoryexceptions-in-asp-net.aspx

http://blogs.msdn.com/b/tom/archive/2007/11/26/high-memory-cpu-or-other-performance-problems-with-net.aspx

http://www.onpreinit.com/2009/09/windbg-sos-dumpdatatables-aspxpages-etc.html

http://www.stevestechspot.com/default.aspx#a994b5c77-6fbc-4994-9623-3772b0505469

http://blogs.iis.net/webtopics/archive/2009/03/12/high-memory-due-to-system-weakreference.aspx

http://naveensrinivasan.com

http://support.microsoft.com/kb/892277

http://blogs.msdn.com/b/tess/archive/2009/04/16/net-exceptions-quick-windbg-sos-tip-on-how-to-dump-all-the-net-exceptions-on-the-heap.aspx

http://blogs.msdn.com/b/tess/archive/2005/11/30/are-you-aware-that-you-have-thrown-over-40-000-exceptions-in-the-last-3-hours.aspx#9557706

http://www.slideshare.net/CoryFoy/debugging-net-applications-with-windbg/download

http://blogs.msdn.com/tess/archive/2008/03/17/net-debugging-demos-lab-6-memory-leak-review.aspx

http://www.codeproject.com/KB/dotnet/Memory_Leak_Detection.aspx

http://msdn.microsoft.com/en-us/magazine/cc163491.aspx

http://support.microsoft.com/kb/919790

http://blogs.msdn.com/b/dougste/archive/2006/10/17/clrdebug.aspx

http://www.simple-talk.com/dotnet/.net-framework/a-look-at-exceptions-in-.net-applications/

http://blogs.msdn.com/b/santiagocanepa/archive/2011/02/28/memory-based-recycling-in-iis-6-0.aspx

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: