10 minute read

Here are of some of the reader emails I got this week and my answers to them…

  • How do I troubleshoot this memory leak
  • Debugging techniques for capturing stacks of OOMS
  • Do you have a list of Debugging 101 links?
  • Debugging managed code in IE
  • Getting to dependency properties with sos

let me know if this is something you find useful so I should keep doing it from time to time, or if it’s only useful for the selected few who’s email I happened to pick this week:)

Note: some of the emails have been edited to fit better in a Q/A format, hopefully I haven’t missed any important bits…

How do I troubleshoot this memory leak

I started observing a slow leak in task manager of about 100 MB / 30 mins, in a web application consisting of WCF services.

In perfmon Gen 0, 1, and 2 collections steadily grew, and Gen 2 size kept growing but Gen 0 and Gen 1 just bounced around staying around the same level. Private bytes and virtual bytes for the process didn’t change much so this didn’t allow me to go much further with the steps shown in the lab examples.

The threads (~* kb) were sitting in a state which you mentioned in one of your articles could be ignored, and !dumpheap -stat didn’t show one object in particular taking up all the room.

Can you recommend a direction?

Supporting data:

0:000> !address -summary
-------------------- Usage SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots) Pct(Busy)   Usage
    d866000 (  221592) : 10.57%    53.38%    : RegionUsageIsVAD
   66a90000 ( 1681984) : 80.21%    00.00%    : RegionUsageFree
    8fd6000 (  147288) : 07.02%    35.48%    : RegionUsageImage
     6fb000 (    7148) : 00.34%    01.72%    : RegionUsageStack
      1c000 (     112) : 00.01%    00.03%    : RegionUsageTeb
    260a000 (   38952) : 01.86%    09.38%    : RegionUsageHeap
          0 (       0) : 00.00%    00.00%    : RegionUsagePageHeap
       1000 (       4) : 00.00%    00.00%    : RegionUsagePeb
       1000 (       4) : 00.00%    00.00%    : RegionUsageProcessParametrs
       1000 (       4) : 00.00%    00.00%    : RegionUsageEnvironmentBlock
       Tot: 7fff0000 (2097088 KB) Busy: 19560000 (415104 KB)

-------------------- Type SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   66a90000 ( 1681984) : 80.21%   : <free>
    997f000 (  157180) : 07.50%   : MEM_IMAGE
     708000 (    7200) : 00.34%   : MEM_MAPPED
    f4d9000 (  250724) : 11.96%   : MEM_PRIVATE

-------------------- State SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   1266a000 (  301480) : 14.38%   : MEM_COMMIT
   66a90000 ( 1681984) : 80.21%   : MEM_FREE
    6ef6000 (  113624) : 05.42%   : MEM_RESERVE

Largest free region: Base 11810000 - Size 38f00000 (932864 KB)

!dumpheap -stat
7912dae8     2000      1217176 System.Byte[]
79131840     1213      1256668 System.DateTime[]
65410fd0    22839      1278984 System.Data.PropertyCollection
6540a734    29644      1422912 System.Data.Index+IndexTree
65409b14    29644      1660064 System.Data.Index
790fd8c4    37893      2876484 System.String
654088b4    23193      3432564 System.Data.DataColumn
65412bb4     3599      3728564 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]
7912d9bc    27483      4171320 System.Collections.Hashtable+bucket[]
7912d7c0   119608      5075228 System.Int32[]
7912d8f8    98188     11996192 System.Object[]
654359c8    29650     30717400 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
Total 861370 objects

My answer:

You will always see Gen 1 size and Gen 0 size stay pretty flat since there are limit to how big these can grow to be. In fact the Gen 0 size is not really the size of Gen 0, rather it is the budget for Gen 0. These counters are updated at the end of each garbage collection, and at that point Gen 0 will always contain 0 bytes, so the size wouldn’t make much sense…

Since Gen 2 is continuously growing, the interesting question here would be, is it growing at approximately the same rate as the process? If so, you have a managed memory issue. You can also check out .net clr memory\#Bytes in all heaps

The dump you have taken is only at around 300 MB (1266a000 ( 301480) : 14.38% : MEM_COMMIT) which makes it a bit hard to say anything about memory usage since dlls and core parts of the framework will take up a large part of that. What I would recommend that you do is to take multiple memory dumps, preferably 2-3 taken about 100 MB apart, and then compare the size of the dumps or MEM_COMMIT, the GC heap sizes with !eeheap -gc to see if they grow at the same rate, and then finally, if they do, then compare the !dumpheap -stat output to see what objects increase the most.

Just judging by your !dumpheap -stat output you seem to have mostly datasets so it might be worthwhile checking the sizes with !objsize on dataset items and !gcroot them to see why they are sticking around. Having said that though, the dump is pretty small so the amount of dataset related items isn’t by any means alarming. Hopefully looking at the sizes etc. of the datasets you can see if they seem to be an issue, comparing it to what you think the application should use.

Debugging techniques for capturing stacks of OOMS

I’d like know what software or technique you use in OutOfMemoryExceptions while remoting large datasets to capture full stack trace of execution?

Can this technique or software be used for ASP.NET?

My answer:

There are multiple ways to capture the stack trace of an OOM. The technique I used in this case was to attach to the process with windbg (from the debugging tools for windows) and set a breakpoint on mscorwks!ThrowOutOfMemory

bp mscorwks!ThrowOutOfMemory

This is of course only feasible if you can break into the process, which you usually can’t do on a live asp.net application. In an ASP.NET application you could set up an adplus config file with this breakpoint, and have it take a dump when it encountered the breakpoint, or you can set up GCFailFastOnOOM per http://support.microsoft.com/kb/820745

I have been reading your blogs and using your Lab Demos. So far they have been great but I was wondering if you had any recommendations about learning the debugging tool a little better. There are points where I get really confused and lost but I will admit that is because I don’t know or understand a command or two (or three) in the debugger. I kind of jumped into this in the middle so I would like to get “back to the basics” and try and understand the debugger a little better before I start debugging.

Any simple Debugging 101 links you could recommend would be greatly appreciated. In the mean time, looking forward to your posts!

My answer:

For debugging basics, have a look at the following posts

and of course the labs mentioned in the email .NET Debugging Labs - Information and setup instructions

Fragmentation of the LOH

I have been trying to reduce the memory foot print of my application recently and I found something interesting. My application takes advantage of the new Script Combining features in .NET 3.5 SP1 to reduce the number of HTTP requests as well as improve client load time.

After reviewing some memory dumps I am starting to see many large strings stored on the LOH. Many of these strings contain the output of the combined scripts. All of the combined scripts are embedded resources within the System.Web.Extensions.dll as well as our own embedded resources within our custom control library.

Will this cause heavy fragmentation within the LOH? Are you aware of anything I can do to help this?

Supporting data:

0:034> !dumpheap -stat -min 85000
      MT    Count    TotalSize Class Name
7933335c        2      1048608 System.Byte[]
000dbb08       10     13264408      Free
793308ec      432     45672312 System.String
Total 444 objects

My answer:

I haven’t worked a whole lot with this, so I don’t know how often these strings are generated and what the normal size of such a script is, I guess it depends on the content that is being combined. More generally I can say that heavy usage of the LOH will cause pockets of Free memory like this, and the amount of fragmentation you see over time depends on how well new strings fit into these pockets. If you are allocating and de-allocating the same size objects all the time it should be fine. If the get continuously larger you will see more fragmentation.

I would just observe it over time and see if it gets really large. I don’t know how long you have been running at this point, but depending on the circumstances, ~13 MB doesn’t seem extreme. If it gets too bad then the options would be to reduce the size of the combined scripts or finding a way such that you cache the scripts or something so they don’t get generated as often. Not sure how that would be done though without researching more…

Debugging managed code in IE

I have a .NET component that I’ve written using the 1.1 version of the framework. This component gets invoked by Internet Explorer via COM, like an ActiveX control. One of our testers is experiencing some hanging - it takes 3-4 minutes to load something up that comes up instantly on just about every other machine.

I grabbed a memory dump of the process while it was hanging, but nothing in the dump is considered “managed”… so there’s not really much I can do with it. I know that it has loaded up managed code though. Have you ever attempted to debug iexplore.exe like this, and if so, would you be able to explain how you were able to do it?

My answer:

Yes, I have debugged win forms user controls in IE. If by “noting in the dump is considered managed” you mean that all threads are reporting

Thread 27
Not a managed thread.

then I think the simple answer is that either

  • you didn’t get the dump at the time the process hung or
  • you are not hanging in managed code, check out ~* kb to see if any of the threads are doing anything interesting in the native part of the stacks… or
  • the “hang” is outside of the process, i.e. some network latency or similar

Are you perhaps running multiple queries to the same server so that you are trying to load up multiple user controls at once. If I recall correctly you can run into blocking in those situations, but I don’t remember the details.

Getting to dependency properties with sos

I’m trying to debug a WPF application and I want to reach through GCRoot which of my object remain in heap and what of those It’s doesn’t begin disposed, and for this task I want to know the name of the control, but name is a Dependency Property (DP) so it’s a shared property, I can access to the instance of class DepenedecyProperty of Name property, but not to his value, because values are stores in other side. The only interesting filed was GlobalIndexCount. My question is, how access to get a value for a control stored in a DP?, Thank you.

My answer:

Unfortunately I don’t know enough about dependency properties to know how they are stored in memory. I do remember it being tricky though, the few times I have debugged WPF applications. My advice would be create a small sample and hook up visual studio and stop at a place where you have access to the dependency property and use the watch window there to get a better feel for how it is laid out, and then you can bring that knowledge back and use it with windbg and sos in the “real” scenario.

Contact disclaimer

As I have mentioned before, there isn’t enough time in the day to answer all the emails I get through the blog, so I usually only answer emails where I feel that the answer is something that might benefit many people.

If you have an issue, and you’re considering emailing me about it I should mention that you have a better chance of getting the question answered if you write it as a comment on a post it relates to. If you’re in an extreme hurry and/or have a very specific situation, you should definitely call support (see my Contacting Tess… post for more info)

Laters, Tess