Q&A Reader emails about .NET memory leaks and random questions
Here are of some of the reader emails I got this week and my answers to them…
- How do I troubleshoot this memory leak
- Debugging techniques for capturing stacks of OOMS
- Do you have a list of Debugging 101 links?
- Debugging managed code in IE
- Getting to dependency properties with sos
let me know if this is something you find useful so I should keep doing it from time to time, or if it’s only useful for the selected few who’s email I happened to pick this week:)
Note: some of the emails have been edited to fit better in a Q/A format, hopefully I haven’t missed any important bits…
How do I troubleshoot this memory leak
I started observing a slow leak in task manager of about 100 MB / 30 mins, in a web application consisting of WCF services.
In perfmon Gen 0, 1, and 2 collections steadily grew, and Gen 2 size kept growing but Gen 0 and Gen 1 just bounced around staying around the same level. Private bytes and virtual bytes for the process didn’t change much so this didn’t allow me to go much further with the steps shown in the lab examples.
The threads (~* kb) were sitting in a state which you mentioned in one of your articles could be ignored, and !dumpheap -stat didn’t show one object in particular taking up all the room.
Can you recommend a direction?
Supporting data:
0:000> !address -summary
-------------------- Usage SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Pct(Busy) Usage
d866000 ( 221592) : 10.57% 53.38% : RegionUsageIsVAD
66a90000 ( 1681984) : 80.21% 00.00% : RegionUsageFree
8fd6000 ( 147288) : 07.02% 35.48% : RegionUsageImage
6fb000 ( 7148) : 00.34% 01.72% : RegionUsageStack
1c000 ( 112) : 00.01% 00.03% : RegionUsageTeb
260a000 ( 38952) : 01.86% 09.38% : RegionUsageHeap
0 ( 0) : 00.00% 00.00% : RegionUsagePageHeap
1000 ( 4) : 00.00% 00.00% : RegionUsagePeb
1000 ( 4) : 00.00% 00.00% : RegionUsageProcessParametrs
1000 ( 4) : 00.00% 00.00% : RegionUsageEnvironmentBlock
Tot: 7fff0000 (2097088 KB) Busy: 19560000 (415104 KB)
-------------------- Type SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
66a90000 ( 1681984) : 80.21% : <free>
997f000 ( 157180) : 07.50% : MEM_IMAGE
708000 ( 7200) : 00.34% : MEM_MAPPED
f4d9000 ( 250724) : 11.96% : MEM_PRIVATE
-------------------- State SUMMARY --------------------------
TotSize ( KB) Pct(Tots) Usage
1266a000 ( 301480) : 14.38% : MEM_COMMIT
66a90000 ( 1681984) : 80.21% : MEM_FREE
6ef6000 ( 113624) : 05.42% : MEM_RESERVE
Largest free region: Base 11810000 - Size 38f00000 (932864 KB)
!dumpheap -stat
...
7912dae8 2000 1217176 System.Byte[]
79131840 1213 1256668 System.DateTime[]
65410fd0 22839 1278984 System.Data.PropertyCollection
6540a734 29644 1422912 System.Data.Index+IndexTree
65409b14 29644 1660064 System.Data.Index
790fd8c4 37893 2876484 System.String
654088b4 23193 3432564 System.Data.DataColumn
65412bb4 3599 3728564 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]
7912d9bc 27483 4171320 System.Collections.Hashtable+bucket[]
7912d7c0 119608 5075228 System.Int32[]
7912d8f8 98188 11996192 System.Object[]
654359c8 29650 30717400 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
Total 861370 objects
My answer:
You will always see Gen 1 size and Gen 0 size stay pretty flat since there are limit to how big these can grow to be. In fact the Gen 0 size is not really the size of Gen 0, rather it is the budget for Gen 0. These counters are updated at the end of each garbage collection, and at that point Gen 0 will always contain 0 bytes, so the size wouldn’t make much sense…
Since Gen 2 is continuously growing, the interesting question here would be, is it growing at approximately the same rate as the process? If so, you have a managed memory issue. You can also check out .net clr memory\#Bytes in all heaps
The dump you have taken is only at around 300 MB (1266a000 ( 301480) : 14.38% : MEM_COMMIT) which makes it a bit hard to say anything about memory usage since dlls and core parts of the framework will take up a large part of that. What I would recommend that you do is to take multiple memory dumps, preferably 2-3 taken about 100 MB apart, and then compare the size of the dumps or MEM_COMMIT, the GC heap sizes with !eeheap -gc to see if they grow at the same rate, and then finally, if they do, then compare the !dumpheap -stat output to see what objects increase the most.
Just judging by your !dumpheap -stat output you seem to have mostly datasets so it might be worthwhile checking the sizes with !objsize on dataset items and !gcroot them to see why they are sticking around. Having said that though, the dump is pretty small so the amount of dataset related items isn’t by any means alarming. Hopefully looking at the sizes etc. of the datasets you can see if they seem to be an issue, comparing it to what you think the application should use.
Debugging techniques for capturing stacks of OOMS
I’d like know what software or technique you use in OutOfMemoryExceptions while remoting large datasets to capture full stack trace of execution?
Can this technique or software be used for ASP.NET?
My answer:
There are multiple ways to capture the stack trace of an OOM. The technique I used in this case was to attach to the process with windbg (from the debugging tools for windows) and set a breakpoint on mscorwks!ThrowOutOfMemory
bp mscorwks!ThrowOutOfMemory
This is of course only feasible if you can break into the process, which you usually can’t do on a live asp.net application. In an ASP.NET application you could set up an adplus config file with this breakpoint, and have it take a dump when it encountered the breakpoint, or you can set up GCFailFastOnOOM per http://support.microsoft.com/kb/820745
Do you have a list of Debugging 101 links?
I have been reading your blogs and using your Lab Demos. So far they have been great but I was wondering if you had any recommendations about learning the debugging tool a little better. There are points where I get really confused and lost but I will admit that is because I don’t know or understand a command or two (or three) in the debugger. I kind of jumped into this in the middle so I would like to get “back to the basics” and try and understand the debugger a little better before I start debugging.
Any simple Debugging 101 links you could recommend would be greatly appreciated. In the mean time, looking forward to your posts!
My answer:
For debugging basics, have a look at the following posts
- Back to Basics - How do I get the memory dumps in the first place? And what is SOS.dll?
- Learning .NET debugging
- .NET Hang Debugging Walkthrough
- ASP.NET Memory Investigation
and of course the labs mentioned in the email .NET Debugging Labs - Information and setup instructions
Fragmentation of the LOH
I have been trying to reduce the memory foot print of my application recently and I found something interesting. My application takes advantage of the new Script Combining features in .NET 3.5 SP1 to reduce the number of HTTP requests as well as improve client load time.
After reviewing some memory dumps I am starting to see many large strings stored on the LOH. Many of these strings contain the output of the combined scripts. All of the combined scripts are embedded resources within the System.Web.Extensions.dll as well as our own embedded resources within our custom control library.
Will this cause heavy fragmentation within the LOH? Are you aware of anything I can do to help this?
Supporting data:
0:034> !dumpheap -stat -min 85000
Statistics:
MT Count TotalSize Class Name
7933335c 2 1048608 System.Byte[]
000dbb08 10 13264408 Free
793308ec 432 45672312 System.String
Total 444 objects
My answer:
I haven’t worked a whole lot with this, so I don’t know how often these strings are generated and what the normal size of such a script is, I guess it depends on the content that is being combined. More generally I can say that heavy usage of the LOH will cause pockets of Free memory like this, and the amount of fragmentation you see over time depends on how well new strings fit into these pockets. If you are allocating and de-allocating the same size objects all the time it should be fine. If the get continuously larger you will see more fragmentation.
I would just observe it over time and see if it gets really large. I don’t know how long you have been running at this point, but depending on the circumstances, ~13 MB doesn’t seem extreme. If it gets too bad then the options would be to reduce the size of the combined scripts or finding a way such that you cache the scripts or something so they don’t get generated as often. Not sure how that would be done though without researching more…
Debugging managed code in IE
I have a .NET component that I’ve written using the 1.1 version of the framework. This component gets invoked by Internet Explorer via COM, like an ActiveX control. One of our testers is experiencing some hanging - it takes 3-4 minutes to load something up that comes up instantly on just about every other machine.
I grabbed a memory dump of the process while it was hanging, but nothing in the dump is considered “managed”… so there’s not really much I can do with it. I know that it has loaded up managed code though. Have you ever attempted to debug iexplore.exe like this, and if so, would you be able to explain how you were able to do it?
My answer:
Yes, I have debugged win forms user controls in IE. If by “noting in the dump is considered managed” you mean that all threads are reporting
Thread 27
Not a managed thread.
then I think the simple answer is that either
- you didn’t get the dump at the time the process hung or
- you are not hanging in managed code, check out ~* kb to see if any of the threads are doing anything interesting in the native part of the stacks… or
- the “hang” is outside of the process, i.e. some network latency or similar
Are you perhaps running multiple queries to the same server so that you are trying to load up multiple user controls at once. If I recall correctly you can run into blocking in those situations, but I don’t remember the details.
Getting to dependency properties with sos
I’m trying to debug a WPF application and I want to reach through GCRoot which of my object remain in heap and what of those It’s doesn’t begin disposed, and for this task I want to know the name of the control, but name is a Dependency Property (DP) so it’s a shared property, I can access to the instance of class DepenedecyProperty of Name property, but not to his value, because values are stores in other side. The only interesting filed was GlobalIndexCount. My question is, how access to get a value for a control stored in a DP?, Thank you.
My answer:
Unfortunately I don’t know enough about dependency properties to know how they are stored in memory. I do remember it being tricky though, the few times I have debugged WPF applications. My advice would be create a small sample and hook up visual studio and stop at a place where you have access to the dependency property and use the watch window there to get a better feel for how it is laid out, and then you can bring that knowledge back and use it with windbg and sos in the “real” scenario.
Contact disclaimer
As I have mentioned before, there isn’t enough time in the day to answer all the emails I get through the blog, so I usually only answer emails where I feel that the answer is something that might benefit many people.
If you have an issue, and you’re considering emailing me about it I should mention that you have a better chance of getting the question answered if you write it as a comment on a post it relates to. If you’re in an extreme hurry and/or have a very specific situation, you should definitely call support (see my Contacting Tess… post for more info)
Laters, Tess
