This is the last debugging lab in the .NET Debugging Labs series. By now you should have the basics down for troubleshooting hangs, perf issues, memory leaks and crashes in .net applications. Hope you have enjoyed your debugging sessions.
The last one in the series is a managed memory leak caused by holding on to resources in an unexpected way. Since it is the last one I have tried to make the questions a little bit less leading than the previous ones :)
Previous labs and setup instructions
If you are new to the debugging labs, here you can find information on how to set up the labs as well as links to the previous labs in the series.
- Information and setup instructions
- Lab 1: Hang
- Lab 2: Crash
- Lab 3: Memory
- Lab 4: High CPU hang
- Lab 5: Crash
- Lab 6: Memory
The problem description is very similar to Lab 6.
We have started getting out of memory exceptions on the Buggy Bits site and we have been able to determine a scenario in which we think we are leaking memory but we can’t seem to figure out where the memory is going.
The leak seems to be occurring on our News page for example and we can reproduce it by stress testing.
It seems like it is leaking just a small bit every time but since it is something that customers look at a lot and over time the process will crash with an out of memory exception.
Reproduce the issue and gather data
- Start the Buggy Bits application and browse to the News site
- Start monitoring performance counters with
Stress the application
.\tinyget.ps1 -url https://localhost:44350/News -numTimes 1000
After tinyget has finished, capture a memory dump
dotnet-dump collect -n iisexpress
or with procdump
procdump64.exe -ma iisexpress.exe
Review the performance counters to figure out what we are leaking
GC Heap Sizeand
Number of Assemblies Loaded
- Can you tell if the issue we are facing is a virtual bytes leak, a native leak or a .NET leak?
Debug the memory dump
Open the memory dump in WinDbg, load up the symbols and load sos.dll (see information and setup instructions for more info)
- What is the size of the memory dump (on disk)
- What is the size of the .NET heap according to !eeheap -gc?
- Is most of the memory stored on the regular heaps or on the large object heap?
We saw from the performance counters that we appeared to be leaking .net memory, so the next step is to determine what the memory is used for.
- What types of objects seem to use up most of the memory?
- Looking at the 10-20 bottommost object types in
!dumpheap -stat, can you see any patterns among the objects? I.e. can they be grouped in any way?
Normally you will be able to see patterns such as many data related objects, many ui related objects, many xml related objects etc. like in this memory investigation
- Looking at the 10-20 bottom most object types in
!dumpheap -stat, do the quantities of each of them seem normal or does anything seem out of the ordinary?
Dump the large objects using
!dumpheap -min 0n85000
- What type of objects are stored on the Large Object Heap (LOH)?
Take any of the objects on the LOH and run
!gcroot <object address>to find out where they are rooted
- What types of roots would you normally see when running !gcroot and what do they mean?
An example of a root chain would look like this (if we are looking to see why the string at address
02ebf628is still around)
!gcroot 02ebf628 ... DOMAIN(001CCA88):HANDLE(Strong):2c41188:Root: 02ec05c0(MyNamespace.Person)-> 02ec0578(MyNamespace.Address)-> 02ebf628(System.String)
In this case we have a strongly rooted (static) object of type
Personhas a member variable of type
Addresshas a member variable of type
02ebf628, so the reason that the string is still around is because it is linked to an Address that is linked to a Person and that Person is stored as a static variable. Unless the chain is broken the string will not be available for collection until the application domain is recycled since static objects never go out of scope.
Look at the root chain of the object that you did
- Why is it sticking around? How could the root chain be broken?
Note: If you walk it from the bottom up and reach an object type that you don’t recognize, look it up on the docs to get more info about it in order to understand why there is a link.
Putting it all together and determining the cause of the memory leak
Look at the code for the News controller and use the knowledge you gathered from the debugging session to figure out how the leak was generated.
- How long will the objects stick around?
- What can you do to avoid this type of event handler leak?
Resolve the issue and rerun the test to verify the solution
- Search this site or the Microsoft docs to find a resolution to the problem and resolve the issue
- Rerun the test to verify that the assembly “leak” no longer exists.