I have written before about high memory usage caused by improper usage of XmlSerializer objects both in a case study and in a debugging lab. The problem there was that every time you create a new XmlSerializer object with a non-default constructor, you generate a new dynamic assembly that contains the definition and methods for the serializer. Since assemblies can’t be unloaded from a process unless the application domain they are loaded in is unloaded memory will keep increasing if you create new XmlSerializer objects until eventually you end up with a System.OutOfMemoryException.
In the case of the XmlSerializer dynamic assemblies, they are quite easy to spot as you can look at meta-data (see previous posts) and figure out that they are XmlSerializer generated assemblies, and you can even see what type is being serialized.
This time I’m going to walk through a similar issue where unfortunately the dynamic assembly generator is a bit more difficult to spot.
Any issue where you are leaking dynamic assemblies will have very similar symptoms and they go something like this
- The perfmon counter
Process/Private Byteskeeps increasing (Mem Usage in task manager will keep increasing too for that matter)
.NET CLR Memory/#Bytes in all heapsmay increase but unless you also have a .net object “memory leak”
#Bytes in all heapsshould not follow the Private Bytes pattern as the dynamic assemblies are not stored on the GC heaps. For a typical assembly leak problem,
#Bytes in all heapswill be relatively low compared to Private Bytes.
.NET CLR Loading/Current assemblieskeeps increasing throughout the lifetime of the process. Normally the number of loaded assemblies should flatten out once all components in the process are loaded so if Current assemblies keeps increasing, that is a pretty sure sign that you have an assembly leak.
Once memory usage for the process is fairly high, you can capture a memory dump of the process with any of your favorite tools (debug diag, adplus, task manager, procdmp etc.). Using debug diag for example, you just right-click the w3wp.exe process in the processes view and select the “Create Full Userdump” option.
In my case,
Private Bytes was around 790 MB,
#Bytes in all heaps was 28 MB and I had 1031
If you only have a memory dump but no perfmon counters, you can get this data from the dump as well if you open it up in windbg.
Note: for some commands in this article you will need to load psscor.dll in windbg.
0:026> !address -summary ... -------------------- State SUMMARY -------------------------- TotSize ( KB) Pct(Tots) Usage 3081a000 ( 794728) : 18.95% : MEM_COMMIT <- very similar to private bytes a5eaf000 ( 2718396) : 64.81% : MEM_FREE 29927000 ( 681116) : 16.24% : MEM_RESERVE ...
0:026> !eeheap -gc Number of GC Heaps: 2 ------------------------------ ... GC Heap Size 0x1b5396c(28,653,932) <- #Bytes in all heaps
For current assemblies, you can’t easily get the total number of assemblies, but
!dumpdomain –stat will tell you how many assemblies are loaded in each domain. Note however that there may be some overlap here as some assemblies will be listed in multiple domains. Also, the size of the dynamic assemblies may or may not show up here in the size assemblies column.
The cool thing about this output though is that you can easily identify the domain that is leaking assemblies. In this case it is probably not all that surprising that it happens to be my XSLMemoryLeak demo application where I am reproducing the problem :)
0:026> !dumpdomain -stat Domain Num Assemblies Size Assemblies Name 7a3bd058 0 0 System Domain 7a3bc9a8 27 85,086,720 Shared Domain 0019f280 9 48,486,400 DefaultDomain 001d3038 1,031 85,100,032 /LM/w3svc/1/ROOT/XSLMemoryLeak-1-129175353121633629
What are these assemblies?
If you don’t care for a deep dive in potentially non-interesting windbg / psscor gory details, feel free to move right to the conclusion at the bottom of the post :)
Time for a new function in psscor called
!dda for those of us who don’t want to end up with COBOL fingers).
This function has two purposes
- list all the dynamic assemblies in the process, or
- save all dynamic assemblies to disc using !dda –save, so that you can open them up in reflector or similar
Running this on the dump I can see that there is a total of 1002 dynamic assemblies in the dump and that all of them are in the XSLMemoryLeak domain. Each assembly seems to have 2 modules associated with it so there is a total of 2004 modules associated with these assemblies.
0:026> !dda Domain: DefaultDomain ------------------- Domain: /LM/w3svc/1/ROOT/XSLMemoryLeak-1-129175353121633629 ------------------- Assembly: 0x10514528  Dynamic Module: 0x02af8740 loaded at: 0x00000000 Size: 0x00000000(0) Assembly: 0x10514528  Dynamic Module: 0x02af8ac0 loaded at: 0x00000000 Size: 0x00000000(0) Assembly: 0x10589558  Dynamic Module: 0x106122c4 loaded at: 0x00000000 Size: 0x00000000(0) Assembly: 0x10589558  Dynamic Module: 0x10612644 loaded at: 0x00000000 Size: 0x00000000(0) ... Assembly: 0x3472c168  Dynamic Module: 0x53402e8c loaded at: 0x00000000 Size: 0x00000000(0) Assembly: 0x3472c168  Dynamic Module: 0x5340320c loaded at: 0x00000000 Size: 0x00000000(0) Assembly: 0x347728b8  Dynamic Module: 0x53403630 loaded at: 0x00000000 Size: 0x00000000(0) Assembly: 0x347728b8  Dynamic Module: 0x534039b0 loaded at: 0x00000000 Size: 0x00000000(0) Assembly: 0x347b9010  Dynamic Module: 0x53403dd4 loaded at: 0x00000000 Size: 0x00000000(0) Assembly: 0x347b9010  Dynamic Module: 0x53404154 loaded at: 0x00000000 Size: 0x00000000(0) -------------------------------------- Total 1,002 Dynamic Assemblies, Total size: 0x0(0) bytes. =======================================
Unfortunately, because the reported size is 0, saving them out will do me no good as they don’t have any meta-data so I will have to find another way to figure out what they are.
I was working on this issue with my colleague Doug and after doing some random “checking around the dump to see if we can find anything even remotely interesting” we noticed that on the heap we had exactly the same amount of
System.Reflection.Emit.ModuleBuilderData objects as the number of modules listed above.
0:026> !dumpheap -stat -type System.Reflection.Emit* Loading the heap objects into our cache. ------------------------------ Heap 0 total 38,045 objects ------------------------------ Heap 1 total 40,553 objects ------------------------------ total 78,598 objects Statistics: MT Count TotalSize Change Class Name 0x79305120 2,004 64,128 2,004 System.Reflection.Emit.AssemblyBuilder 0x79318c9c 2,004 88,176 2,004 System.Reflection.Emit.ModuleBuilderData
From this we were able to get some very important data.
First we dumped out the
0:026> !dumpheap -type System.Reflection.Emit.ModuleBuilderData Loading the heap objects into our cache. ------------------------------ Address MT Size ... 07c8bc98 79318c9c 44 0 System.Reflection.Emit.ModuleBuilderData 07c8c6ec 79318c9c 44 0 System.Reflection.Emit.ModuleBuilderData 07cc04d8 79318c9c 44 0 System.Reflection.Emit.ModuleBuilderData 07cc0f2c 79318c9c 44 0 System.Reflection.Emit.ModuleBuilderData
And took a closer look at one of them
0:026> !do 07cc0f2c Name: System.Reflection.Emit.ModuleBuilderData ... Fields: MT Field Offset Type VT Attr Value Name 793308ec 4002612 4 System.String 0 instance 02dc67fc m_strModuleName 793308ec 4002613 8 System.String 0 instance 02dc67fc m_strFileName ... 79318b90 4002617 10 ...mit.ModuleBuilder 0 instance 07cc0c94 m_module 79332b38 4002618 20 System.Int32 1 instance 0 m_tkFile ...
m_strFileName proved to be a crucial piece of information telling us that the module was called
System.Xml.Xsl.CompiledQuery so now we know that they are generated when loading XSL Transforms
0:026> !do 02dc67fc Name: System.String MethodTable: 793308ec EEClass: 790ed64c Size: 74(0x4a) bytes GC Generation: 2 String: System.Xml.Xsl.CompiledQuery ...
But how can we prove that these
ModuleBuilderData objects are really related to the dynamic assemblies?
For this we need to look closer at the
m_module field and dump out this
ModuleBuilder and later also its
0:026> !do 07cc0c94 Name: System.Reflection.Emit.ModuleBuilder ... Fields: MT Field Offset Type VT Attr Value Name ... 79318b90 400260e 2c ...mit.ModuleBuilder 0 instance 07cc0c5c m_internalModuleBuilder 79305120 400260f 30 ...t.AssemblyBuilder 0 instance 07cc0418 m_assemblyBuilder ...
0:026> !do 07cc0c5c Name: System.Reflection.Emit.ModuleBuilder ... Fields: MT Field Offset Type VT Attr Value Name ... 793331b4 4000d21 14 System.IntPtr 1 instance 0 m__pRefClass 793331b4 4000d22 18 System.IntPtr 1 instance 1396719956 m__pData 793331b4 4000d23 1c System.IntPtr 1 instance 669415152 m__pInternalSymWriter 793331b4 4000d24 20 System.IntPtr 1 instance 0 m__pGlobals ...
Don’t ask me how you come up with this and the next part :) This is the fruit of some serious desperation, trying to find anything that fits.
We then see that the
m__pData member variable is an
IntPtr so we can turn this into hex by running
0:026> ?0n1396719956 Evaluate expression: 1396719956 = 53404154
And voilà, this just happens to be the address of one of the dynamic assembly modules, so Houston, we have a match…
Assembly: 0x347b9010  Dynamic Module: 0x53403dd4 loaded at: 0x00000000 Size: 0x00000000(0) Assembly: 0x347b9010  Dynamic Module: 0x53404154 loaded at: 0x00000000 Size: 0x00000000(0)
Ok, so all that crazy debugging to find out that the dynamic assemblies were related to XSL Transforms.
As it turns out, if you create a new
XslCompiledTransform(true) where true means enable debugging, a new dynamic assembly will be generated when you load the stylesheet.
On the other hand if this is set to false, or if you use new
XslCompiledTransform() this will not happen, so just remember to turn it off when you go into production, not only for this but to stop emitting debug info at that point.
Have a good one, Tess