.NET Memory Leak: XslCompiledTransform and leaked dynamic assemblies

I have written before about high memory usage caused by improper usage of XmlSerializer objects both in a case study and in a debugging lab. The problem there was that every time you create a new XmlSerializer object with a non-default constructor, you generate a new dynamic assembly that contains the definition and methods for the serializer. Since assemblies can’t be unloaded from a process unless the application domain they are loaded in is unloaded memory will keep increasing if you create new XmlSerializer objects until eventually you end up with a System.OutOfMemoryException.

In the case of the XmlSerializer dynamic assemblies, they are quite easy to spot as you can look at meta-data (see previous posts) and figure out that they are XmlSerializer generated assemblies, and you can even see what type is being serialized.

This time I’m going to walk through a similar issue where unfortunately the dynamic assembly generator is a bit more difficult to spot.

Symptoms

Any issue where you are leaking dynamic assemblies will have very similar symptoms and they go something like this

1. The perfmon counter Process/Private Bytes keeps increasing (Mem Usage in task manager will keep increasing too for that matter)
2. .NET CLR Memory/#Bytes in all heaps may increase but unless you also have a .net object “memory leak” #Bytes in all heaps should not follow the Private Bytes pattern as the dynamic assemblies are not stored on the GC heaps. For a typical assembly leak problem, #Bytes in all heaps will be relatively low compared to Private Bytes.
3. .NET CLR Loading/Current assemblies keeps increasing throughout the lifetime of the process. Normally the number of loaded assemblies should flatten out once all components in the process are loaded so if Current assemblies keeps increasing, that is a pretty sure sign that you have an assembly leak.

Debugging

Once memory usage for the process is fairly high, you can capture a memory dump of the process with any of your favorite tools (debug diag, adplus, task manager, procdmp etc.). Using debug diag for example, you just right-click the w3wp.exe process in the processes view and select the “Create Full Userdump” option.

In my case, Private Bytes was around 790 MB, #Bytes in all heaps was 28 MB and I had 1031 Current assemblies

If you only have a memory dump but no perfmon counters, you can get this data from the dump as well if you open it up in windbg.

Note: for some commands in this article you will need to load psscor.dll in windbg.

0:026> !address -summary
...
-------------------- State SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   3081a000 (  794728) : 18.95%   : MEM_COMMIT      <- very similar to private bytes
   a5eaf000 ( 2718396) : 64.81%   : MEM_FREE
   29927000 (  681116) : 16.24%   : MEM_RESERVE
...

0:026> !eeheap -gc
Number of GC Heaps: 2
------------------------------
...
GC Heap Size  0x1b5396c(28,653,932)                <- #Bytes in all heaps

For current assemblies, you can’t easily get the total number of assemblies, but !dumpdomain –stat will tell you how many assemblies are loaded in each domain. Note however that there may be some overlap here as some assemblies will be listed in multiple domains. Also, the size of the dynamic assemblies may or may not show up here in the size assemblies column.

The cool thing about this output though is that you can easily identify the domain that is leaking assemblies. In this case it is probably not all that surprising that it happens to be my XSLMemoryLeak demo application where I am reproducing the problem :)

0:026> !dumpdomain -stat
    Domain     Num Assemblies     Size Assemblies     Name
7a3bd058                  0                   0     System Domain
7a3bc9a8                 27          85,086,720     Shared Domain
0019f280                  9          48,486,400     DefaultDomain
001d3038              1,031          85,100,032     /LM/w3svc/1/ROOT/XSLMemoryLeak-1-129175353121633629

What are these assemblies?

If you don’t care for a deep dive in potentially non-interesting windbg / psscor gory details, feel free to move right to the conclusion at the bottom of the post :)

Time for a new function in psscor called !dumpdynamicassemblies (or !dda for those of us who don’t want to end up with COBOL fingers).

This function has two purposes

1. list all the dynamic assemblies in the process, or
2. save all dynamic assemblies to disc using !dda –save, so that you can open them up in reflector or similar

Running this on the dump I can see that there is a total of 1002 dynamic assemblies in the dump and that all of them are in the XSLMemoryLeak domain. Each assembly seems to have 2 modules associated with it so there is a total of 2004 modules associated with these assemblies.

0:026> !dda
Domain: DefaultDomain
-------------------
Domain: /LM/w3svc/1/ROOT/XSLMemoryLeak-1-129175353121633629
-------------------
Assembly: 0x10514528 [] Dynamic Module: 0x02af8740 loaded at: 0x00000000 Size: 0x00000000(0)
Assembly: 0x10514528 [] Dynamic Module: 0x02af8ac0 loaded at: 0x00000000 Size: 0x00000000(0)
Assembly: 0x10589558 [] Dynamic Module: 0x106122c4 loaded at: 0x00000000 Size: 0x00000000(0)
Assembly: 0x10589558 [] Dynamic Module: 0x10612644 loaded at: 0x00000000 Size: 0x00000000(0)
...
Assembly: 0x3472c168 [] Dynamic Module: 0x53402e8c loaded at: 0x00000000 Size: 0x00000000(0)
Assembly: 0x3472c168 [] Dynamic Module: 0x5340320c loaded at: 0x00000000 Size: 0x00000000(0)
Assembly: 0x347728b8 [] Dynamic Module: 0x53403630 loaded at: 0x00000000 Size: 0x00000000(0)
Assembly: 0x347728b8 [] Dynamic Module: 0x534039b0 loaded at: 0x00000000 Size: 0x00000000(0)
Assembly: 0x347b9010 [] Dynamic Module: 0x53403dd4 loaded at: 0x00000000 Size: 0x00000000(0)
Assembly: 0x347b9010 [] Dynamic Module: 0x53404154 loaded at: 0x00000000 Size: 0x00000000(0)
--------------------------------------
Total 1,002 Dynamic Assemblies, Total size: 0x0(0) bytes.
=======================================

Unfortunately, because the reported size is 0, saving them out will do me no good as they don’t have any meta-data so I will have to find another way to figure out what they are.

I was working on this issue with my colleague Doug and after doing some random “checking around the dump to see if we can find anything even remotely interesting” we noticed that on the heap we had exactly the same amount of System.Reflection.Emit.ModuleBuilderData objects as the number of modules listed above.

0:026> !dumpheap -stat -type System.Reflection.Emit*
Loading the heap objects into our cache.
------------------------------
Heap 0
total 38,045 objects
------------------------------
Heap 1
total 40,553 objects
------------------------------
total 78,598 objects

Statistics:
        MT    Count    TotalSize       Change Class Name
0x79305120    2,004       64,128        2,004 System.Reflection.Emit.AssemblyBuilder
0x79318c9c    2,004       88,176        2,004 System.Reflection.Emit.ModuleBuilderData

From this we were able to get some very important data.

First we dumped out the ModuleBuilderData objects

0:026> !dumpheap -type System.Reflection.Emit.ModuleBuilderData
Loading the heap objects into our cache.
------------------------------
   Address         MT     Size
...
07c8bc98 79318c9c       44    0 System.Reflection.Emit.ModuleBuilderData
07c8c6ec 79318c9c       44    0 System.Reflection.Emit.ModuleBuilderData
07cc04d8 79318c9c       44    0 System.Reflection.Emit.ModuleBuilderData
07cc0f2c 79318c9c       44    0 System.Reflection.Emit.ModuleBuilderData

And took a closer look at one of them

0:026> !do 07cc0f2c
Name: System.Reflection.Emit.ModuleBuilderData
...
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
793308ec  4002612        4        System.String  0 instance 02dc67fc m_strModuleName
793308ec  4002613        8        System.String  0 instance 02dc67fc m_strFileName
...
79318b90  4002617       10 ...mit.ModuleBuilder  0 instance 07cc0c94 m_module
79332b38  4002618       20         System.Int32  1 instance        0 m_tkFile
...

The m_strFileName proved to be a crucial piece of information telling us that the module was called System.Xml.Xsl.CompiledQuery so now we know that they are generated when loading XSL Transforms

0:026> !do 02dc67fc
Name: System.String
MethodTable: 793308ec
EEClass: 790ed64c
Size: 74(0x4a) bytes
GC Generation: 2
String: System.Xml.Xsl.CompiledQuery
...

But how can we prove that these ModuleBuilderData objects are really related to the dynamic assemblies?

For this we need to look closer at the ModuleBuilderData objects m_module field and dump out this ModuleBuilder and later also its m_internalModuleBuilder

0:026> !do 07cc0c94
Name: System.Reflection.Emit.ModuleBuilder
...
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
...
79318b90  400260e       2c ...mit.ModuleBuilder  0 instance 07cc0c5c m_internalModuleBuilder
79305120  400260f       30 ...t.AssemblyBuilder  0 instance 07cc0418 m_assemblyBuilder
...

0:026> !do 07cc0c5c
Name: System.Reflection.Emit.ModuleBuilder
...
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
...
793331b4  4000d21       14        System.IntPtr  1 instance        0 m__pRefClass
793331b4  4000d22       18        System.IntPtr  1 instance 1396719956 m__pData
793331b4  4000d23       1c        System.IntPtr  1 instance 669415152 m__pInternalSymWriter
793331b4  4000d24       20        System.IntPtr  1 instance        0 m__pGlobals
...

Don’t ask me how you come up with this and the next part :) This is the fruit of some serious desperation, trying to find anything that fits.

We then see that the m__pData member variable is an IntPtr so we can turn this into hex by running

0:026> ?0n1396719956
Evaluate expression: 1396719956 = 53404154

And voilà, this just happens to be the address of one of the dynamic assembly modules, so Houston, we have a match…

Assembly: 0x347b9010 [] Dynamic Module: 0x53403dd4 loaded at: 0x00000000 Size: 0x00000000(0)
Assembly: 0x347b9010 [] Dynamic Module: 0x53404154 loaded at: 0x00000000 Size: 0x00000000(0)

Conclusion

Ok, so all that crazy debugging to find out that the dynamic assemblies were related to XSL Transforms.

As it turns out, if you create a new XslCompiledTransform(true) where true means enable debugging, a new dynamic assembly will be generated when you load the stylesheet.

On the other hand if this is set to false, or if you use new XslCompiledTransform() this will not happen, so just remember to turn it off when you go into production, not only for this but to stop emitting debug info at that point.

Have a good one, Tess