8 minute read

I was helping a colleague out with an OOM (OutOfMemory) situation he was dealing with.

Problem description

Their applications memory usage would grow over time until they finally ended up with an out of memory exception.

First debug

They had gotten a memory dump when memory usage was really high 1.4 GB using debug diag and I opened it up in windbg.exe, loaded up sos (.loadby sos mscorwks) and ran !dumpheap -stat to get the content of the GC heaps.

0:028> !dumpheap -stat
Statistics:
      MT    Count    TotalSize Class Name
...
7ae77c54   59,028    1,416,672 System.Drawing.Color
663bb5a0   16,530    1,454,640 System.Web.UI.WebControls.RequiredFieldValidator
66411ac8   67,503    1,620,072 System.Web.UI.WebControls.Unit
663bd0a0   20,581    1,646,480 System.Web.UI.WebControls.Label
663bc55c   17,218    1,721,800 System.Web.UI.WebControls.DataGridItem
663c3f04  148,559    1,782,708 System.Web.UI.WebControls.FontInfo
7912dd40   16,219    1,937,904 System.Char[]
7a75a878  126,071    2,017,136 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
6640a7cc   32,408    2,074,112 System.Web.UI.HtmlControls.HtmlGenericControl
65431bb4    3,483    2,185,332 System.Collections.Generic.Dictionary`2+Entry[[System.Data.DataRow, System.Data],[System.Data.DataRowView, System.Data]][]
6540b178   31,055    2,235,960 System.Data.DataColumnPropertyDescriptor
79101fe4   50,490    2,827,440 System.Collections.Hashtable
65421898   59,458    2,853,984 System.Data.Common.ObjectStorage
6540addc  121,403    2,913,672 System.Data.DataRowView
6540b09c   71,366    3,140,104 System.Data.Common.StringStorage
79131488    3,149    3,239,368 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.Resources.ResourceLocator, mscorlib]][]
664162d4  104,417    3,759,012 System.Web.UI.EmptyControlCollection
66401a78   98,714    4,343,416 System.Web.UI.WebControls.Style
663eb1d0  283,592    4,537,472 System.Web.UI.AttributeCollection
79102290  389,910    4,678,920 System.Int32
79131840    4,588    4,753,168 System.DateTime[]
663dd2b0  110,687    4,870,228 System.Web.UI.WebControls.TableItemStyle
65407d48   16,545    4,897,320 System.Data.DataTable
664140a4   89,233    5,353,980 System.Web.UI.LiteralControl
79104368  247,402    5,937,648 System.Collections.ArrayList
66412f04  227,678    6,374,984 System.Web.UI.WebControls.ListItem
7912d7c0  107,276    8,479,900 System.Int32[]
663c7308  297,688   10,716,768 System.Web.UI.ControlCollection
6641194c  700,898   11,214,368 System.Web.UI.StateBag
7912dae8   14,050   11,661,948 System.Byte[]
7a7580d0  764,922   15,298,440 System.Collections.Specialized.HybridDictionary
663d7328  219,634   18,449,256 System.Web.UI.WebControls.TableCell
7912d9bc   54,737   18,666,456 System.Collections.Hashtable+bucket[]
663c1de8 1,203,791   19,260,656 System.Web.UI.StateItem
7a75820c  690,058   19,321,624 System.Collections.Specialized.ListDictionary
6641f33c  444,421   19,554,524 System.Web.UI.Control+OccasionalFields
7a7582d8 1,198,303   23,966,060 System.Collections.Specialized.ListDictionary+DictionaryNode
79105a0c 1,048,528   25,164,672 System.Guid
654088b4  170,538   25,239,624 System.Data.DataColumn
654359c8   11,068   25,859,792 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
65412bb4   24,380   42,023,632 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]
65408b8c  761,406   48,729,984 System.Data.DataRow
7912d8f8  579,883  115,071,292 System.Object[]
000dc6e8      247  241,601,256      Free
790fd8c4 3,524,118  373,698,368 System.String
Total 15,860,201 objects, Total size: 1,214,968,272

The output above, showing the most memory consuming objects, tells us that there are pretty much two types of objects that consume most of the memory. Data related items and UI related items.

Notes about finding lots of UI objects on the heap

If you have followed my blog, you have probably noticed that I have spoken about this particular pattern before, with many UI related items, and that it is usually due to storing user controls in session or cache, and/or using static controls that where you set up event handlers in the page class to handle events for these static controls.

Basically, anytime you store a control or a UI item in session scope, cache or static variables, you will also hold a reference to the page that it was created on, as well as any controls it has and any data that might be data bound to any of the controls on the pages. In other words, until the control goes out of scope, i.e. is removed from cache or session state, these objects will not be available for garbage collection.

Here are a few posts that I have written on the topic:

Next debugging actions

When you see a lot of UI objects on the heap like this, the next step is to figure out why they are sticking around. One of the things I will always do first in these cases is to look at aspx and ascx pages on the heap and !gcroot them to see where they are rooted, i.e. what is keeping them in memory.

0:028> !dumpheap -type *aspx
Using our cache to search the heap.
   Address         MT     Size  Gen
...
6d43ede4 10303dd4      408    2 ASP.default_aspx
6d59eb50 10303dd4      408    2 ASP.default_aspx
6d67cb28 10303dd4      408    2 ASP.default_aspx
6d97e558 10303dd4      408    2 ASP.default_aspx
6df13390 10303dd4      408    0 ASP.default_aspx
6e6b53f0 10303dd4      408    0 ASP.default_aspx
Statistics:
      MT    Count    TotalSize Class Name
...
10303dd4      436      177,888 ASP.default_aspx
Total 1,230 objects, Total size: 573,544

In this case there were 1230 aspx pages on the heap, in reality there should be approximately one per currently executing request, so this tells us that they are definitely staying longer than they should since I couldn’t find a single thread executing a request when I printed out all the call stacks with ~* e !clrstack.

I gc-rooted one of the pages to see why it is sticking around and found that it was stored in a static HybridDictionary…

0:028> !gcroot 6d67cb28
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 11 OSTHread 5d8
Scan Thread 15 OSTHread 19c
Scan Thread 16 OSTHread e8c
Scan Thread 17 OSTHread c0c
Scan Thread 9 OSTHread 12f0
Scan Thread 18 OSTHread 15f0
Scan Thread 4 OSTHread 1138
Scan Thread 5 OSTHread 120c
Scan Thread 3 OSTHread 1664
Scan Thread 22 OSTHread 12e8
Scan Thread 25 OSTHread 1098
DOMAIN(000FA020):HANDLE(Pinned):22211f0:Root:  0a629ac8(System.Object[])->
  02777208(System.Collections.Specialized.HybridDictionary)->
  06946578(System.Collections.Hashtable)->
  3eac4e0c(System.Collections.Hashtable+bucket[])->
  6d67cb28(ASP.default_aspx)

And if I run !objsize on this Hybrid dictionary I find that it holds on to about 941 MB of data so this is certainly very interesting…

0:028> !objsize 02777208
sizeof(02777208) =  941,224,148 (  0x3819f0d4) bytes (System.Collections.Specialized.HybridDictionary)

I have to add a small caveat here, if you !objsize something that contains an aspx page, your objsize will include the size of the cache since the page has an indirect reference to the cache.

0:028> !dumpheap -type System.Web.Caching.Cache
Using our cache to search the heap.
   Address         MT     Size  Gen
0662fe54 6639d878       12    2 System.Web.Caching.Cache
Statistics:
      MT    Count    TotalSize Class Name
6639d878        1           12 System.Web.Caching.Cache
Total 1 objects, Total size: 12
0:028> !objsize 0662fe54
sizeof(0662fe54) =   10,770,512 (    0xa45850) bytes (System.Web.Caching.Cache)

In this case though the cache is very small so most of the memory held up by this HybridDictionary is the pages itself.

Ok, so now we know that our issue is due to the fact that we have a lot of pages in memory, and they are sticking around in a static HybridDictionary. The next step is to find out what this hybrid dictionary is and who is populating it with pages, and why…

To do this I dump out the HashTable+bucket[] that contains the page and search for the page address (6d67cb28) and the result was the entry displayed below

0:028> !da -details 3eac4e0c
...

[132] 3eac5444
    Name: System.Collections.Hashtable+bucket
    MethodTable 791021d8
    EEClass: 79102154
    Size: 20(0x14) bytes
     (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
    Fields:
          MT    Field   Offset                 Type VT     Attr    Value Name
    790fd0f0  4000937        0        System.Object  0 instance 6d67cb28 key
    790fd0f0  4000938        4        System.Object  0 instance 6d67d388 val
    79102290  4000939        8         System.Int32  1 instance 27189591 hash_coll
...

The page seems to be stored as the Key of this entry, so someone is populating a HybridDictionary with a key/value pair, where key=<the page>.

The value in this case is a ListDictionary

0:028> !do 6d67d388
Name: System.Collections.Specialized.ListDictionary
MethodTable: 7a75820c
EEClass: 7a75819c
Size: 28(0x1c) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
7a7582d8  4001157        4 ...ry+DictionaryNode  0 instance 6d67d3a4 head
79102290  4001158       10         System.Int32  1 instance        4 version
79102290  4001159       14         System.Int32  1 instance        4 count
79115ea8  400115a        8 ...ections.IComparer  0 instance 00000000 comparer
790fd0f0  400115b        c        System.Object  0 instance 00000000 _syncRoot

and if we print out the first entry of the list dictionary (the head node) we find that the value is Ajax.NET.Prototype…

0:028> !do 6d67d3a4
Name: System.Collections.Specialized.ListDictionary+DictionaryNode
MethodTable: 7a7582d8
EEClass: 7a7c4220
Size: 20(0x14) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fd0f0  4001167        4        System.Object  0 instance 027773d8 key
790fd0f0  4001168        8        System.Object  0 instance 6d67d2e8 value
7a7582d8  4001169        c ...ry+DictionaryNode  0 instance 6d67d44c next
0:028> !do 027773d8
Name: System.String
MethodTable: 790fd8c4
EEClass: 790fd824
Size: 54(0x36) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: Ajax.NET.prototype
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
79102290  4000096        4         System.Int32  1 instance       19 m_arrayLength
79102290  4000097        8         System.Int32  1 instance       18 m_stringLength
790ff328  4000098        c          System.Char  1 instance       41 m_firstChar
790fd8c4  4000099       10        System.String  0   shared   static Empty
    >> Domain:Value  000d5d68:790d884c 000fa020:790d884c <<
7912dd40  400009a       14        System.Char[]  0   shared   static WhitespaceChars
    >> Domain:Value  000d5d68:026203f4 000fa020:0262429c <<

To figure out where this comes from, I saved out all the modules to disc using !for_each_module

!savemodule ${@#Base} f:\blog\modules\${@#ModuleName}.dll

as described in this post

I loaded them all up in reflector, and did a string search for Ajax.NET.Prototype, which returned a method containing this string(AjaxPro.Utility.RegisterCommonAjax, in AjaxPro.dll)

ajaxpro

Armed with this, I searched the internet for “AjaxPro memory leak HybridDictionary” and found http://www.ajaxpro.info/changes.txt which shows a list of changes in the AjaxPro library. Of particular interest is a fix made in version 6.4.27.1

Version 6.4.27.1 (beta) - Fixed null values to DBNull.Value for System.Data.DataTable. - Fixed memory leak with HybridDictionary for JavaScript include rendering.

And the solution here was to upgrade to the latest version of AjaxPro.

About AjaxPro

Normally I don’t write about 3rd party products, especially issues with 3rd party products, but I know that AjaxPro is a nice AJAX library that is used by quite a few of our customers so I hope this post is of general interest, as this issue is resolved in a later version of AjaxPro. I also hope that the post can be of use for finding other similar issues outside of AjaxPro.

I should also mention that I am posting this with the permission of Michael Schwartz who developed AjaxPro, and I must say that I was extremely impressed with the openness in his response when I asked if it was ok to post about this. His comment was, “feel free to write about it, I love to hear critics if there are any, to improve development and/or fix bugs”

AjaxPro is now open source and can be downloaded from codeplex here http://www.codeplex.com/AjaxPro/. If you are interested in this you should also visit Michaels blog at http://weblogs.asp.net/mschwarz/

Laters, Tess