Jekyll2023-11-08T14:48:34+00:00https://www.tessferrandez.com/feed.xmlIf broken it is, fix it you shouldUsing the powers of the debugger to solve the problems of the world -- and a bag of chipsTess FerrandezNaming tests2023-11-08T08:31:14+00:002023-11-08T08:31:14+00:00https://www.tessferrandez.com/blog/2023/11/08/naming-tests<p>I’m a sucker for clean, minimalistic code, that is easy to read, understand and maintain.</p>
<p>When it comes to “regular” code, most developers are fairly good about this, and we have a lot of opinionated content and examples, of code smells and patterns, but I find that when it comes to tests we tend to neglect this. Tests are often a hot mess with a plethora of problems, problems that discourage people to write tests because they are simply more of a headache than what they are worth.</p>
<ul>
<li>Badly named or badly organized tests</li>
<li>Tests that don’t serve a purpose, or test the wrong thing</li>
<li>Tests that are hard to understand</li>
<li>Tests that are hard to maintain, and act as a barrier to refactoring instead of enabling it</li>
</ul>
<p>At work, I have started a bi-weekly “gathering” where me and some like minded people get together and discuss interesting technical things. Lately, one of the topics we have discussed is writing good tests, and I thought I would share some of the things we have discussed, starting with some thoughts on naming.</p>
<h2 id="naming-tests">Naming tests</h2>
<p>For the purposes of this, I’ll use the example of the <a href="https://kata-log.rocks/banking-kata">Banking kata</a>, as it is very simple to understand.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">BankAccount</span>
<span class="p">{</span>
<span class="k">public</span> <span class="kt">int</span> <span class="n">Balance</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">private</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="nf">BankAccount</span><span class="p">()</span> <span class="p">{</span> <span class="p">}</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Deposit</span><span class="p">(</span><span class="kt">int</span> <span class="n">amount</span><span class="p">)</span> <span class="p">{</span> <span class="p">}</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Withdraw</span><span class="p">(</span><span class="kt">int</span> <span class="n">amount</span><span class="p">)</span> <span class="p">{</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>A not uncommon way to approach writing unit tests for this class would be to write a test for each method, and then a test for each edge case. This would result in a test class that looks something like this:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">BankAccountTests</span> <span class="p">{</span>
<span class="p">[</span><span class="n">Fact</span><span class="p">]</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">TestConstructor</span><span class="p">()</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">sut</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">BankAccount</span><span class="p">();</span>
<span class="n">sut</span><span class="p">.</span><span class="n">Balance</span><span class="p">.</span><span class="nf">Should</span><span class="p">().</span><span class="nf">Be</span><span class="p">(</span><span class="m">0</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">[</span><span class="n">Fact</span><span class="p">]</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">TestDeposit</span><span class="p">()</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">sut</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">BankAccount</span><span class="p">();</span>
<span class="n">sut</span><span class="p">.</span><span class="nf">Deposit</span><span class="p">(</span><span class="m">100</span><span class="p">);</span>
<span class="n">sut</span><span class="p">.</span><span class="n">Balance</span><span class="p">.</span><span class="nf">Should</span><span class="p">().</span><span class="nf">Be</span><span class="p">(</span><span class="m">100</span><span class="p">);</span>
<span class="n">sut</span><span class="p">.</span><span class="nf">Deposit</span><span class="p">(</span><span class="m">200</span><span class="p">);</span>
<span class="n">sut</span><span class="p">.</span><span class="n">Balance</span><span class="p">.</span><span class="nf">Should</span><span class="p">().</span><span class="nf">Be</span><span class="p">(</span><span class="m">300</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">[</span><span class="n">Fact</span><span class="p">]</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">TestWithdraw</span><span class="p">()</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">sut</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">BankAccount</span><span class="p">();</span>
<span class="n">sut</span><span class="p">.</span><span class="nf">Deposit</span><span class="p">(</span><span class="m">100</span><span class="p">);</span>
<span class="n">sut</span><span class="p">.</span><span class="nf">Withdraw</span><span class="p">(</span><span class="m">20</span><span class="p">);</span>
<span class="n">sut</span><span class="p">.</span><span class="n">Balance</span><span class="p">.</span><span class="nf">Should</span><span class="p">().</span><span class="nf">Be</span><span class="p">(</span><span class="m">80</span><span class="p">);</span>
<span class="n">Action</span> <span class="n">action1</span> <span class="p">=</span> <span class="p">()</span> <span class="p">=></span> <span class="n">sut</span><span class="p">.</span><span class="nf">Withdraw</span><span class="p">(</span><span class="m">90</span><span class="p">);</span>
<span class="n">action1</span><span class="p">.</span><span class="nf">Should</span><span class="p">().</span><span class="n">Throw</span><span class="p"><</span><span class="n">InvalidOperationException</span><span class="p">>();</span>
<span class="n">Action</span> <span class="n">action2</span> <span class="p">=</span> <span class="p">()</span> <span class="p">=></span> <span class="n">sut</span><span class="p">.</span><span class="nf">Withdraw</span><span class="p">(-</span><span class="m">10</span><span class="p">);</span>
<span class="n">action2</span><span class="p">.</span><span class="nf">Should</span><span class="p">().</span><span class="n">Throw</span><span class="p"><</span><span class="n">InvalidOperationException</span><span class="p">>();</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This strategy has a number of issues.</p>
<ol>
<li>The name of the tests says very little about what it is we are testing, and what the expected result is. If we see this test fail, we have no idea what the problem is, and as someone reviewing this code, we would have to go into the test to see what it is testing.</li>
<li>We are testing multiple scenarios in each test, which makes it hard to understand both what is being tested, and which scenario is failing if the test fails.</li>
</ol>
<p>From a readability perspective, I am also allergic to the use of <strong>sut</strong> (Subject Under Test), it makes it a lot harder to read the code than if we would have used a more descriptive name.</p>
<p>So, ideally, we want to…</p>
<ul>
<li>split the tests such that each test tests its own scenario</li>
<li>have test names that reflect what we are testing and what the expected result is</li>
<li>avoid testing things that we don’t need to test, such as .net internals, or even implementation details internal to the method TBH.</li>
</ul>
<h2 id="the-microsoft-way-and-gherkin">The Microsoft way and Gherkin</h2>
<p><a href="https://docs.microsoft.com/en-us/dotnet/core/testing/unit-testing-best-practices">Microsoft’s unit testing best practices</a> suggest that you should name your tests in the format <code class="language-plaintext highlighter-rouge">UnitOfWork_StateUnderTest_ExpectedBehavior</code>. An example of a test for the <code class="language-plaintext highlighter-rouge">Withdraw</code> method would then be</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="n">Fact</span><span class="p">]</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Withdraw_WhenAmountIsLessThanBalance_ShouldDecreaseBalanceByAmount</span><span class="p">()</span>
<span class="p">{</span>
<span class="c1">// arrange</span>
<span class="kt">var</span> <span class="n">account</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">BankAccount</span><span class="p">();</span>
<span class="n">account</span><span class="p">.</span><span class="nf">Deposit</span><span class="p">(</span><span class="m">100</span><span class="p">);</span>
<span class="c1">// act</span>
<span class="n">account</span><span class="p">.</span><span class="nf">Withdraw</span><span class="p">(</span><span class="m">20</span><span class="p">);</span>
<span class="c1">// assert</span>
<span class="n">account</span><span class="p">.</span><span class="n">Balance</span><span class="p">.</span><span class="nf">Should</span><span class="p">().</span><span class="nf">Be</span><span class="p">(</span><span class="m">80</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>A variation on this theme, is using Gherkin naming</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="n">Fact</span><span class="p">]</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">GivenAnAccountWithABalance_WhenCallingWithdrawWithLessThanTheBalance_TheBalanceShouldBeDecreasedByTheAmount</span><span class="p">()</span>
<span class="p">{</span>
<span class="c1">// arrange</span>
<span class="kt">var</span> <span class="n">account</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">BankAccount</span><span class="p">();</span>
<span class="n">account</span><span class="p">.</span><span class="nf">Deposit</span><span class="p">(</span><span class="m">100</span><span class="p">);</span>
<span class="c1">// act</span>
<span class="n">account</span><span class="p">.</span><span class="nf">Withdraw</span><span class="p">(</span><span class="m">20</span><span class="p">);</span>
<span class="c1">// assert</span>
<span class="n">account</span><span class="p">.</span><span class="n">Balance</span><span class="p">.</span><span class="nf">Should</span><span class="p">().</span><span class="nf">Be</span><span class="p">(</span><span class="m">80</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>On the positive side:</p>
<ul>
<li>we are now testing one scenario per test</li>
<li>it is pretty clear what we are testing, and what the expected result is</li>
</ul>
<p>I have also sneaked in the arrange/act/assert pattern to make it even clearer what we are testing. I find that this pattern is helpful both when reading tests but also to help guide when writing tests to make sure that we are only testing one concept per test.</p>
<p>On the negative side:</p>
<ul>
<li>OMG! The test names are so long that they are unreadable. I have to scroll horizontally to read the entire name. This is not a good thing.</li>
<li>The mix of using underscores and camel casing is very jarring, and makes it very hard to read.</li>
<li>Since we specifically name the function <code class="language-plaintext highlighter-rouge">Withdraw</code> in the test name, if we were to rename the Withdraw method, we might be left with code drift as these test names will not be updated automatically.</li>
<li>These schemes lock you in to a specific naming convention. When using these conventions I often find developers struggling with what to put where and where to place the underscores.</li>
<li>The given, when, then scheme also has a lot of redundancy that makes the names even longer.</li>
</ul>
<h2 id="a-small-detour-about-words-that-we-should-avoid">A small detour about words that we should avoid</h2>
<p>There are plenty of words that often make it into tests, that I think we should avoid. The use of the word should above made me think of this, but there are plenty of others.</p>
<ul>
<li><strong>Correct</strong> as in <code class="language-plaintext highlighter-rouge">ShouldCorrectlyCalculateBalance</code> or <code class="language-plaintext highlighter-rouge">ShouldCorrectlyHandleNegativeAmounts</code>. What is correct? What is the expected result? What is the scenario we are testing? The same obviously goes for <strong>Valid</strong>, <strong>Invalid</strong>, <strong>Good</strong>, <strong>Bad</strong>, etc. We need to be specific about what correct means in this context.</li>
<li><strong>Can</strong> as in <code class="language-plaintext highlighter-rouge">CanCalculateBalance</code> or <code class="language-plaintext highlighter-rouge">CanHandleNegativeAmounts</code>. This is another fluff word that only makes the test name longer and vaguer. Be direct and specific, <code class="language-plaintext highlighter-rouge">ThrowsAnArgumentExceptionIfANegativeAmountIsWithdrawn</code></li>
<li><strong>Should</strong> as in <code class="language-plaintext highlighter-rouge">ShouldThrowAnArgumentExceptionIfANegativeAmountIsWithdrawn</code>. We are not testing wishes or desires, we are testing absolutes, the test either passes or fails. Skip the should, and just say <code class="language-plaintext highlighter-rouge">ThrowsAnArgumentExceptionIfANegativeAmountIsWithdrawn</code></li>
<li><strong>Test</strong> as in <code class="language-plaintext highlighter-rouge">ThrowsAnArgumentExceptionIfANegativeAmountIsWithdrawnTest</code>. Unless this is required by the test framework, there is no need to add the word Test to our tests. I sometimes add the word Tests to the test class, ex. <code class="language-plaintext highlighter-rouge">BankAccountTests</code> but this is mainly to avoid naming conflicts with the class under test, and to make it clear that this is a test class.</li>
</ul>
<h2 id="a-better-way">A better way</h2>
<p>I vividly recall listening to a presentation by <a href="https://kevlinhenney.medium.com/">Kevlin Henney</a> on naming, that completely made me change my thinking about test names. A lot of my ramblings so far are honestly a respectful rip off of his ideas from the excellent talk <a href="https://www.youtube.com/watch?v=wCx_6kOo99M">Test smells and fragrances</a> along with some lessons I learned myself, the hard way. Everything he says about naming in general resonates immensely with me.</p>
<p>He suggested that we are allowed to use snake_casing in our test names, even in C# ?!?!?!</p>
<p>Blasphemy, are we really allowed, what is this sorcery you speak of.</p>
<p>He said something akin to “if you are writing legal contracts or text messages you may be writing them both in English, but you would use completely different forms, to better suit your needs”. The same goes for code, we should use the tools at our disposal to make our code as readable as possible.</p>
<p>And… if we are to be honest, <code class="language-plaintext highlighter-rouge">snake_casing_is_a_lot_more_readable</code> <code class="language-plaintext highlighter-rouge">ThanCamelCasingOrPascalCasingWhenTheSentencesGetLonger</code>.</p>
<p>We can also group our tests (into nested classes if need be, nested classes in C#, really?!?!). Grouping our tests not only allows us to shorten our test names. As an added bonus, it also allows us to group functionality which can be helpful as we can use the same setup for a group of tests.</p>
<p>Applying this to our example, we could end up with something like this:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">BankAccountTests</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">A_new_account</span>
<span class="p">{</span>
<span class="p">[</span><span class="n">Fact</span><span class="p">]</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">has_a_balance_of_zero</span><span class="p">()</span> <span class="p">{}</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">Depositing_to_an_account</span>
<span class="p">{</span>
<span class="p">[</span><span class="n">Fact</span><span class="p">]</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">adds_the_deposit_to_the_account_balance</span><span class="p">()</span> <span class="p">{}</span>
<span class="p">[</span><span class="n">Fact</span><span class="p">]</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">throws_an_exception_if_the_amount_is_negative</span><span class="p">()</span> <span class="p">{}</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">Withdrawing_from_an_account</span>
<span class="p">{</span>
<span class="p">[</span><span class="n">Fact</span><span class="p">]</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">subtracts_the_withdrawal_from_the_account_balance</span><span class="p">()</span> <span class="p">{}</span>
<span class="p">[</span><span class="n">Fact</span><span class="p">]</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">throws_an_exception_if_the_amount_is_negative</span><span class="p">()</span> <span class="p">{}</span>
<span class="p">[</span><span class="n">Fact</span><span class="p">]</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">throws_an_exception_if_the_amount_is_greater_than_the_balance</span><span class="p">()</span> <span class="p">{}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>When we collapse the functions, both in the code and also in the test explorer, we end up with a spec of the functionality, and the intentions behind the implementation.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I know, as with many testing examples, this was a very simple example, but I have found that it extrapolates well to more complex scenarios.</p>
<p>When I write code, usually, the first thing I do, is pair up with a colleague, look through the story, and write these test skeletons of the functionality we are about to implement. That way we can iron out our understanding of the functionality before we even write one line of code. And as a bonus, when someone comes to review the story, they can just browse the tests first to ensure that we have covered all the scenarios.</p>
<p>I have a lot of ideas around readability and maintainability of the actual test code as well, but I’ll save that for another post.
I would be very interested in your comments though, so whether you agree or disagree or have other tips or experiences, I would love to hear from you on twitter (X) <a href="https://twitter.com/tessferrandez">@tessferrandez</a>.</p>Tess FerrandezI’m a sucker for clean, minimalistic code, that is easy to read, understand and maintain.Organizing ASP.NET Core Minimal APIs2023-10-31T08:31:14+00:002023-10-31T08:31:14+00:00https://www.tessferrandez.com/blog/2023/10/31/organizing-minimal-apis<p>I often hear (on social media or direct comments) people saying you can’t use minimal APIs for “real production” apps.</p>
<p>I think the reason for this, is that most samples that we see are very simple, and don’t show how to organize the code in a way that makes sense for a larger application.</p>
<p>Although truth be told, many “real production” apps are not that complicated either, but, that’s another story.</p>
<p>I work with a lot of large organizations, helping them move their workloads to Azure, and over the past two years or so, since minimal APIs were introduced in ASP.NET Core, I’ve been using them for all our projects, and I’ve been very happy with them. Before then, I have also used similar minimal apis in python (FastAPI and Flask).</p>
<p>The issue though is that in ASP.NET, the samples have all ended with some endpoints in program.cs, with the code inline (like below), and that’s not really how you want to organize your code in a larger application, so I wanted to do a quick writeup of a few easy steps to make the code more maintainable.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">var</span> <span class="n">builder</span> <span class="p">=</span> <span class="n">WebApplication</span><span class="p">.</span><span class="nf">CreateBuilder</span><span class="p">(</span><span class="n">args</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">app</span> <span class="p">=</span> <span class="n">builder</span><span class="p">.</span><span class="nf">Build</span><span class="p">();</span>
<span class="p">...</span>
<span class="n">app</span><span class="p">.</span><span class="nf">MapGet</span><span class="p">(</span><span class="s">"/todoitems"</span><span class="p">,</span> <span class="k">async</span> <span class="p">(</span><span class="n">TodoDb</span> <span class="n">db</span><span class="p">)</span> <span class="p">=></span>
<span class="k">await</span> <span class="n">db</span><span class="p">.</span><span class="n">Todos</span><span class="p">.</span><span class="nf">ToListAsync</span><span class="p">());</span>
<span class="n">app</span><span class="p">.</span><span class="nf">MapGet</span><span class="p">(</span><span class="s">"/todoitems/complete"</span><span class="p">,</span> <span class="k">async</span> <span class="p">(</span><span class="n">TodoDb</span> <span class="n">db</span><span class="p">)</span> <span class="p">=></span>
<span class="k">await</span> <span class="n">db</span><span class="p">.</span><span class="n">Todos</span><span class="p">.</span><span class="nf">Where</span><span class="p">(</span><span class="n">t</span> <span class="p">=></span> <span class="n">t</span><span class="p">.</span><span class="n">IsComplete</span><span class="p">).</span><span class="nf">ToListAsync</span><span class="p">());</span>
<span class="n">app</span><span class="p">.</span><span class="nf">MapGet</span><span class="p">(</span><span class="s">"/todoitems/{id}"</span><span class="p">,</span> <span class="k">async</span> <span class="p">(</span><span class="kt">int</span> <span class="n">id</span><span class="p">,</span> <span class="n">TodoDb</span> <span class="n">db</span><span class="p">)</span> <span class="p">=></span>
<span class="k">await</span> <span class="n">db</span><span class="p">.</span><span class="n">Todos</span><span class="p">.</span><span class="nf">FindAsync</span><span class="p">(</span><span class="n">id</span><span class="p">)</span>
<span class="k">is</span> <span class="n">Todo</span> <span class="n">todo</span>
<span class="p">?</span> <span class="n">Results</span><span class="p">.</span><span class="nf">Ok</span><span class="p">(</span><span class="n">todo</span><span class="p">)</span>
<span class="p">:</span> <span class="n">Results</span><span class="p">.</span><span class="nf">NotFound</span><span class="p">());</span>
<span class="n">app</span><span class="p">.</span><span class="nf">Run</span><span class="p">();</span>
</code></pre></div></div>
<blockquote>
<p>I’m using an example from Microsoft Learn <a href="https://learn.microsoft.com/en-us/aspnet/core/tutorials/min-web-api?view=aspnetcore-7.0&tabs=visual-studio">Tutorial: Create a minimal API with ASP.NET Core</a>, and a few of the tips below come from that post, but I have added some variations that I have found useful in the projects I have been working on.</p>
</blockquote>
<h2 id="use-extension-methods-to-organize-the-endpoints">Use extension methods to organize the endpoints</h2>
<p>A simple solution to organize the endpoints is through using extension methods.</p>
<p>In a folder called <code class="language-plaintext highlighter-rouge">Endpoints</code> or in a <code class="language-plaintext highlighter-rouge">TodoItems</code> folder if you do clean architecture, we can create a file called <code class="language-plaintext highlighter-rouge">TodoItemsEndpoints.cs</code> and add the following code:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">static</span> <span class="k">class</span> <span class="nc">TodoItemsEndpoints</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">static</span> <span class="k">void</span> <span class="nf">RegisterTodoItemsEndpoints</span><span class="p">(</span><span class="k">this</span> <span class="n">WebApplication</span> <span class="n">app</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">app</span><span class="p">.</span><span class="nf">MapGet</span><span class="p">(</span><span class="s">"/todoitems"</span><span class="p">,</span> <span class="k">async</span> <span class="p">(</span><span class="n">TodoDb</span> <span class="n">db</span><span class="p">)</span> <span class="p">=></span>
<span class="k">await</span> <span class="n">db</span><span class="p">.</span><span class="n">Todos</span><span class="p">.</span><span class="nf">ToListAsync</span><span class="p">());</span>
<span class="n">app</span><span class="p">.</span><span class="nf">MapGet</span><span class="p">(</span><span class="s">"/todoitems/complete"</span><span class="p">,</span> <span class="k">async</span> <span class="p">(</span><span class="n">TodoDb</span> <span class="n">db</span><span class="p">)</span> <span class="p">=></span>
<span class="k">await</span> <span class="n">db</span><span class="p">.</span><span class="n">Todos</span><span class="p">.</span><span class="nf">Where</span><span class="p">(</span><span class="n">t</span> <span class="p">=></span> <span class="n">t</span><span class="p">.</span><span class="n">IsComplete</span><span class="p">).</span><span class="nf">ToListAsync</span><span class="p">());</span>
<span class="n">app</span><span class="p">.</span><span class="nf">MapGet</span><span class="p">(</span><span class="s">"/todoitems/{id}"</span><span class="p">,</span> <span class="k">async</span> <span class="p">(</span><span class="kt">int</span> <span class="n">id</span><span class="p">,</span> <span class="n">TodoDb</span> <span class="n">db</span><span class="p">)</span> <span class="p">=></span>
<span class="k">await</span> <span class="n">db</span><span class="p">.</span><span class="n">Todos</span><span class="p">.</span><span class="nf">FindAsync</span><span class="p">(</span><span class="n">id</span><span class="p">)</span>
<span class="k">is</span> <span class="n">Todo</span> <span class="n">todo</span>
<span class="p">?</span> <span class="n">Results</span><span class="p">.</span><span class="nf">Ok</span><span class="p">(</span><span class="n">todo</span><span class="p">)</span>
<span class="p">:</span> <span class="n">Results</span><span class="p">.</span><span class="nf">NotFound</span><span class="p">());</span>
<span class="p">...</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This instantly makes our <code class="language-plaintext highlighter-rouge">Program.cs</code> file much cleaner, and we can separate out all our endpoints into separate files, and organize them in a way that makes sense for our application.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">var</span> <span class="n">builder</span> <span class="p">=</span> <span class="n">WebApplication</span><span class="p">.</span><span class="nf">CreateBuilder</span><span class="p">(</span><span class="n">args</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">app</span> <span class="p">=</span> <span class="n">builder</span><span class="p">.</span><span class="nf">Build</span><span class="p">();</span>
<span class="p">...</span>
<span class="n">app</span><span class="p">.</span><span class="nf">RegisterTodoItemsEndpoints</span><span class="p">();</span>
<span class="n">app</span><span class="p">.</span><span class="nf">Run</span><span class="p">();</span>
</code></pre></div></div>
<blockquote>
<p>I should mention, there are nuget packages (like <a href="https://github.com/CarterCommunity/Carter">Carter</a>) that will allow you to create endpoint registrations without calling them from <code class="language-plaintext highlighter-rouge">program.cs</code> but I have personally never found a need for this.</p>
</blockquote>
<h2 id="use-typedresults-instead-of-results">Use TypedResults instead of Results</h2>
<p>Using <strong>TypedResults</strong> instead of <strong>Results</strong>, we can make our code even cleaner.</p>
<p>If you use Results, you would typically want to add a Produces attribute, to specify for swagger etc. what the valid response types are.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">app</span><span class="p">.</span><span class="nf">MapGet</span><span class="p">(</span><span class="s">"/todoitems/{id}"</span><span class="p">,</span> <span class="k">async</span> <span class="p">(</span><span class="kt">int</span> <span class="n">id</span><span class="p">,</span> <span class="n">TodoDb</span> <span class="n">db</span><span class="p">)</span> <span class="p">=></span>
<span class="k">await</span> <span class="n">db</span><span class="p">.</span><span class="n">Todos</span><span class="p">.</span><span class="nf">FindAsync</span><span class="p">(</span><span class="n">id</span><span class="p">)</span>
<span class="k">is</span> <span class="n">Todo</span> <span class="n">todo</span>
<span class="p">?</span> <span class="n">Results</span><span class="p">.</span><span class="nf">Ok</span><span class="p">(</span><span class="n">todo</span><span class="p">)</span>
<span class="p">:</span> <span class="n">Results</span><span class="p">.</span><span class="nf">NotFound</span><span class="p">())</span>
<span class="p">.</span><span class="n">Produces</span><span class="p"><</span><span class="n">Todo</span><span class="p">>(</span><span class="n">StatusCodes</span><span class="p">.</span><span class="n">Status200OK</span><span class="p">)</span>
<span class="p">.</span><span class="nf">Produces</span><span class="p">(</span><span class="n">StatusCodes</span><span class="p">.</span><span class="n">Status404NotFound</span><span class="p">);</span>
</code></pre></div></div>
<p>with <strong>TypedResults</strong>, we can skip the <code class="language-plaintext highlighter-rouge">.Produces()</code> attribute, as it will be inferred from the return type.
This means that we will have less clutter, we can check at compile time if we have made mistakes, and our unit tests will be simpler as well.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">app</span><span class="p">.</span><span class="nf">MapGet</span><span class="p">(</span><span class="s">"/todoitems/{id}"</span><span class="p">,</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">Results</span><span class="p"><</span><span class="n">Ok</span><span class="p"><</span><span class="n">Todo</span><span class="p">>,</span> <span class="n">NotFound</span><span class="p">>></span> <span class="p">(</span><span class="kt">int</span> <span class="n">id</span><span class="p">,</span> <span class="n">TodoDb</span> <span class="n">db</span><span class="p">)</span> <span class="p">=></span>
<span class="k">await</span> <span class="n">db</span><span class="p">.</span><span class="n">Todos</span><span class="p">.</span><span class="nf">FindAsync</span><span class="p">(</span><span class="n">id</span><span class="p">)</span>
<span class="k">is</span> <span class="n">Todo</span> <span class="n">todo</span>
<span class="p">?</span> <span class="n">TypedResults</span><span class="p">.</span><span class="nf">Ok</span><span class="p">(</span><span class="n">todo</span><span class="p">)</span>
<span class="p">:</span> <span class="n">TypedResults</span><span class="p">.</span><span class="nf">NotFound</span><span class="p">());</span>
</code></pre></div></div>
<h2 id="separate-out-the-functionality-from-the-endpoint-registrations">Separate out the functionality from the endpoint registrations</h2>
<p>Even though we have separated this out into a separate file, it still looks very messy, and it’s not very testable.</p>
<p>We can fix both of these issues by moving our lambdas to separate methods.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">app</span><span class="p">.</span><span class="nf">MapGet</span><span class="p">(</span><span class="s">"/todoitems/{id}"</span><span class="p">,</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">Results</span><span class="p"><</span><span class="n">Ok</span><span class="p"><</span><span class="n">Todo</span><span class="p">>,</span> <span class="n">NotFound</span><span class="p">>></span> <span class="p">(</span><span class="kt">int</span> <span class="n">id</span><span class="p">,</span> <span class="n">TodoDb</span> <span class="n">db</span><span class="p">)</span> <span class="p">=></span>
<span class="k">await</span> <span class="n">db</span><span class="p">.</span><span class="n">Todos</span><span class="p">.</span><span class="nf">FindAsync</span><span class="p">(</span><span class="n">id</span><span class="p">)</span>
<span class="k">is</span> <span class="n">Todo</span> <span class="n">todo</span>
<span class="p">?</span> <span class="n">TypedResults</span><span class="p">.</span><span class="nf">Ok</span><span class="p">(</span><span class="n">todo</span><span class="p">)</span>
<span class="p">:</span> <span class="n">TypedResults</span><span class="p">.</span><span class="nf">NotFound</span><span class="p">());</span>
</code></pre></div></div>
<p>becomes</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">app</span><span class="p">.</span><span class="nf">MapGet</span><span class="p">(</span><span class="s">"/todoitems/{id}"</span><span class="p">,</span> <span class="n">GetTodoById</span><span class="p">);</span>
<span class="k">static</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">Results</span><span class="p"><</span><span class="n">Ok</span><span class="p"><</span><span class="n">Todo</span><span class="p">>,</span> <span class="n">NotFound</span><span class="p">>></span> <span class="nf">GetTodoById</span><span class="p">(</span><span class="kt">int</span> <span class="n">id</span><span class="p">,</span> <span class="n">TodoDb</span> <span class="n">db</span><span class="p">)</span> <span class="p">=></span>
<span class="k">await</span> <span class="n">db</span><span class="p">.</span><span class="n">Todos</span><span class="p">.</span><span class="nf">FindAsync</span><span class="p">(</span><span class="n">id</span><span class="p">)</span>
<span class="k">is</span> <span class="n">Todo</span> <span class="n">todo</span>
<span class="p">?</span> <span class="n">TypedResults</span><span class="p">.</span><span class="nf">Ok</span><span class="p">(</span><span class="n">todo</span><span class="p">)</span>
<span class="p">:</span> <span class="n">TypedResults</span><span class="p">.</span><span class="nf">NotFound</span><span class="p">();</span>
</code></pre></div></div>
<p>So now the registration part of TodoItemsEndpoints looks like this:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">app</span><span class="p">.</span><span class="nf">MapGet</span><span class="p">(</span><span class="s">"/todoitems"</span><span class="p">,</span> <span class="n">GetAllTodos</span><span class="p">);</span>
<span class="n">app</span><span class="p">.</span><span class="nf">MapGet</span><span class="p">(</span><span class="s">"/todoitems/complete"</span><span class="p">,</span> <span class="n">GetCompleteTodos</span><span class="p">);</span>
<span class="n">app</span><span class="p">.</span><span class="nf">MapGet</span><span class="p">(</span><span class="s">"/todoitems/{id}"</span><span class="p">,</span> <span class="n">GetTodoById</span><span class="p">);</span>
<span class="n">app</span><span class="p">.</span><span class="nf">MapPost</span><span class="p">(</span><span class="s">"/todoitems/"</span><span class="p">,</span> <span class="n">CreateTodo</span><span class="p">);</span>
<span class="n">app</span><span class="p">.</span><span class="nf">MapPut</span><span class="p">(</span><span class="s">"/todoitems/{id}"</span><span class="p">,</span> <span class="n">UpdateTodoById</span><span class="p">);</span>
<span class="n">app</span><span class="p">.</span><span class="nf">MapDelete</span><span class="p">(</span><span class="s">"/todoitems/{id}"</span><span class="p">,</span> <span class="n">DeleteTodo</span><span class="p">);</span>
</code></pre></div></div>
<p>followed by the methods that we just moved out of the registration</p>
<p>And now if we want to test any of our methods, we can just call them directly, without having to go through the API.</p>
<h2 id="grouping-endpoints">Grouping endpoints</h2>
<p>We can further clean this up, by grouping the endpoints so that we can avoid repetition.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">var</span> <span class="n">todoItems</span> <span class="p">=</span> <span class="n">app</span><span class="p">.</span><span class="nf">MapGroup</span><span class="p">(</span><span class="s">"/todoitems"</span><span class="p">);</span>
<span class="n">todoItems</span><span class="p">.</span><span class="nf">MapGet</span><span class="p">(</span><span class="s">"/"</span><span class="p">,</span> <span class="n">GetAllTodos</span><span class="p">);</span>
<span class="n">todoItems</span><span class="p">.</span><span class="nf">MapGet</span><span class="p">(</span><span class="s">"/complete"</span><span class="p">,</span> <span class="n">GetCompleteTodos</span><span class="p">);</span>
<span class="n">todoItems</span><span class="p">.</span><span class="nf">MapGet</span><span class="p">(</span><span class="s">"/{id}"</span><span class="p">,</span> <span class="n">GetTodoById</span><span class="p">);</span>
<span class="n">todoItems</span><span class="p">.</span><span class="nf">MapPost</span><span class="p">(</span><span class="s">"/"</span><span class="p">,</span> <span class="n">CreateTodo</span><span class="p">);</span>
<span class="n">todoItems</span><span class="p">.</span><span class="nf">MapPut</span><span class="p">(</span><span class="s">"/{id}"</span><span class="p">,</span> <span class="n">UpdateTodoById</span><span class="p">);</span>
<span class="n">todoItems</span><span class="p">.</span><span class="nf">MapDelete</span><span class="p">(</span><span class="s">"/{id}"</span><span class="p">,</span> <span class="n">DeleteTodo</span><span class="p">);</span>
</code></pre></div></div>
<p>This saves us from writing todoitems 6 times, but it also allows us to apply other attributes to the whole group if we want to.</p>
<p>We can require authorization, or add tags or CORS or rate limiting to all the endpoints in the group in one go</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">var</span> <span class="n">todoItems</span> <span class="p">=</span> <span class="n">app</span><span class="p">.</span><span class="nf">MapGroup</span><span class="p">(</span><span class="s">"/todoitems"</span><span class="p">)</span>
<span class="p">.</span><span class="nf">RequireAuthorization</span><span class="p">()</span>
<span class="p">.</span><span class="nf">WithTags</span><span class="p">(</span><span class="s">"Todo Items"</span><span class="p">);</span>
</code></pre></div></div>
<p>You can of course also add the authorization and tags to the individual endpoints if you want to.</p>
<h2 id="summary">Summary</h2>
<p>Over the past number of years, when I have split my time between python and .NET, I have come to appreciate simplicity and minimalism more and more in my code. The less boiler plate and less clutter the better, so minimal APIs have been a really great fit for me, and so far, I have not found any limitations that have been a problem for me independent of project size. Whether you agree or disagree or have other tips or experiences, I would love to hear from you on twitter (X) <a href="https://twitter.com/tessferrandez">@tessferrandez</a>.</p>Tess FerrandezI often hear (on social media or direct comments) people saying you can’t use minimal APIs for “real production” apps.Debugging .NET Core memory issues (on Linux) with dotnet dump2021-03-18T08:31:14+00:002021-03-18T08:31:14+00:00https://www.tessferrandez.com/blog/2021/03/18/debugging-a-netcore-memory-issue-with-dotnet-dump<p>Hi all,</p>
<p>In my last few projects I’ve been spending a substantial portion of the time on Linux (more than I really care for TBH). This, together with a new found interest for debugging and .NET Core - after a long detour to Python land - led me to look into some of the new debugging tools for .NET Core.</p>
<p>I have a very simple console app that keeps accumulating memory in a static list, to create a high memory usage issue. I almost ventured to call it a leak, but technically it’s not a leak since we can still reclaim the memory by removing the Products from the list. Nevertheless, it does cause a high memory issue and could eventually lead to an OutOfMemoryException.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="nn">System</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">System.Collections.Generic</span><span class="p">;</span>
<span class="k">namespace</span> <span class="nn">simple_memory_leak</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">Product</span><span class="p">{</span>
<span class="kt">string</span> <span class="n">name</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">id</span><span class="p">;</span>
<span class="kt">char</span><span class="p">[]</span> <span class="n">details</span> <span class="p">=</span> <span class="k">new</span> <span class="kt">char</span><span class="p">[</span><span class="m">10000</span><span class="p">];</span>
<span class="k">public</span> <span class="nf">Product</span><span class="p">(</span><span class="kt">int</span> <span class="n">id</span><span class="p">,</span> <span class="kt">string</span> <span class="n">name</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="n">id</span> <span class="p">=</span> <span class="n">id</span><span class="p">;</span>
<span class="k">this</span><span class="p">.</span><span class="n">name</span> <span class="p">=</span> <span class="n">name</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">class</span> <span class="nc">Program</span>
<span class="p">{</span>
<span class="k">static</span> <span class="n">List</span><span class="p"><</span><span class="n">Product</span><span class="p">></span> <span class="n">products</span> <span class="p">=</span> <span class="k">new</span> <span class="n">List</span><span class="p"><</span><span class="n">Product</span><span class="p">>();</span>
<span class="k">static</span> <span class="k">void</span> <span class="nf">Main</span><span class="p">(</span><span class="kt">string</span><span class="p">[]</span> <span class="n">args</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">WriteLine</span><span class="p">(</span><span class="s">"NOTE! KEEP WATCHING THE GC HEAP SIZE IN COUNTERS"</span><span class="p">);</span>
<span class="kt">string</span> <span class="n">answer</span> <span class="p">=</span> <span class="s">""</span><span class="p">;</span>
<span class="k">do</span><span class="p">{</span>
<span class="k">for</span><span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="p">=</span> <span class="m">0</span><span class="p">;</span> <span class="n">i</span> <span class="p"><</span> <span class="m">10000</span><span class="p">;</span> <span class="n">i</span><span class="p">++){</span>
<span class="n">products</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="k">new</span> <span class="nf">Product</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="s">"product"</span> <span class="p">+</span> <span class="n">i</span><span class="p">));</span>
<span class="p">}</span>
<span class="n">Console</span><span class="p">.</span><span class="nf">WriteLine</span><span class="p">(</span><span class="s">"Leak some more? Y/N"</span><span class="p">);</span>
<span class="n">answer</span> <span class="p">=</span> <span class="n">Console</span><span class="p">.</span><span class="nf">ReadLine</span><span class="p">().</span><span class="nf">ToUpper</span><span class="p">();</span>
<span class="p">}</span> <span class="k">while</span> <span class="p">(</span><span class="n">answer</span> <span class="p">==</span> <span class="s">"Y"</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>On Windows, we could have used tools like the <code class="language-plaintext highlighter-rouge">task manager</code>, or <code class="language-plaintext highlighter-rouge">procdump</code> to gather memory dumps and then reviewed them in WindDbg, as we have done in numerous case studies in the past like: <a href="/blog/2006/01/23/net-memory-leak-case-study-the-eventhandlers.html">.NET Memory Leak Case Study: The Event Handlers That Made The Memory balloon</a> or <a href="/blog/2007/08/13/asp-net-memory-investigation.html">ASP.NET Memory Investigation</a>.</p>
<p class="notice--warning"><strong>Edit:</strong> Paulo Morgado informed me that <a href="https://github.com/Sysinternals/ProcDump-for-Linux">ProcDump is available for Linux as well</a> - You learn something new every day :) thanks Paulo</p>
<p>So how do we do this on Linux?</p>
<h2 id="the-dotnet-debugging-tools">The dotnet debugging tools</h2>
<p>Luckily for us, the .NET team has released a number of dotnet tools for cross platform debugging:</p>
<ul>
<li><a href="https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump"><strong>dotnet dump:</strong></a> Collects and Analyzes memory dumps from .NET core applications</li>
<li><a href="https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-counters"><strong>dotnet counters:</strong></a> Collects or Monitors .NET performance counters</li>
<li><a href="https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-gcdump"><strong>dotnet gcdump:</strong></a> Collects a snapshot of the .NET GC heaps - you can open this snapshot in Visual Studio to analyze the “leak”</li>
<li><a href="https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-trace"><strong>dotnet trace:</strong></a> Collects profiling traces for a .NET Core process</li>
<li><a href="https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-symbol"><strong>dotnet symbol:</strong></a> Downloads .NET core symbols - useful when you debug a memory dump from another machine (in WinDbg or other native debuggers)</li>
<li><a href="https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-sos"><strong>dotnet sos:</strong></a> Installs the latest version of sos.dll for .NET Core debugging</li>
</ul>
<p>Many of these are useful when troubleshooting memory leaks, (in particular <code class="language-plaintext highlighter-rouge">dotnet gcdump</code> would have been a good choice here, but I will leave that for another post, and focus today on <code class="language-plaintext highlighter-rouge">dotnet counters</code> and <code class="language-plaintext highlighter-rouge">dotnet dump</code>).</p>
<h3 id="dotnet-counters">dotnet counters</h3>
<p><code class="language-plaintext highlighter-rouge">dotnet counters</code> helps you either monitor or collect .NET core performance counters, or ASP.NET core performance counters like the size of the GC heap (where all your .NET objects are stored), or the number of garbage collections, assemblies loaded, ASP.NET requests etc.</p>
<p>Some useful commands here are:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">dotnet counters ps</code> - list all the .NET Core processes we can monitor, so we can get the Process ID</li>
<li><code class="language-plaintext highlighter-rouge">dotnet counters monitor -p [Process ID] --refresh-interval 1</code> - displays all the counters and updates every second</li>
<li><code class="language-plaintext highlighter-rouge">dotnet counters monitor --counters Microsoft.AspNetCore.Hosting</code> - monitors the ASP.NET Core counters</li>
</ul>
<p class="notice--success"><strong>Note:</strong> Most of the commands have the ps switch that lets you list the processes that can be monitored</p>
<h3 id="dotnet-dump">dotnet dump</h3>
<p><code class="language-plaintext highlighter-rouge">dotnet dump</code> collects a memory dump similar to the dumps you collect with <a href="https://docs.microsoft.com/en-us/sysinternals/downloads/procdump">ProcDump</a> or <a href="https://www.microsoft.com/en-us/download/details.aspx?id=58210">DebugDiag</a> or any other debugging tool.</p>
<p>If you use it on Windows to collect memory dumps you can review the dumps in <a href="https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-download-tools">WindDbg</a> or DebugDiag or any dump debugging tool.</p>
<p>You can also review these dumps (both from Windows and Linux) in Visual Studio - I will write a more modern post on how this works nowadays, but you can see a walkthrough of this in one of my old posts <a href="https://www.tessferrandez.com/blog/2009/06/19/first-look-at-debugging-net-40-dumps-in-visual-studio-2010.html">Debugging .NET 4.0 dumps in Visual Studio 2010</a> as the concepts are still very similar.</p>
<p>However, the really neat thing is that you can also debug these dumps with <code class="language-plaintext highlighter-rouge">dotnet dump analyze [dumpfile]</code> both on Linux and Windows.</p>
<p>Once you are in the tool, it can feel a bit daunting if you’ve never used it before, but typing the command <code class="language-plaintext highlighter-rouge">help</code> will list all the commands you can use. The commands are extremely similar to the sos commands you use in WinDbg (except you don’t have to start the commands with <code class="language-plaintext highlighter-rouge">!</code> since it is not an extension here). So if you look through any of the <a href="https://www.tessferrandez.com/postindex/">case studies on this blog</a> you can most likely replicate it in dotnet dump.</p>
<h2 id="debugging-in-action">Debugging in action</h2>
<p>Here you can see the tools in action on Ubuntu 20.04 running in WSL2 on Windows.</p>
<script id="asciicast-Wq3owKkbCSecBV4jmVvpjqK4h" src="https://asciinema.org/a/Wq3owKkbCSecBV4jmVvpjqK4h.js" async=""></script>
<p class="notice--success"><strong>Note:</strong> This recording is not a video but an <a href="https://asciinema.org/">asciinema</a> capture, so you can stop the recording and select text at any point. I noticed that the recording unfortunately is a little too wide - so if you can’t see the counters they grow by 200 MB each round - you can also see the recording in <a href="https://asciinema.org/a/Wq3owKkbCSecBV4jmVvpjqK4h">full view here</a></p>
<ol>
<li>Reproduce the problem by running the application (every loop adds some more products to the static list)</li>
<li>Run <code class="language-plaintext highlighter-rouge">dotnet counters monitor -p [PID]</code> to start looking at the .NET Core counters - We can see that the GC Heap Size increases with around 200 MB each iteration</li>
<li>Run <code class="language-plaintext highlighter-rouge">dotnet dump collect -p [PID]</code> to collect a memory dump - watch the process informing us that we are capturing the dump. The dump is a snapshot of all the memory used by the process at a given point in time.</li>
<li>Run <code class="language-plaintext highlighter-rouge">dotnet dump analyze [filename]</code> to start analyzing the memory dump.</li>
</ol>
<p>At this point we are looking to answer two questions</p>
<ol>
<li>Where is our memory going?</li>
<li>Why is it not reclaimed?</li>
</ol>
<p>The following commands are commands we run inside the dotnet dump debugger</p>
<ol>
<li>We run <code class="language-plaintext highlighter-rouge">eeversion</code> just to check the version of .NET Core we are running (5.0) and also the version of the GC. In this case we see that it is Workstation mode - this informs us that we will see one GC heap and we won’t have dedicated GC threads. If you want to dig deeper into what all this means - check my old post on <a href="https://www.tessferrandez.com/blog/2008/04/17/how-does-the-gc-work.html">how the GC works here</a> if you want the cliff notes or <a href="https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md">Maoni’s much more detailed and up to date information about how the GC really works</a>.</li>
<li><code class="language-plaintext highlighter-rouge">eeheap -gc</code> lists the GC heaps where we can see how much GC memory we are using - and how much we are using for each generation. Of special interest here is the <strong>Large Ojbect Heap (LOH)</strong> where we store anything that is larger than 85000 bytes. Why is this interesting? Well, the LOH is collected way more seldom than the other heaps (only on full collections) - and if we use the LOH a lot, we end up “over triggering” the GC. Read more about that in my post about <a href="/blog/2006/06/22/aspnet-case-study-high-cpu-in-gc.html">High CPU in GC</a> or try it out for yourself with <a href="/blog/2008/02/22/net-debugging-demos-lab-4.html">the debugging labs</a>. In our case though we can see that we use about 404MB of .NET GC memory, but most of it is on the small object heaps.</li>
<li><code class="language-plaintext highlighter-rouge">dumpheap -stat</code> is very useful as it lists statistics of all the objects on the heap, in ascending order, with the objects that collectively use the most memory last. You need to be a bit careful when reading this though, we can see that we have 20.000 Product objects totaling 800.000 bytes. This only includes the memory used by the “raw” object - i.e. in this case each <code class="language-plaintext highlighter-rouge">Product</code> has a link to a String (<code class="language-plaintext highlighter-rouge">name</code>), an int (<code class="language-plaintext highlighter-rouge">id</code>) and a link to a System.Char[] (<code class="language-plaintext highlighter-rouge">details</code>), so the <code class="language-plaintext highlighter-rouge">Product</code> object will always be 40 bytes, independent of how big the <code class="language-plaintext highlighter-rouge">details</code> array is, or the <code class="language-plaintext highlighter-rouge">name</code> string. Nonetheless, a lot of our memory seems to be devoted to these Product objects (SURPRISE SURPRISE :)). <strong>So now we know the what - next is finding out the why</strong></li>
<li><code class="language-plaintext highlighter-rouge">dumpheap -type simple_memory_leak.Product</code> dumps a list of the addresses of every individual <code class="language-plaintext highlighter-rouge">Product</code> object. Thanks to the GC and its layout, we know exactly where all the objects are which lets us traverse the heaps and dump the objects like this. Doing this in a C++ app or some language that doesn’t have a managed heap would be impossible since we wouldn’t have a common list of all the allocated memory, but this is very neat for troubleshooting memory issues.</li>
<li>Next we grab an object at random… in reality we would do this a couple of times to validate our findings, and run <code class="language-plaintext highlighter-rouge">dumpobj [address]</code> - this way we can see the actual contents of the object, and we see the links to the member variables (name, id, details). And then we <code class="language-plaintext highlighter-rouge">dumpobj [address of the name string]</code> to get even more context - so this is <strong>product7844</strong> - but why is this hanging around on the heap, why isn’t it garbage collected?</li>
<li><code class="language-plaintext highlighter-rouge">gcroot [address]</code> either on the string or the Product, gives us a list of all the <strong>roots</strong>, or chains leading to the root, telling us why it is not collected yet. In this case we see that the <strong>String</strong> is belongs to a <strong>Product</strong> (as a member variable), and the Product belongs to a <strong>List of Products</strong> which in turn sits on the stack (rbp) on <strong>Thread 4559</strong>. So as long as that List is on the stack, and the Product is part of the list etc. we won’t garbage collect it, because it is still in use.</li>
<li><code class="language-plaintext highlighter-rouge">threads</code> lists all the threads in the process, and we can see that <strong>0x4559</strong> is also known as thread 0. The * tells us that this is the active thread, but if we would have been on another thread we could have ran <code class="language-plaintext highlighter-rouge">setthread 0</code> to switch to thread 0.</li>
<li>Finally we run <code class="language-plaintext highlighter-rouge">clrstack</code> to see what the thread is doing, and find that we are sitting in <code class="language-plaintext highlighter-rouge">Program.Main</code> so now we can go back to the code and check out what its doing.</li>
</ol>
<h2 id="summary">Summary</h2>
<p>This was an extremely simple case, and the memory leaks can get a bit more hairy, especially if you have a lot of different things on the heap that make it a bit harder to find the needle in the haystack. But, the technique stays the same whether it’s simple or harder. If you are working with .NET Core applications, try this out on your own applications and get to know your memory usage and your memory patterns a bit better, both to find potential issues to solve, but also to understand what good looks like, when things go bad.</p>
<p>Hope you have enjoyed this… if you have comments, or things you want me to blog about, reach out on <a href="https://twitter.com/TessFerrandez">twitter @tessferrandez</a> and we’ll continue the conversation there (until I find a good comment solution for the blog)</p>
<p>Have a good one, Tess</p>Tess FerrandezHi all,Reviving the blog and updating the Buggy Bits Debugging Labs2021-02-12T08:31:14+00:002021-02-12T08:31:14+00:00https://www.tessferrandez.com/blog/2021/02/12/updating-debugging-labs<p>Hi friends,</p>
<p>It’s been forever and a day, and as of many years ago I no longer work with debugging on a daily basis. A number of years back, the MSDN blogs were taken down, some if it was moved over to an archive, but around a year ago, most of it just disappeared.</p>
<p>I was constantly pushing the “save the blog away and put it somewhere else” to another day, until one day it was just too late and it was all gone.</p>
<p>However, I keep getting requests for articles that are linked from somewhere, and lately, with a new upcoming revision to debug diag, I needed to do something about it - so I am slowly restoring my blog from the way back machine. It’s a painstakingly slow manual process, and many of the links are now gone. But its also quite rewarding as I go through posts, reviewing all the nice comments everyone wrote. Big love to all of you who have read and commented on my blog throughout the years.</p>
<p>I am also trying to post all the <a href="https://github.com/TessFerrandez/ThePresidentsDebuggingDemo">sample code</a> in <a href="https://github.com/TessFerrandez/DebuggingDemos">github repos</a>, upload all the <a href="https://www.slideshare.net/TessFerrandez">slide decks</a>, recreate some of the <a href="https://github.com/TessFerrandez/OldDebuggingPresentations">demos</a> and <a href="https://github.com/TessFerrandez/DebuggingScripts">scripts</a> with modern frameworks and tools.</p>
<p>One such restoration was my <a href="2008-02-04-debugging-demos-setup-instructions.md">Buggy Bits labs</a> - that over the years were downloaded by somewhere around 100.000 people.</p>
<p>The instructions for the debugging labs <a href="/blog/2008/02/04/debugging-demos-setup-instructions.html">are available here</a>.</p>
<p>I still have about 5 or so blog-years to go for the restoration at this point - but I wanted to thank you for stopping by.</p>
<p>Nowadays I work as a developer, most of my projects are either services written in C# or Machine Learning and Data Related projects in Python. If you are curious about some of my current endeavors, you can check out some of my later presentations on the <a href="https://tessferrandez.github.io/presentations/">presentations</a> page. You can also find some of my debugging presentations there.</p>
<p>If you came here because you have a hang, crash, memory leak or something in between, have a look at the <a href="https://tessferrandez.github.io/postindex/">post index</a> for a more structured view of the posts on the blog.</p>
<p>The blog currently doesn’t have a comment function, but if you want to chat about something, you can reach me on twitter at <a href="https://twitter.com/tessferrandez">@TessFerrandez</a></p>
<p>Laters, Tess</p>Tess FerrandezHi friends,Entering geek territory or exiting geek territory? That’s the question…2011-05-06T08:31:14+00:002011-05-06T08:31:14+00:00https://www.tessferrandez.com/blog/2011/05/06/entering-geek-territory-or-exiting<p>After 12 years in support I decided to lift my head above the WinDbg window and take a look at what the rest of the world has to offer.</p>
<p>In fact, I decided not only to have a look but take a long long jump to the other end of the software spectrum, miles away from “My ASP.NET app hangs” to the world of “This new stuff that will be released in 2014 is so cool I’m gonna die”, and believe me, it’s a scary leap.</p>
<p>For the next two months I’ll be working in a team at Microsoft called DPE (Developer Evangelism), looking at what’s next in Web, Phone, Kinect, HTML5, you name it, still doing what I love most though, which is making people excited about technology and talking about what I learn along the way.</p>
<p>Just to explain how far away from my “day job” this is, I’ll share what my first assigment was… my first task in this new role was to design a character for this site we are starting up with all swedish content for people getting started with app and web development.</p>
<p>Several people have said, ooooh so you are going to join the geeks now. Am I? I thought it couldn’t get more geeky than spending your days in WinDbg, I guess I’ll just have to wait and see…</p>
<h2 id="what-does-this-mean-for-the-blog">What does this mean for the blog?</h2>
<p>To be honest I don’t know… It’s been so busy lately that I haven’t had time to take care of the blog as much as I should have, but I think it means that you will be seeing a lot more varied content here.</p>
<p>I am too in love with WinDbg and debugging to stop it all together, and the plan is for this stint to be temporary, but it might mean that I’ll lean more towards showing stuff that I work on in my current role for a while. Hope you don’t mind</p>
<p>Have a great weekend all, Tess</p>Tess FerrandezAfter 12 years in support I decided to lift my head above the WinDbg window and take a look at what the rest of the world has to offer.Reader email: Need help troubleshooting perf/memory issues2010-12-06T08:31:14+00:002010-12-06T08:31:14+00:00https://www.tessferrandez.com/blog/2010/12/06/reader-email-need-help-troubleshooting-perf-memory-issues<p>I get several emails every day (through the blog) with urgent requests to help troubleshoot memory issues or memory related performance issues. Some just say <strong>“We have a memory issue, how do I troubleshoot it?”</strong> and others are far far more detailed with links to dumps and everything :)</p>
<p>I love getting emails, but since I have a day job too (troubleshooting such issues :)) I unfortunately have to ignore most such requests through the blog :( to have time to help the customers that contact me through the normal support channels, and therefore I would recommend that if you do have a pressing issue, that you create a support case on <a href="http://support.microsoft.com/">http://support.microsoft.com/</a> to make sure that you get a dedicated support engineer and that your issue gets the attention it deserves.</p>
<p>Having said all that, I wanted to point out one email I got today, both because I could see that the person writing it had put some effort into try troubleshooting the issue before contacting me, and because I think and hope that my answer might be able to give some starting points to other people in his situation.</p>
<h2 id="the-email">The email</h2>
<p>Hi Tess,</p>
<p>We have a week to go for production and are stuck in one performance issue.
Please guide me in resolving this.</p>
<p>We are running an ASP .Net (2.0) App on Windows Server 2003 (SP2), dual-processor, 4GB RAM.
During stess-testing our app with 60 users, we see a drop in throughput from 17th minute to 22nd minute.
It rises and drops again around 40th minute.</p>
<p>I took 4 memory dumps for 8 mins, ran the perfmon counters.
Here are the details:</p>
<table>
<thead>
<tr>
<th> </th>
<th>Dump1</th>
<th>Dump2</th>
<th>Dump3</th>
<th>Dump4</th>
</tr>
</thead>
<tbody>
<tr>
<td>Size</td>
<td>1.16G</td>
<td>1.12G</td>
<td>1.04G</td>
<td>1.13G</td>
</tr>
<tr>
<td>GC</td>
<td>802MB</td>
<td>750MB</td>
<td>648MB</td>
<td>738MB</td>
</tr>
<tr>
<td>Loader</td>
<td>18MB</td>
<td>18MB</td>
<td>18MB</td>
<td>18MB</td>
</tr>
<tr>
<td>Image</td>
<td>236MB</td>
<td>236MB</td>
<td>236MB</td>
<td>236MB</td>
</tr>
<tr>
<td>MEM_RESV</td>
<td>220MB</td>
<td>236MB</td>
<td>347MB</td>
<td>255MB</td>
</tr>
<tr>
<td>MEM_COMM</td>
<td>1.19GB</td>
<td>1.06GB</td>
<td>1021MB</td>
<td>1.08GB</td>
</tr>
</tbody>
</table>
<p>Perfmon Counters (8 mins)</p>
<table>
<thead>
<tr>
<th>Counter</th>
<th>Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>LOH</td>
<td>307MB</td>
</tr>
<tr>
<td>#Bytes in all heaps</td>
<td>798MB</td>
</tr>
<tr>
<td>%Time in GC</td>
<td>49%(avg)</td>
</tr>
<tr>
<td>#Induced GC</td>
<td>14 (avg)</td>
</tr>
</tbody>
</table>
<p>So I can deduce the following:</p>
<ol>
<li>We are stuck in GC for the 8 minutes.</li>
<li>Assembly leak is not the problem.</li>
<li>
<p>Am not sure if Finalizer thread is being blocked.</p>
<p>The <code class="language-plaintext highlighter-rouge">kb</code> reads as below:</p>
<pre><code class="language-cmd"> 0:017> kb
ChildEBP RetAddr Args to Child
0e1afc74 7c827d0b 77e61d1e 00000334 00000000 ntdll!KiFastSystemCallRet
0e1afc78 77e61d1e 00000334 00000000 0e1afcbc ntdll!NtWaitForSingleObject+0xc
0e1afce8 79e8c5f9 00000334 000007d0 00000000 kernel32!WaitForSingleObjectEx+0xac
0e1afd2c 79e8c52f 00000334 000007d0 00000000 mscorwks!PEImage::LoadImage+0x1af
0e1afd7c 79e8c54e 000007d0 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117
0e1afd90 79fc2397 000007d0 00000000 00000000 mscorwks!CLREvent::Wait+0x17
0e1afdac 79fc24b4 000f2b28 0e1afeb0 000f3328 mscorwks!SVR::WaitForFinalizerEvent+0x4a
0e1afdc0 79e9845f 0e1afeb0 00000000 00000000 mscorwks!SVR::GCHeap::FinalizerThreadWorker+0x49
0e1afdd4 79e983fb 0e1afeb0 0e1afe5c 79f7759b mscorwks!Thread::DoADCallBack+0x32a
0e1afe68 79e98321 0e1afeb0 9b91703f 00000000 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
0e1afea4 79eef6cc 0e1afeb0 00000000 0010f770 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a
0e1afecc 79eef6dd 79fc2469 00000008 0e1aff14 mscorwks!ManagedThreadBase_NoADTransition+0x32
0e1afedc 79ef12ab 79fc2469 9b91718f 00000000 mscorwks!ManagedThreadBase::FinalizerBase+0xd
0e1aff14 79f92015 00000000 00000000 89244470 mscorwks!SVR::GCHeap::FinalizerThreadStart+0xbb
0e1affb8 77e64829 000f3328 00000000 00000000 mscorwks!Thread::intermediateThreadProc+0x49
0e1affec 00000000 79f91fcf 000f3328 00000000 kernel32!BaseThreadStart+0x34
</code></pre>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">!finalizequeue</code> results</p>
<pre><code class="language-cmd"> Heap 0
generation 0 has 158 finalizable objects (1d1d42c8->1d1d4540)
generation 1 has 269 finalizable objects (1d1d3e94->1d1d42c8)
generation 2 has 1961 finalizable objects (1d1d1ff0->1d1d3e94)
Ready for finalization 0 objects (1d1d4540->1d1d4540)
------------------------------
Heap 1
generation 0 has 830 finalizable objects (1d23d7c0->1d23e4b8)
generation 1 has 284 finalizable objects (1d23d350->1d23d7c0)
generation 2 has 1238 finalizable objects (1d23bff8->1d23d350)
Ready for finalization 0 objects (1d23e4b8->1d23e4b8)
</code></pre>
</li>
</ol>
<p>How do I proceed from here to isolate the issue?</p>
<p>Thanks,</p>
<h2 id="my-answer">My answer</h2>
<p>Looking at the above, these were the comments and suggestions for further troubleshooting that I sent him.</p>
<p>Hi,</p>
<p>I don’t agree that you are stuck in a GC for 8 minutes, you can check with <code class="language-plaintext highlighter-rouge"># Gen 0</code>/<code class="language-plaintext highlighter-rouge">Gen 1</code>/<code class="language-plaintext highlighter-rouge">Gen 2</code> collections and will probably see them increasing. The reason I say that you are not stuck is because the <code class="language-plaintext highlighter-rouge">% time in GC</code> counter will only update on completion of a GC so if you were stuck it would not update.</p>
<p>I do think however that your % Time in GC looks really high and I think there might be two reasons for this</p>
<ol>
<li>
<p>You are reaching the point where memory usage gets really close the OOM range (typically around 1 GB private bytes) so the process is probably working really hard to get rid of everything it can. <code class="language-plaintext highlighter-rouge"># Induced GC</code> here is not 0 which means that someone called <strong>GC.Collect</strong>, and unless it is your app that calls <strong>GC.Collect</strong> it is probably ASP.NET doing this to clean up at an OOM.</p>
<p>Each induced GC will be a full GC which can get pretty costly performance wise.</p>
<p class="notice--success"><strong>Recommendation:</strong> Look at general memory usage, i.e. look at <code class="language-plaintext highlighter-rouge">!dumpheap –stat</code> etc. and see what is using all this .net memory and try to find places where you can reduce the memory usage, or alternatively, if you need to use all this memory, then you would need to move to a 64-bit environment. (You might want to look at some of <a href="../post-index.md#memory-isues">my posts on troubleshooting .net memory issues</a> to narrow this down). A potential also, if you are running multiple sites here, would be to separate them into different app pools.</p>
</li>
<li>
<p>Your LOH looks pretty large (307 MB), if you are continuously allocating on the LOH you will also trigger a lot of full GC’s.</p>
<p class="notice--success"><strong>Recommendation:</strong> Have a look at <code class="language-plaintext highlighter-rouge">!dumpheap –min 85000</code> and see if there are any unexpected large objects, or if there are any that you can reduce in size or in number of objects. (See this post on <a href="/blog/2006/06/22/aspnet-case-study-high-cpu-in-gc.html">high cpu in GC</a> for more details)</p>
</li>
</ol>
<p>Regarding the finalizer, the finalizer is not stuck, it is sitting in <strong>mscorwks!SVR::WaitForFinalizerEvent+0x4a</strong> which means that it is just idling, waiting for the GC to tell it to finalize.</p>
<p>I have a post with common asp.net threads and how they look in <a href="/blog/2007/04/02/things-to-ignore-when-debugging-a-net-hang.html">the “normal” idling state here</a> that might be useful for further debugging.</p>
<p>You can further see that finalization is not a problem because <code class="language-plaintext highlighter-rouge">!finalizequeue</code> reports no objects to ready for finalization in either heap <strong>“Ready for finalization 0 objects”</strong>. Here you can find an <a href="/blog/2006/03/26/net-memory-leak-unblock-my-finalizer.html">example of what it would look like if the finalizer was actually stuck</a>.</p>
<p>Hope that helps, Tess</p>Tess FerrandezI get several emails every day (through the blog) with urgent requests to help troubleshoot memory issues or memory related performance issues. Some just say “We have a memory issue, how do I troubleshoot it?” and others are far far more detailed with links to dumps and everything :)Fix for RestartWWWService=false in Visual Studio 20102010-11-09T08:31:14+00:002010-11-09T08:31:14+00:00https://www.tessferrandez.com/blog/2010/11/09/fix-for-restartwwwservice<p>We recently published a fix for the following problem:</p>
<p>You create a deployment project for a web application in Visual Studio 2010, set the deployment property RestartWWWService=false and deploy to a Windows 2003 server.
<a href="http://msdn.microsoft.com/en-us/library/22f9106b.aspx">http://msdn.microsoft.com/en-us/library/22f9106b.aspx</a></p>
<p>In this situation the w3wp.exe process will still be recycled even though the RestartWWWService is set to false.</p>
<p>The fix is <a href="http://connect.microsoft.com/VisualStudio/Downloads/DownloadDetails.aspx?DownloadID=32285">available here</a> and should only be installed on development machines if you experience this problem.</p>
<p>Have a good one, Tess</p>Tess FerrandezWe recently published a fix for the following problem:Debug Diag 1.2 (Beta)2010-09-30T08:31:14+00:002010-09-30T08:31:14+00:00https://www.tessferrandez.com/blog/2010/09/30/debug-diag-12-beta<p>Are you looking for some good stuff to put in your family’s Christmas stockings this year? Maybe a new phone, a controller for your gaming console, why not a brand new debugging tool :)</p>
<p>Jokes aside, I get peppered with questions about Debug Diag and if there is a new version coming that will support Windows 7 / Windows 2008 and luckily some of my EE colleagues have been and are still working hard at making this happen.</p>
<p>While it is not available for public download yet at the MS download site, Beta 1 of Debug Diag has been released and is ready for use. The main new stuff in Beta 1 is that it now supports the aforementioned Windows 7 and 2008, and if you need it right now, you can send an email to dbgdiag (at) microsoft.com, or if you have a case open, you can ask the support engineer you are working with to send it to you.</p>
<p>It has a 32-bit and a 64-bit version (for 1.2, install the 64bit version on 64bit OSes and 32bit version on 32bit OSes, independently of the bitness of the target process).</p>
<p>I will write more when Beta2 is out but Beta2 looks very promising with some new features that will help our apps easier to debug.</p>
<p>Tess</p>Tess FerrandezAre you looking for some good stuff to put in your family’s Christmas stockings this year? Maybe a new phone, a controller for your gaming console, why not a brand new debugging tool :)Capturing memory dumps for 32-bit processes on an x64 machine2010-09-29T08:31:14+00:002010-09-29T08:31:14+00:00https://www.tessferrandez.com/blog/2010/09/29/capturing-memory-dumps-for-32-bit-processes-on-an-x64-machine<p>This is an issue that I often get questions around and we often have cases where we have to re-capture memory dumps because the memory dumps were captured the “wrong” way.</p>
<p>The short story is: If you are executing a 32-bit process on a 64-bit machine, (which is the default case for IIS on x64 machines for example) you need to capture the dump with a tool that allows you to create 32-bit dumps.</p>
<h2 id="how-do-you-know-if-your-process-is-32-bit">How do you know if your process is 32-bit</h2>
<p>If you are on a 64-bit machine, you can check task manager to see what architecture your process is using.</p>
<p><img src="/assets/images/32bit.png" alt="image" /></p>
<p>Processes with *32 are 32-bit and the rest are 64-bit so in the example above we can see that for example w3wp.exe is executing 32-bit code.</p>
<h2 id="why-is-it-important-to-capture-them-with-the-right-tools">Why is it important to capture them with the right tools</h2>
<p>If you capture a dump with a tool that captures 64-bit dumps you will still get a memory dump, but you will get a memory dump of the syswow64 which means that a lot of extensions like sos and psscor2 won’t be able to read the data. Some things may still work but it is very limited and you might get errors or erroneous call stacks etc.</p>
<p>Typical stuff that happens when you try to read a 64-bit memory dump of a 32-bit process is:</p>
<ol>
<li>
<p>You may see warnings like</p>
<pre><code class="language-cmd"> WARNING: wkscli overlaps srvcli
..............WARNING: wship6 overlaps dnsapi
.WARNING: IPHLPAPI overlaps dsrole
...WARNING: FWPUCLNT overlaps rasadhlp
WARNING: FWPUCLNT overlaps dnsapi
.....WARNING: compstat overlaps iisres
</code></pre>
</li>
<li>
<p>The stacks show wow64cpu methods</p>
<pre><code class="language-cmd"> 0:000> kp
Child-SP RetAddr Call Site
00000000`000ce728 00000000`73a22bcd wow64cpu!CpupSyscallStub+0x9
00000000`000ce730 00000000`73a9d07e wow64cpu!Thunk0ArgReloadState+0x1a
00000000`000ce7f0 00000000`73a9c549 wow64!RunCpuSimulation+0xa
00000000`000ce840 00000000`76d684c8 wow64!Wow64LdrpInitialize+0x429
00000000`000ced90 00000000`76d67623 ntdll!LdrpInitializeProcess+0x17e2
00000000`000cf290 00000000`76d5308e ntdll! ?? ::FNODOBFM::`string'+0x2bea0
00000000`000cf300 00000000`00000000 ntdll!LdrInitializeThunk+0xe
</code></pre>
</li>
<li>
<p>You see that all addresses are 64 bit addresses, (i.e. 00000000`76d5308e rather than 76d5308e), even though it is a 32-bit process.</p>
</li>
<li>
<p>You get errors like the below when trying to run sos commands.</p>
<p class="notice--info"><strong>Note:</strong> You can get these errors if you don’t have symbols properly set up as well, so this is not the only reason for getting these errors)</p>
<pre><code class="language-cmd"> 0:000> !eeheap -gc
Failed to load data access DLL, 0x80004005
Verify that
1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of clr.dll is in the version directory
3) or, if you are debugging a dump file, verify that the file mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
4) you are debugging on the same architecture as the dump file. For example, an IA64 dump file must be debugged on an IA64 machine.
</code></pre>
</li>
</ol>
<p>You can also run the debugger command <code class="language-plaintext highlighter-rouge">.cordll</code> to control the debuggers load of mscordacwks.dll. <code class="language-plaintext highlighter-rouge">.cordll -ve -u -l</code> will do a verbose reload.</p>
<p>If that succeeds, the SOS command should work on retry.</p>
<p>If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.</p>
<h2 id="what-tools-should-you-use">What tools should you use</h2>
<p>The best way to capture a process dump in this scenario is to use 32-bit versions of tools like <strong>Debug Diag</strong>, or <strong>adplus+cdb</strong> (32 bit debugging tools for windows). You can install the 32-bit versions of these tools on a 64-bit system. In the next version of Debug Diag you will be able to capture 32-bit dumps with the 64-bit version of Debug Diag.</p>
<p>I have mentioned before that you can capture memory dumps with <strong>task manager</strong> on Vista+, but if you do this on a 64-bit machine you will get a 64-bit dump. You can however use the 32-bit task manager, located in <strong>C:\Windows\SysWOW64\taskmgr.exe</strong> to get 32-bit dumps. If you want to verify that you are running the 32-bit version of task manager, you can check that <strong>taskmgr.exe</strong> is listed with *32 in task manager itself.</p>
<p><strong>Edit:</strong> Sorry Mark and David, I of course didn’t mean to leave out the nice and very lightweight <strong>procdump</strong> tool from SysInternals which will capture 32-bit dumps of 32-bit processes even on 64-bit Windows. Thanks for reminding me :)</p>
<p>For some reason I thought that I had written about this tool in an earlier post but a quick search reminded me that this is still in my blog-todo bin.</p>
<p class="notice--success"><strong>Edit 2021:</strong> the previous edit was based on a conversation in the comments from some of my debugging idols David Solomon and Mark Russinovich - I saved the comments below this post, because you know…</p>
<p>For all other dump capturing tools, I would suggest that you read the documentation to see if there is a 32-bit version (like for procmon where you can pass in the <code class="language-plaintext highlighter-rouge">/run32</code> parameter) or if they will capture 32-bit dumps of 32-bit processes.</p>
<h2 id="is-it-important-to-read-the-dump-with-the-right-bit-debugger">Is it important to read the dump with the right “bit” debugger</h2>
<p>Yes, to read 32-bit dumps you should use 32-bit windbg (either on a 32-bit or 64-bit OS), and for 64-bit dumps, the machine that you debug the dumps on, needs to be a 64-bit machine, running 64-bit windbg.</p>
<p>Remember that you should also use the 64-bit versions of <strong>sos</strong>, <strong>psscor2</strong>, <strong>sosex</strong> and any other extensions that you might use, if you are debugging a 64-bit dump.</p>
<p>Have a good one,
/Tess</p>
<hr />
<p><strong>David Solomon 29 Sep 2010 10:37 AM</strong></p>
<p>Tess - am really surprised (Russinovich was saddened) that you don’t mention Procdump in your article - that surely would be the preferred tool since it is so lightweight (no install like the tools you suggestion). So just curious - why didn’t you recommend it?</p>
<p>Also, technical detail: there is no such thing as a 32-bit process on 64-bit Windows - a Wow64 process is a full 64-bit process who’s threads start out in 64-bit mode but then switch to 32-bit mode to execute 32-bit code. And when the 32-bit code calls system calls, a switch is made back to 64-bit mode to call 64-bit OS services. You say above in #3 “You see that all addresses are 64 bit addresses, (i.e. 00000000`76d5308e rather than 76d5308e), even though it is a 32-bit process” - in fact, this is proof that it is a 64-bit thread (Windbg can show the 32-bit stack as well).</p>
<p><strong>Mark Russinovich 29 Sep 2010 10:42 AM</strong></p>
<p>And procdump has a significant number of other features that make it much more flexible and applicable to more scenarios….</p>
<p>-Mark (with a heavy heart)</p>
<p><strong>Tess 29 Sep 2010 11:58 PM</strong></p>
<p>Mark, David,</p>
<p>I was actually thinking of adding it to the list before along with a few other dump tools, but the list got a bit long, so I just included debug diag and adplus and left the others under (other dump creation tools).</p>
<p>The reason I singled debug diag and adplus out was because of the types of issues i normally deal with where you often need to do leak tracking or dump on specific exceptions (handled). For gathering dumps on “hangs”, or straight snapshots for memory analysis we usually use task manager (because it is already there). Having said that, I personally like procdump a lot and use it very often, and I do agree that it does have a lot of neat features like gathering dumps on memory/CPU thresholds and based on performance counter info.</p>
<p>As I wrote in the post (in the edit) I thought I had written a post about it already but just realized it is still in my todo bin, but I think it does merit it’s own post.</p>
<p>On a separate note, since I love the nitpicking :) perhaps you should change the help then for procdump technet.microsoft.com/…/dd996900.aspx from “By default Procdump will capture a 32-bit dump of a 32-bit process when running on 64-bit Windows.” to … a 32-bit dump of a 64-bit process running 32-bit code:) … sorry couldn’t resist, thanks for the clarification btw…</p>Tess FerrandezThis is an issue that I often get questions around and we often have cases where we have to re-capture memory dumps because the memory dumps were captured the “wrong” way.Help! !clrstack is not showing any method names2010-08-25T08:31:14+00:002010-08-25T08:31:14+00:00https://www.tessferrandez.com/blog/2010/08/25/help-clrstack-is-not-showing-any-method-names<p>From time to time I get questions about missing method names in the sos !clrstack output, like this</p>
<pre><code class="language-cmd">0:019> !clrstack
OS Thread Id: 0x5d0 (19)
ESP EIP
0f31f340 7c80bef7 [HelperMethodFrame_1OBJ: 0f31f340]
0f31f398 1449167a
0f31f3c4 144915e6
0f31f3f4 10d81b48
0f31f4b4 793e25cf
0f31f4bc 79366b3d
0f31f4d4 793e2734
0f31f4e8 793e26ac
0f31f678 79e7c0e3 [GCFrame: 0f31f678]
</code></pre>
<p>Jonathan Dickinson from SourceCode, just emailed me and told me that he had <a href="http://jonathan.dickinsons.co.za/blog/2010/08/windbg-stack-fix/">written an article about this and how to resolve it</a>. Thanks Jonathan for writing that article.</p>
<h2 id="solution">Solution</h2>
<p>As mentioned in the article there are a couple of things you need to get rid of the problem.</p>
<h3 id="the-correct-version-of-mscordacwksdll">The correct version of MSCORDACWKS.dll</h3>
<p>You can get this in two ways</p>
<ol>
<li>From the Microsoft public symbol server by setting the symbol path to <code class="language-plaintext highlighter-rouge">srv*c:\mycache*http://msdl.microsoft.com/download/symbols</code></li>
<li>By copying it from the machine you got the dump from, it is located in the <code class="language-plaintext highlighter-rouge">Framework\<version></code> directory. The <strong>mscordacwks.dll</strong> needs to be copied to your symbol path, and to differentiate them if you work with multiple versions you can name it <strong>mscordacwks_x86_x86_2.0.50727.42.dll</strong> for example. On my machine, because I am working with so many different versions I have a <code class="language-plaintext highlighter-rouge">c:\dac\x86</code> and a <code class="language-plaintext highlighter-rouge">c:\dac\x64</code> directory where I store these.</li>
</ol>
<h3 id="a-compatible-version-of-sosdll">A compatible version of sos.dll</h3>
<p><strong>sos.dll</strong> can be found in your <code class="language-plaintext highlighter-rouge">framework\<version></code> directory and needs to match the architecture and major version of the framework used in the dump. If there is too big of a difference between the minor versions of the framework on your machine and the machine where you got the dump, you can try copying the <strong>sos.dll</strong> from the machine that the dump was taken from. Or you can use <a href="https://www.microsoft.com/en-us/download/details.aspx?id=21255"><strong>psscor2.dll</strong></a>.</p>
<p>Have a good one, Tess</p>Tess FerrandezFrom time to time I get questions about missing method names in the sos !clrstack output, like this