Recently at work we had some serious production environment issues. Our web application basically made the IIS application pool terminate unexpectedly. And we didn't get a single stack trace or a line of code. The only thing that was logged inside of the Windows Event Viewer, was event id 1009, which basically told that the application pool serving our app terminated unexpectedly. And that was it. All active user sessions were wiped, and the users had to login again.

The error started after we did a major upgrade. A lot of our code had changed since the previous version, so we set up an new website and application pool inside IIS 6.0 for the new version, so we could upload the new application, and then turn off the old one, to get as seamless an upgrade as possible. It worked fine, and we were happy. Until a few hours later, when we saw our users were logged off.

At first we thought it was IIS settings we had done wrong, so we did a complete comparison with the old website and application pool to see if we forgot anything. There was no difference at all. After a while, we realized that the only thing that could make the IIS crash like that, was our own code. But with no clue of where the danger in our code were - we were lost.

After some time, we got WinDbg attached to the worker process of IIS, serving our application. We caught a few memory dumps, but those were not of the exception we were looking for.

Later I found an HttpModule for ASP.NET. Basically the .Net framework 2.0 has changed the way it handles exceptions from other threads in IIS. In ASP.NET 1.1, unhandled exceptions from asynchronous threads inside IIS were ignored. But in ASP.NET 2.0, those unhandled exceptions makes the IIS application pool crash.

Make the IIS crash yourself

So try to do this. Create a new ASP.NET 2.0 website running on your local IIS. You only need a single page, so you're fine with the default.aspx Visual Studio creates for you. Add a button to the page, and create an event handler for the buttons OnClick event.

Add this to your code-behind:

  /// <summary>
  /// Handles the Click event of the btnMakeCrash control.
  /// </summary>
  /// <param name="sender">The source of the event.</param>
  /// <param name="e">The EventArgs instance containing the event data.</param>
  protected void btnMakeCrash_Click(object sender, EventArgs e)
  {
    ThreadPool.QueueUserWorkItem(new WaitCallback(MakeIisCrash));
  }

  /// <summary>
  /// Makes the IIS crash.
  /// </summary>
  /// <param name="stateInfo">The state info.</param>
  private void MakeIisCrash(object stateInfo)
  {
    // Instantiate the DataSet to null
    DataSet ds = null;

    // Make an unhandled NullReferenceException
    ds.CaseSensitive = true;
  }

The buttons click event handler uses the ThreadPool to execute the MakeIisCrash() method. This method instantiates a DataSet as null, and the sets a property. Since the DataSet is null, this throws a NullReferenceException which is not handled, as you can see.

Click the button, and see how IIS will crash.

You can sense that your computer is working harder - that is because IIS terminates the application pool. If you go and check the Windows Application log, you can see that is has added an error:

Now this doesn't tell you much. Imagine if you had hundreds of thousands line of code - how would you find the cause of the error?

Inside Visual Studio, add a new class to the website - call it UnhandledExceptionModule, and apply this code:

#region Using

using System;
using System.Diagnostics;
using System.Globalization;
using System.IO;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading;
using System.Web;

#endregion

/// <summary>
/// Handles all unhandled exceptions from in the current AppDomain. 
/// 
/// Works great to catch unhandled exceptions thrown by IIS's child threads, 
/// which will make the application pool terminate unexpectedly 

/// without logging. This makes sure your Exception 
/// is logged to the Application event log.
/// </summary>
public class UnhandledExceptionModule : IHttpModule
{
  #region Fields

  private static int _UnhandledExceptionCount = 0;
  private static string _SourceName = null;
  private static object _InitLock = new object();
  private static bool _Initialized = false;

  #endregion

  #region IHttpModule members

  public void Init(HttpApplication app)
  {

    // Do this one time for each AppDomain.
    if (!_Initialized)
    {
      lock (_InitLock)
      {
        if (!_Initialized)
        {
          string webenginePath = Path.Combine(RuntimeEnvironment.GetRuntimeDirectory(), "webengine.dll");

          if (!File.Exists(webenginePath))
          {
            throw new Exception(String.Format(CultureInfo.InvariantCulture, 
                "Failed to locate webengine.dll at '{0}'.  This module requires .NET Framework 2.0.", webenginePath));
          }

          FileVersionInfo ver = FileVersionInfo.GetVersionInfo(webenginePath);
          _SourceName = string.Format(CultureInfo.InvariantCulture, "ASP.NET {0}.{1}.{2}.0", ver.FileMajorPart, 
                ver.FileMinorPart, ver.FileBuildPart);

          if (!EventLog.SourceExists(_SourceName))
          {
            throw new Exception(String.Format(CultureInfo.InvariantCulture, 
                "There is no EventLog source named '{0}'. This module requires .NET Framework 2.0.", _SourceName));
          }

          AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(OnUnhandledException);

          _Initialized = true;
        }
      }
    }
  }

  public void Dispose()
  {
  }

  #endregion

  #region UnhandledException event handler

  public void OnUnhandledException(object o, UnhandledExceptionEventArgs e)
  {
    // Let this occur one time for each AppDomain.
    if (Interlocked.Exchange(ref _UnhandledExceptionCount, 1) != 0)
      return;

    StringBuilder message = new StringBuilder("\r\n\r\nUnhandledException logged by UnhandledExceptionModule:\r\n\r\nappId=");

    string appId = (string)AppDomain.CurrentDomain.GetData(".appId");
    if (appId != null)
    {
      message.Append(appId);
    }

    Exception currentException = null;
    for (currentException = (Exception)e.ExceptionObject; currentException != null; currentException = currentException.InnerException)
    {
      message.AppendFormat("\r\n\r\ntype={0}\r\n\r\nmessage={1}\r\n\r\nstack=\r\n{2}\r\n\r\n",
                           currentException.GetType().FullName,
                           currentException.Message,
                           currentException.StackTrace);
    }

    EventLog Log = new EventLog();
    Log.Source = _SourceName;
    Log.WriteEntry(message.ToString(), EventLogEntryType.Error);
  }

  #endregion
}

Modify the httpModules section inside web.config to look lige this:

        <httpModules>
            <add type="UnhandledExceptionModule" name="UnhandledExceptionModule"/>
        </httpModules>

This is your new friend. It is a must for any ASP.NET 2.0 application running in production environment. This will catch your exception, and write the message and the stack trace to the Windows Application Event log, and now you can see what caused the crash:

So after we applied this HttpModule to our production environment, we get a nice entry in our Application event log from ASP.NET. It includes a stack trace, which makes us able to find the problem, and fix it!

Further reading regarding ASP.NET production environment issues:

• Tess Ferrandez - She has a lot of great posts about debugging ASP.NET using WinDbg 
• WinDbg
• David Wang - HOWTO: Understand and Diagnose an Application Pool Crash 
• Advanced .NET debugging

Hope this will help someone.

If you haven't heard .NET Rocks before, now is a good time to do it. In their last show, they interviewed Brad Abrams during Remix Boston where he did a keynote. So who is Brad Abrams.

Quote from his own blog:

Brad Abrams was a founding member of both the Common Language Runtime, and .NET Framework teams at Microsoft Corporation where he is currently the Group Program Manager for the UI Framework and Services team which is responsible for delivering the developer platform that spans both clients and web based applications as well as the common services that are available to all applications. Specific technologies owned by this team include ASP.NET and ASP.NET AJAX, parts of Silverlight, and Windows Forms.

In the interview, they talk a lot about Silverlight, what they are trying to do and how open the technology actually is. And they talk a lot about that openness Microsoft has recently approached, and how the Silverlight technology could be adapted by a developer of any of the three major platforms. (Windows, Mac & Linux). I really like that approach, and the result is clear. Already - even though Silverlight just hit the 1.0 RTM recently, there's a huge amount of applications running live from notable companies.

They also talk a lot about Brad Abrams' keynote at Remix Boston, where he did a demo on debugging a Silverlight 1.1 app.

And he didn't do it the easy way! He had a Linux box with a small PHP page, hosting the Silverlight 1.1 app. That worked great. The he fired up a Mac, and hit that page from Safari. And to achieve the debugging experience of Visual Studio, he attached the VS debugger to the Safari process on the Mac, set a breakpoint in the code, and refreshed the page on the Mac, to hit the breakpoint. And there you go. A Mac hitting a breakpoint on a Vista box, and the website being hosted on a Linux box - how cool is that?

During the show Richard Campbell said that he didn't believe it would work. I don't know - but I want to try the same thing, and se if it really works.