This bug appears only in load-balanced / remote events setups for CMS versions between 7.5 and 9.0.2 (released September 14th 2015, in which it was fixed). It’s reported as bug CMS-1405. This memory leak is relatively slow. Because of the way that ASP.NET handles cache, the application’s memory allocation will remain constant while the cache portion of it will continuously decrease ending up in an application with no cache at all.

Here’s a quick patch that fixes the issue for sites running this code:

using EPiServer.Framework;
using EPiServer.Framework.Timers;
using EPiServer.ServiceLocation;
using System;
using EPiEvents = EPiServer.Events;

namespace Hotfix.CMS1405.EPiServer.Events
{
    [InitializableModule]
    public class PatchInitializer : IInitializableModule
    {
        public void Initialize(global::EPiServer.Framework.Initialization.InitializationEngine context)
        {
            context.InitComplete += context_InitComplete;
        }

        void context_InitComplete(object sender, EventArgs e)
        {

            var eps = EPiEvents.Providers.EventProviderService.Instance;
            var emvField = eps.GetType().GetField("_messageValidator", System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic);
            if (emvField == null)
            {
                throw new InvalidOperationException("Could not locate _messageValidator variable; this patch might not be valid for the current application.");
            }

            var emv = emvField.GetValue(eps);
            var timerField = emv.GetType().GetField("_sequenceCheckTimer", System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic);
            if (timerField == null)
            {
                throw new InvalidOperationException("Could not locate _sequenceCheckTimer variable to patch; this patch might not be valid for the current application.");
            }

            var timer = (ITimer)timerField.GetValue(emv);
            if (timer.Enabled)
            {
                throw new InvalidOperationException("Timer already enabled; this patch might not be valid for the current application.");
            }
            timer.Enabled = true;
        }

        public void Preload(string[] parameters)
        {
        }

        public void Uninitialize(global::EPiServer.Framework.Initialization.InitializationEngine context)
        {
        }

    }
}

Recognizing it

After leaving your application running for some time, take a full user dump of the application (using through Task Manager/Right Click w3wp.exe/Create Dump File, ProcDump -ma and the like).
Open it up in WinDBG, and load the SOS extension. Being lazy, I first let the .loadby sos clr attempt loading and fail. It fails because the dump I analyze was taken off an Azure server, which has the system loaded on the D:-drive. Copy the path, sans the first character which I change to a C, and the SOS extension loads properly.

0:000> .loadby sos clr      
The call to LoadLibrary(D:\Windows\Microsoft.NET\Framework64\v4.0.30319\sos) failed, Win32 error 0n126       
 "The specified module could not be found."      
Please check your debugger configuration and/or network access.      
0:000> .load c:\Windows\Microsoft.NET\Framework64\v4.0.30319\sos      

Then I call !dumpheap -stat . After crunching for a while, it will write out a table with all the different types of objects in your application, sorted by the biggest space-takers in the bottom:

0:000> !dumpheap -stat
Statistics:
              MT    Count    TotalSize Class Name
... and close to the bottom, you'll find ...
000007f8f37e98a0  3204485    102543520 EPiServer.Events.Providers.EventSequence
000007f8f37ed210  3204485    153815280 System.Collections.Generic.LinkedListNode`1[[EPiServer.Events.Providers.EventSequence, EPiServer.Events]]

In this dump, I found more than 3 million instances of each of these types. Together, they take up 102+153 MB of memory. You would typically not expect any more instances than the number of remote events that could be emitted within a 5 second time span. That number is well below 3 million! That’s 255 MB wasted!

Downloads

Source package that you may open and build yourself.
Binary package with a DLL that’s simply installed into the /bin/-folder.