Memory gates checking failed because the free memory is less than 5% of total memory

Question

·         Business
Goal: Achieve a stable environment
·         Repro
Steps: n/a
·         Expected
Results: I expect the MSCRM Async services to use a relatively static amount of RAM
·         Actual
Results: Over the weekend, the memory hit critical mass and CRM failed
·         Environment:
 Windows Server 2008, SQL Server 2008, CRM 2013 On Premise - both on the same box, Email Router is installed on the same box. We have not yet applied either of the two new UR for 2013.
I'm currently experiencing a memory leak in our production environment. I will fill out the details shortly.
Here's the error:
Log Name:      Application
Source:        System.ServiceModel 4.0.0.0
Date:          4/20/2014 11:39:14 PM
Event ID:      3
Task Category: WebHost
Level:         Error
Keywords:      Classic
User:          NETWORK SERVICE
Computer:      server.domain.com
Description:
WebHost failed to process a request.
Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/58366981
Exception: System.ServiceModel.ServiceActivationException: The service '/VisualWorkplace/XRMServices/2011/Organization.svc' cannot be activated due to an exception during compilation. 
The exception message is:Memory gates checking failed because the free memory (1240145920 bytes) is less than
5% of total memory.  As a result, the service will not be available for incoming requests.  To resolve this, either reduce the load on the machine or adjust the value of minFreeMemoryPercentageToActivateService on the serviceHostingEnvironment config
element.. ---> System.InsufficientMemoryException: Memory gates checking failed because the free memory (1240145920 bytes) is less than 5% of total memory.  As a result, the service will not be available for incoming requests.  To resolve
this, either reduce the load on the machine or adjust the value of minFreeMemoryPercentageToActivateService on the serviceHostingEnvironment config element.
   at System.ServiceModel.Activation.ServiceMemoryGates.Check(Int32 minFreeMemoryPercentage, Boolean throwOnLowMemory, UInt64& availableMemoryBytes)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.CheckMemoryCloseIdleServices(EventTraceActivity eventTraceActivity)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath, EventTraceActivity eventTraceActivity)
We just recently increased the available memory from 8GB to 24GB to help ensure that there is enough memory. This error never happened on CRM v4. According to the resource monitor, SQL is only reserving 20.7GB of RAM and there is 694MB of RAM in Standby.
The main Async service is using 660MB and the maintenance Async service is using 265MB.
To resolve the issue temporarily, I restarted both Async services this morning. Shortly after everything came back, they were holding fairly steady around 350MB and 200MB. It's been six hours since then.
The last time I spoke with Microsoft about a similar problem, they suggested increasing the amount of RAM.

Answer 1

It became more and more of a problem. It was impacting the users in a more subtle way at first. You couldn't configure the CRM for Outlook client. After a while, they couldn't use CRM in Outlook at all and users reporting not being able to complete phone
call activities. I opened a ticket with MS and they suggested the same as Wilson Lou did. The organization error had nothing to do with the creep on the async services usage of RAM. It was indeed caused by SQL stealing all of the memory. Limiting the max usage
to 24GB RAM has resolved that error. 
My Async processes are still using a LOT of RAM, but now that I have more than enough RAM, it doesn't seem to be a problem anymore. They are currently using about 700MB RAM each.
Also, I have upgraded to SP1, UR2.
Eric Hula - CRM Administrator