Sitecore high memory usage – not always a problem

Sitecore is known to be a real memory hog. Especially the CM server is known to love memory. One of the memory issues that Sitecore faces is the high usage of the Large Object Heap.

Heap Fragmentation

WHY IS THIS A PROBLEM?

The .net garbage collector will start collecting memory from Generation 0. Objects that cannot be collected is upgraded to Generation 1, which is then collected later. Objects can be further upgraded to Generation 2, assuming that these objects are long-lived and does not need to be collected as often.

The Large Object Heap is where objects larger than 85.000 bytes live. This is mostly strings and arrays, and these objects are rarely garbage collected, and the heap is never compacted (not entirely true as newer .net versions can compact this heap) with can lead to high memory usage.

WHY THIS IS PROBABLY NOT A PROBLEM ANYWAY?

Sitecore is well-known to be a large object heap eater, because Sitecore stores fields and field values as arrays of strings, making every Item a large object. And this is a problem – right?

It definitely can be a problem – but not necessarily:

• The Large Object heap IS garbage collected, but not very often. It will be garbage collected when the dynamic threshold is met, AND when you run low on memory.
• .NET will consume the memory available. There is no need for a garbage collection if you are not running low on memory. It can be nerve wrecking to see the memory being low on your server, but this is often not a problem, it is by design.
• The high fragmentation of your large object heap is a sign that objects are being freed, leaving holes in the heap (so at least your application is not leaking memory). These holes will be filled with new objects if they fit into the hole.
• Memory is becoming increasingly cheaper, making the business case of memory optimization smaller and smaller. Memory leaks is still a problem, but memory usage is not.

HEY, IT’S A PROBLEM FOR ME. WHAT CAN I DO ABOUT IT?

But let’s say that memory IS a problem – what can you do about it?

TUNE YOUR CACHES:

If memory is a problem, you should not let the cache sizes run free. Set Caching.DisableCacheSizeLimits to false and tune your Sitecore caches:

<setting name="Caching.DisableCacheSizeLimits" value="false" />

I have provided a list of links to cache tuning guides at the bottom of this post.

REUSE DUPLICATED FIELD VALUES:

.NET introduced a string object pool pattern that allows you to reference strings that are duplicated. Strings cannot be changed once created (also called “immutable objects”), and can therefore be referenced using interning. But only if you code it that way.

Sitecore have implemented an interning pattern in the \App_Config\Sitecore\CMS.Core\Sitecore.Interning.config file where fields that often have duplicated values can be stored:

<interning>
  <fieldIdsToIntern>
    <workflowState>{3E431DE1-525E-47A3-B6B0-1CCBEC3A8C98}</workflowState>
    <workflow>{A4F985D9-98B3-4B52-AAAF-4344F6E747C6}</workflow>
    <updatedBy>{BADD9CF9-53E0-4D0C-BCC0-2D784C282F6A}</updatedBy>
    <createdBy>{5DD74568-4D4B-44C1-B513-0AF5F4CDA34F}</createdBy>
    ...
    ...
  </fieldIdsToIntern>
</interning>

You can add your own duplicated fields to the list. Nikolay Mitikov created this SQL to find duplicated values:

WITH DuplicatedFieldValues AS  (
SELECT 
	v.FieldId,	
	FieldDefinitionRow.[Name],
	CONVERT(NVARCHAR(250),v.[Value]) AS [Field Value],
	COUNT(1) AS [Hits]
FROM 
	[VersionedFields] v 
JOIN 
	[Items] FieldDefinitionRow ON FieldDefinitionRow.ID = v.fieldID
WHERE 
	v.FieldId NOT IN 
	(
		/* Fields already interned OOB by Sitecore.Interning.config */			
		'BADD9CF9-53E0-4D0C-BCC0-2D784C282F6A' /* updated by */,
		'5DD74568-4D4B-44C1-B513-0AF5F4CDA34F' /* created by */,
		'52807595-0F8F-4B20-8D2A-CB71D28C6103' /* owner */,
		'3E431DE1-525E-47A3-B6B0-1CCBEC3A8C98' /* workflow state */	
	)
GROUP BY 
	FieldId, [Name], CONVERT(NVARCHAR(250),[Value])
HAVING 
	COUNT(*) > 500 /* How many same field values must be met to be shown */)

SELECT * FROM DuplicatedFieldValues
ORDER BY [Hits] DESC

This can release a lot of memory, especially if you have large text fields.

USE Sitecore.Diagnostics.MemoryMonitorHook TO CLEAR CACHE AND FORCE GARBAGE COLLECTION

The Sitecore.Diagnostics.MemoryMonitorHook monitors your memory usage and can be set to force a garbage collection and a cache clear when the threshold is met:

<hooks>
    <hook type="Sitecore.Diagnostics.MemoryMonitorHook, Sitecore.Kernel">
        <param desc="Threshold">8192MB</param>
        <param desc="Check interval">00:15:00</param>
        <ClearCaches>true</ClearCaches>
        <GarbageCollect>true</GarbageCollect>
        <AdjustLoadFactor>false</AdjustLoadFactor>
    </hook>
</hooks>

• Threshold: How much memory should be used before doing anything. Set this to 60-80% of your total memory.
• Check interval: How often should the hook run.
• ClearCaches: When threshold is met, should the cache be cleared?
• GarbageCollect: When threshold is met, should Sitecore do a forced garbage collect?
• AdjustLoadFactor: When threshold is met, should Sitecore attempt to adjust the garbage collection load factor?

MORE TO READ:

• FRIDAY SITECORE BEST PRACTICE: SITECORE CACHE TUNING by Vasiliy Fomichev
• MEMORY OPTIMISATION IN SITECORE by Bartłomiej Mucha
• Does your application waste memory? by Nikolay Mitikov
• Basics of MemoryMonitorHook from Sitecore Basics
• Using threshold warning limits in Sitecore by Anders Laub
• The large object heap on Windows systems from Microsoft