Strange memory behaviour using the System.Collections.Hashtable in object reference

Question

Dear all,
Recently I came across a strange memory behaviour when comparing the system.collections.hashtable versus de scripting.dictionary object and thought to analyse it a bit in depth. First I thought I incorrectly destroyed references to the class and
child classes but even when properly destroying them (and even implemented a "SafeObject" with deallocate method) I kept seeing strange memory behaviour.
Hope this will help others when facing strange memory usage BUT I dont have a solution to the problem (yet) suggestions are welcome.
Setting:
I have a parent class that stores data in child classes through the use of a dictionary object. I want to store many differenct items in memory and fetching or alteging them must be as efficient as possible using the dictionary ability of retrieving key
/ value pairs. When the child class (which I store in the dictionary as value) contains another dictionary object memory handeling is as expected where all used memory is release upon the objects leaving scope (or destroyed via code). When I use a system.collection.hashtable
no memory is released at all though apears to have some internal flag that marks it as useable for another system.collection.hashtable object.
I created a small test snippet of code to test this behaviour with (running excel from the Office Plus 2010 version) The snippet contains a module to instantiate the parent class and child classes that will contain the data. One sub will test the Hash functionality
and the other sub will test the dictionary functionality.
Module1
Option Explicit
Sub testHash()
Dim Parent As parent_class
Dim d_Count As Double
'Instantiate parent class
Set Parent = New parent_class
'Create a child using the hash collection object
Parent.AddChildHash "TEST_hash"
Dim d_CycleCount As Double
d_CycleCount = 50000
'Add dummy data records to the child container with x amount of data For d_Count = 0 To d_CycleCount
Parent.ChildContainer("TEST_hash").InsertDataToHash CStr(d_Count), "dummy data"
Next
'Killing the parent when it goes out of scope should kill the childs. (Try it out and watch for the termination debug messages)
'According to documentation and debug messages not really required!
Set Parent.ChildContainer("TEST_hash") = Nothing
'According to documentation not really as we are leaving scope but just to be consistent.. kill the parent!
Set Parent = Nothing
End Sub
Sub testDict()
Dim Parent As parent_class
Dim d_Count As Double
'Instantiate parent class
Set Parent = New parent_class
'Create a child using the dictionary object
Parent.AddChildDict "TEST_dict"
Dim d_CycleCount As Double
d_CycleCount = 50000
'Blow up the memory with x amount of records
Dim s_SheetCycleCount As String
s_SheetCycleCount = ThisWorkbook.Worksheets("ButtonSheet").Range("K2").Value
If IsNumeric(s_SheetCycleCount) Then d_CycleCount = CDbl(s_SheetCycleCount)
'Add dummy data records to the child container
For d_Count = 0 To d_CycleCount
Parent.ChildContainer("TEST_dict").InsertDataToDict CStr(d_Count), "dummy data"
Next
'Killing the parent when it goes out of scope should kill the childs. (Try it out and watch for the termination debug messages)
'According to documentation and debug messages not really required!
Set Parent.ChildContainer("TEST_dict") = Nothing
'According to documentation not really as we are leaving scope but just to be consistent.. kill the parent!
Set Parent = Nothing
End Sub
parent_class:
Option Explicit
Public ChildContainer As Object
Private Counter As Double
Private Sub Class_Initialize()
Debug.Print "Parent initialized"
Set ChildContainer = CreateObject("Scripting.dictionary")
End Sub
Public Sub AddChildHash(ByRef ChildKey As String)
If Not ChildContainer.Exists(ChildKey) Then
Dim TmpChild As child_class_hashtable
Set TmpChild = New child_class_hashtable
ChildContainer.Add ChildKey, TmpChild
Counter = Counter + 1
Set TmpChild = Nothing
End If
End Sub
Public Sub AddChildDict(ByRef ChildKey As String)
If Not ChildContainer.Exists(ChildKey) Then
Dim TmpChild As child_class_dict
Set TmpChild = New child_class_dict
ChildContainer.Add ChildKey, TmpChild
Counter = Counter + 1
Set TmpChild = Nothing
End If
End Sub
Private Sub Class_Terminate()
Debug.Print "Parent being killed, first kill all childs (if there are any left!) - muahaha"
Set ChildContainer = Nothing
Debug.Print "Parent killed"
End Sub
child_class_dict
Option Explicit
Public MemmoryLeakObject As Object
Private Sub Class_Initialize()
Debug.Print "Child using Scripting.Dictionary initialized"
Set MemmoryLeakObject = CreateObject("Scripting.Dictionary")
End Sub
Public Sub InsertDataToDict(ByRef KeyValue As String, ByRef DataValue As String)
If Not MemmoryLeakObject.Exists(KeyValue) Then MemmoryLeakObject.Add KeyValue, DataValue
End Sub
Private Sub Class_Terminate()
Debug.Print "Child using Scripting.Dictionary terminated"
Set MemmoryLeakObject = Nothing
End Sub
child_class_hash:
Option Explicit
Public MemmoryLeakObject As Object
Private Sub Class_Initialize()
Debug.Print "Child using System.Collections.Hashtable initialized"
Set MemmoryLeakObject = CreateObject("System.Collections.Hashtable")
End Sub
Public Sub InsertDataToHash(ByRef KeyValue As String, ByRef DataValue As String)
If Not MemmoryLeakObject.ContainsKey(KeyValue) Then MemmoryLeakObject.Add KeyValue, DataValue
End Sub
Private Sub Class_Terminate()
Debug.Print "Child using System.Collections.Hashtable terminated"
Set MemmoryLeakObject = Nothing
End Sub
Statistics:
TEST: (Chronologically ordered)
1.1 Excel starting memory: 25.324 kb approximately
Max memory usage after hash (500.000 records) 84.352 kb approximately
Memory released: 0 %
1.2 max memory usages after 2nd consequtive hash usage 81.616 kb approximately
"observation:
- memory is released then reused
- less max memory consumed"
1.3 max memory usages after 3rd consequtive hash usage 80.000 kb approximately
"observation:
- memory is released then reused
- less max memory consumed"
1.4 Running the dictionary procedure after any of the hash runs will start from the already allocated memory
In this case from 80000 kb to 147000 kb
Close excel, free up memory and restart excel
2.1 Excel starting memory: 25.324 kb approximately
Max memory usage after dict (500.000 records) 90.000 kb approximately
Memory released: 91,9%
2.2 Excel starting memory 2nd consequtive dict run: 27.552 kb approximately
Max memory usage after dict (500.000 records) 90.000 kb approximately
Memory released: 99,4%
2.3 Excel starting memory 3rd consequtive dict run: 27.712 kb approximately
Max memory usage after dict (500.000 records) 90.000 kb approximately
Memory released:

Answer 1

Hi Cor,
Thank you for going through my post and took the time to reply :) Most apreciated. The issue I am facing is that the memory is not reallocated when using mixed object types and is not behaving the same. I not understand that .net versus the older methods
use memory allocation differently and perhaps using the .net dictionary object (in stead of the scripting.dictionary) may lead to similar behaviour. {Curious to find that out -> put to the to do list of interesting thingies to explore}
I agree that allocated memory is not a bad thing as the blocks are already assigned to the program and therefore should be reallocated with more performance. However the dictionary object versus hashtable perform almost identical (and sometimes even favor
the dictionary object)
The hashtable is clearly the winner when dealing with small sets of data.
The issue arises when I am using the hash table in conjunction with another type, for example a dictionary, I see that the dictionary object's memory is stacked on top of the claimed memory space of the hashtable. It appears that .net memory allocation and
reuse is for .net references only. :) (Or so it seems)
In another example I got with the similar setup, I see that the total used memory is never released or reclaimed and leakage of around 20% allocated memory remains to eventually crash the system with the out of memory message. (This is when another class
calls the parent class that instantiates the child class but thats not the point of the question at hand)
This leakage drove me to investigate and create the example of this post in the first place. For the solution with the class -> parent class -> child class memory leak I switched all to dictionaries and no leakage occurs anymore but nevertheless thought
it may be good to share / ask if anyone else knows more :D (Never to old to learn something new)