Windows 2012 R2 Deduplication Garbarge Collection eats up space

Question

I have enabled Deduplication on a volume and it's running for a couple of weeks now. Backups are stored on this volume and dedup is doing a good job on this data (Weekly full backups + incrementals).
But when the weekly garbage collection runs, it claims all free space and the job get cancelled when the disk is full (or memory is low). It's going down from 600GB to 50GB. The following event is logged:
Event ID:      4143
Description:
Data Deduplication cancelled job type "Garbage Collection" on volume "\\?\Volume{55bdbbaf-af4f-431f-878e-c6ead208b86c}\". Memory resource is running low on the machine or in the job.
If i run a Full Optimization afterwards, the space is freed again.
Any idea what is happening? Is garbage collection trying to unoptimize some files or something?
Currently running Full Optimization, but here's the volume info:Volume : E:
VolumeId : \\?\Volume{55bdbbaf-af4f-431f-878e-c6ead208b86c}\
Enabled : True
UsageType : HyperV
DataAccessEnabled : True
Capacity : 2.7 TB
FreeSpace : 395.41 GB
UsedSpace : 2.31 TB
UnoptimizedSize : 11.85 TB
SavedSpace : 9.54 TB
SavingsRate : 80 %
MinimumFileAgeDays : 0
MinimumFileSize : 32768
NoCompress : False
ExcludeFolder :
ExcludeFileType :
ExcludeFileTypeDefault : {bin, vsv, slp, xml...}
NoCompressionFileType : {asf, mov, wma, wmv...}
ChunkRedundancyThreshold : 100
Verify : False
OptimizeInUseFiles : False
OptimizePartialFiles : True

Answer 1

Guess it's trying to unseal and move some content away from previously sealed dedupe containers stored in-between NTFS metadata objects. And runs out of memory eventually. Throw in some more spindles, grow volume and see would it make any difference.
StarWind Virtual SAN clusters Hyper-V without SAS, Fibre Channel, SMB 3.0 or iSCSI, uses Ethernet to mirror internally mounted SATA disks between hosts.