W2K12-R deduplication optimization job stuck at 100% for days

Question

Hello,
I've W2K12-R server (VM, 4 vCPU, 14 GB vRAM) with a 7 TB D: drive, deduplication is enabled for this drive. I use this drive for backing up virtual machines, at the moment 3,6 TB are used. Each night ~2 TB new data is backed up, only the C: drives of VMs,
so there is a very high dedup rate, these new 2 TB are usually deduplicated during daytimes.
Capacity                 : 6.64 TB
FreeSpace               : 3.01 TB
UsedSpace              : 3.63 TB
UnoptimizedSize        : 110.74 TB
SavedSpace             : 107.1 TB
SavingsRate             : 96 %
Since a few weeks I've the problem that ThroughputOptimization jobs (all Optimization jobs) do not finish. The job is able to free space but it stays at 100% for days.
Although I've set the DurationHours to 6 or 9 hours, the scheduled job does not end and thus no new Optimization job starts. That leads to a completely filled D drive because the space that gets filled by backup jobs during the night  is not freed during
the planned time frame.
Event Log from the start of the job:
+
System
Provider
[ Name]
Microsoft-Windows-Deduplication
[ Guid]
{F9FE3908-44B8-48D9-9A32-5A763FF5ED79}
EventID
6148
Version
0
Level
4
Task
0
Opcode
0
Keywords
0x8000000000000000
TimeCreated
[ SystemTime]
2014-07-21T14:20:04.531120000Z
EventRecordID
2768
Correlation
Execution
[ ProcessID]
4672
[ ThreadID]
268
Channel
Microsoft-Windows-Deduplication/Operational
Computer
xxxxxx.net
Security
[ UserID]
S-1-5-18
EventData
JobType
1
JobInstanceId
{39D260C4-32D0-4511-8982-598DA53DB423}
VolumeGuidPath
\\?\Volume{27fe6999-df3b-4864-81fe-853126e2c9cc}\
VolumeDisplayName
D:
AvailableMemoryMb
11058
JobPriorityType
3
I don't see further information about job 39D260C4-32D0-4511-8982-598DA53DB423 in event log. In resource monitor I see that fsdmhost has very little disk activity (600 Bytes/s), but a cpu load of 25% (which is one of the four vCPUs).
I stopped the backups this week and started a manual full Optimization job, which stays at 100% since more than one day now.
Any ideas what to check? Any patch I can apply?
Some details:
23.07.014 - 8:40 Uhr
Get-DedupStatus
FreeSpace    SavedSpace   OptimizedFiles     InPolicyFiles      Volume
3.01 TB      107.1 TB     12232              12232              D:
Get-DedupStatus | fl
Volume                             : D:
VolumeId                           : \\?\Volume{27fe6999-df3b-4864-81fe-853126e2c9cc}\
Capacity                           : 6.64 TB
FreeSpace                          : 3.01 TB
UsedSpace                          : 3.63 TB
UnoptimizedSize                    : 110.74 TB
SavedSpace                         : 107.1 TB
SavingsRate                        : 96 %
OptimizedFilesCount                : 12232
OptimizedFilesSize                 : 110.58 TB
OptimizedFilesSavingsRate          : 96 %
InPolicyFilesCount                 : 12232
InPolicyFilesSize                  : 110.58 TB
LastOptimizationTime               : 23.07.2014 04:44:59
LastOptimizationResult             : 0x8056533D
LastOptimizationResultMessage      : The operation was cancelled.
LastGarbageCollectionTime          : 20.07.2014 05:26:12
LastGarbageCollectionResult        : 0x00000000
LastGarbageCollectionResultMessage : The operation completed successfully.
LastScrubbingTime                  : 20.07.2014 16:06:29
LastScrubbingResult                : 0x00000000
LastScrubbingResultMessage         : The operation completed successfully.
Get-DedupVolume
Enabled            UsageType          SavedSpace           SavingsRate         
Volume
True               Default            107.1 TB             96 %                
D:
Get-DedupVolume |fl
Volume                   : D:
VolumeId                 : \\?\Volume{27fe6999-df3b-4864-81fe-853126e2c9cc}\
Enabled                  : True
UsageType                : Default
DataAccessEnabled        : True
Capacity                 : 6.64 TB
FreeSpace                : 3.01 TB
UsedSpace                : 3.63 TB
UnoptimizedSize          : 110.74 TB
SavedSpace               : 107.1 TB
SavingsRate              : 96 %
MinimumFileAgeDays       : 0
MinimumFileSize          : 32768
NoCompress               : False
ExcludeFolder            :
ExcludeFileType          :
ExcludeFileTypeDefault   : {edb, jrs}
NoCompressionFileType    : {asf, mov, wma, wmv...}
ChunkRedundancyThreshold : 100
Verify                   : False
OptimizeInUseFiles       : False
OptimizePartialFiles     : False
Get-DedupMetadata | fl
Volume                         : D:
VolumeId                       : \\?\Volume{27fe6999-df3b-4864-81fe-853126e2c9cc}\
StoreId                        : {1E87DD60-0265-4726-9EDD-81FCE4E9D70A}
DataChunkCount                 : 151954987
DataContainerCount             : 3983
DataChunkAverageSize           : 24.35 KB
DataChunkMedianSize            : 0 B
DataStoreUncompactedFreespace  : 0 B
StreamMapChunkCount            : 95398
StreamMapContainerCount        : 1944
StreamMapAverageDataChunkCount :
StreamMapMedianDataChunkCount  :
StreamMapMaxDataChunkCount     :
HotspotChunkCount              : 1048520
HotspotContainerCount          : 54
HotspotMedianReferenceCount    :
CorruptionLogEntryCount        : 0
TotalChunkStoreSize            : 3.6 TB
Get-DedupSchedule
Enabled    Type               StartTime              Days              
Name
True       Optimization                                                
BackgroundOptimization
False      GarbageCollection  07:30                  {Monday, Tuesda... DailyGarbageCollection
True       Optimization       09:00                  {Sunday, Monday... ThroughputOptimization
False      Optimization       14:30                  {Sunday, Monday... ThroughputOptimization-2
True       GarbageCollection  16:00                  Saturday           WeeklyGarbageCollection
True       Scrubbing          03:45                  Sunday            
WeeklyScrubbing
Get-DedupSchedule | fl
Name                     : BackgroundOptimization
Enabled                  : True
Type                     : Optimization
Days                     :
Start                    :
DurationHours            :
StopWhenSystemBusy       : True
Memory                   : 25 %
Priority                 : Low
InputOutputThrottleLevel : Low
ScheduledTask            : \Microsoft\Windows\Deduplication\BackgroundOptimization
Full                     :
ReadOnly                 :
Name                     : DailyGarbageCollection
Enabled                  : False
Type                     : GarbageCollection
Days                     : {Monday, Tuesday, Wednesday, Thursday...}
Start                    : 17.06.2014 07:30:00
DurationHours            : 0
StopWhenSystemBusy       : False
Memory                   : 50 %
Priority                 : Normal
InputOutputThrottleLevel : None
ScheduledTask            : \Microsoft\Windows\Deduplication\DailyGarbageCollection
Full                     : False
ReadOnly                 : False
Name                     : ThroughputOptimization
Enabled                  : True
Type                     : Optimization
Days                     : {Sunday, Monday, Tuesday, Wednesday...}
Start                    : 14.07.2014 09:00:00
DurationHours            : 6
StopWhenSystemBusy       : False
Memory                   : 40 %
Priority                 : Normal
InputOutputThrottleLevel : None
ScheduledTask            : \Microsoft\Windows\Deduplication\ThroughputOptimization
Full                     : False
ReadOnly                 : False
Name                     : ThroughputOptimization-2
Enabled                  : False
Type                     : Optimization
Days                     : {Sunday, Monday, Tuesday, Wednesday...}
Start                    : 14.07.2014 14:30:00
DurationHours            : 9
StopWhenSystemBusy       : False
Memory                   : 40 %
Priority                 : Normal
InputOutputThrottleLevel : None
ScheduledTask            : \Microsoft\Windows\Deduplication\ThroughputOptimization-2
Full                     : False
ReadOnly                 : False
Name                     : WeeklyGarbageCollection
Enabled                  : True
Type                     : GarbageCollection
Days                     : Saturday
Start                    : 20.06.2014 16:00:00
DurationHours            : 26
StopWhenSystemBusy       : True
Memory                   : 50 %
Priority                 : High
InputOutputThrottleLevel : None
ScheduledTask            : \Microsoft\Windows\Deduplication\WeeklyGarbageCollection
Full                     : True
ReadOnly                 : False
Name                     : WeeklyScrubbing
Enabled                  : True
Type                     : Scrubbing
Days                     : Sunday
Start                    : 17.05.2014 03:45:00
DurationHours            : 0
StopWhenSystemBusy       : True
Memory                   : 50 %
Priority                 : Normal
InputOutputThrottleLevel : None
ScheduledTask            : \Microsoft\Windows\Deduplication\WeeklyScrubbing
Full                     : True
ReadOnly                 : False
Get-DedupJob
Type               ScheduleType       StartTime              Progress   State                 
Volume
Optimization       Scheduled                                 0 %       
Queued                 D:
Optimization       Manual             16:20                  100 %     
Running                D:
Get-DedupJob |fl
Volume                   : D:
VolumeId                 : \\?\Volume{27fe6999-df3b-4864-81fe-853126e2c9cc}\
Type                     : Optimization
ScheduleType             : Scheduled
StartTime                :
Progress                 : 0 %
State                    : Queued
Id                       : {556DC9B2-068C-4CAE-9BE7-4104A0963F49}
StopWhenSystemBusy       : True
Memory                   : 25 %
Priority                 : Low
InputOutputThrottleLevel : Low
ProcessId                : 0
Full                     : False
ReadOnly                 : False
Volume                   : D:
VolumeId                 : \\?\Volume{27fe6999-df3b-4864-81fe-853126e2c9cc}\
Type                     : Optimization
ScheduleType             : Manual
StartTime                : 21.07.2014 16:20:04
Progress                 : 100 %
State                    : Running
Id                       : {39D260C4-32D0-4511-8982-598DA53DB423}
StopWhenSystemBusy       : False
Memory                   : 60 %
Priority                 : High
InputOutputThrottleLevel : None
ProcessId                : 4672
Full                     : True
ReadOnly                 : False

Answer 1

I stopped the manually started optimization job now and see these event log entries.
Data Deduplication job type "Optimization" on volume "D:" was cancelled manually.
Data Deduplication job of type "Optimization" on volume "D:" has completed with return code: 0x8056533d, The operation was cancelled.
Optimization reconciliation has completed.
Volume: D: (\\?\Volume{27fe6999-df3b-4864-81fe-853126e2c9cc}\)
Reconciled containers: 1908
Unreconciled containers: 36
Merged containers: 0
Total reconciled references: 0
Error code: 0x0
Error message: NULL
Optimization job has completed.
Volume: D: (\\?\Volume{27fe6999-df3b-4864-81fe-853126e2c9cc}\)
Error code: 0x0
Error message:
Savings rate: 96
Saved space: 117761059523158
Volume used space: 3988537147392
Volume free space: 3312769826816
Optimized file count: 12232
In-policy file count: 12232
Job processed space (bytes): 1815468262325
Job elapsed time (seconds): 150522
Job throughput (MB/second): 11.5