Data Deduplication

Hi everyone,
I am having trouble to experience data deduplication in saving storage space.
I basically created a virtual server, Windows Server 2012 R2 and only has Data deduplication enabled on the server roles/features. An extra virtual SCSI disk 5GB and set it as K: volume, two .exe files in the volume taking rougly 1GB of space.
When I run the data deduplication task and the job processed in less than 10 seconds.
What have I done wrong?
Thanks guys

Hi,
How do you set the schedule? By default it will not optimize data created less than x days (7 days if I'm correct).
Also generally EXE files will not be dedupped - whether they are same exe file? If not, it is common that no file get dedupped as EXE file cannot be separated like TXT or DOC files.
Please remember to mark the replies as answers if they help and un-mark them if they provide no help. If you have feedback for TechNet Support, contact [email protected]

Similar Messages

Data Deduplication, Defrag & DPM 2012 R2 Churn

We have a protected 2012 R2 server that has Data Deduplication enabled. DPM tried to create a recovery point and ran out of disk space. When I check to see what happened, the job had transferred over 43GB of data. This is way outside of the norm
for this protection group. Upon investigation I found that a scheduled weekly defrag had run. I assume this to be the source of the churn. Can anyone provide an guidance on the interaction of the these processes? The scheduled defrag
was enabled by default. I assume we'll need to disable this? I have never defragged my VM's as it would cause massive churn for the VM backup's in DPM. But this was not a VM backup, just a backup of files on a deduplicated volume.
Thanks,
Scott

Thanks for the reply. It's pretty simple really. I have a 2012 R2 file server with data deduplication enabled on one of the volumes. There was a scheduled defrag that ran on the volume and appears to have caused massive churn for the
DPM recovery point. Question is if this is unique to data deduplication and DPM? The same problem occurs when you defrag a VM that's being backed up by DPM. That's why we don't defrag our VHDs. Normal file servers without data deduplication
do not exhibit this behavior. The scheduled defrag was set by default and I'm assuming needs to be disabled. I'm just looking for guidance on why this is an issue.

Deduplication error: there was a problem retrieving data deduplication schedule

Hi,
After changing dedup schedule once we are receiving this error and not able to change it any more.
there was a problem retrieving data deduplication schedule

Hi,
After changing dedup schedule once we are receiving this error and not able to change it any more.
there was a problem retrieving data deduplication schedule
Take a look @ PowerShell cmdlets for dedupe here:
PowerShell dedupe cmdlets
http://technet.microsoft.com/en-us/library/hh848450.aspx
Of your interest are Remove-DedupeSchedule and New-DedupSchedule ones. Try removing set schedule
you have now, making sure the status is "no jobs" and then create new schedule to see would it make the difference.
Good luck!
StarWind VSAN [Virtual SAN] clusters Hyper-V without SAS, Fibre Channel, SMB 3.0 or iSCSI, uses Ethernet to mirror internally mounted SATA disks between hosts.

Default Data Deduplication Optimisation job with "StopWhenSystemBusy" does not work correctly

I have configured Data Deduplication on Windows Server 2012 R2 and enabled on a volume.
Data Deduplication creates a default Optimisation task called "BackgroundOptimization".
I have placed files on my volume, and started the job manually. It succeeds but does nothing. InPolicyFiles still shows zero.
If I edit this job in Task Scheduler and remove the /backoff switch (StopWhenSystemBusy), then start the job manually again, it succeeds, and also incrememnts the InPolicyFiles.
At the same time, the CPU and RAM on the server is fairly idle, with no users on the system.
Is there a problem with the StopWhenSystemBusy? Should we not use this parameter, and simply run the task out of business hours instead? Has anyone seen this before?
This happens on two Windows Server 2012 R2 servers, so is reproducable here.
David

Hi David,
Please also try -Priority and -Memory with-StopWhenSystemBusy - set Priority to High and set a proper memory value such as 50% to see the result.
If you have any feedback on our support, please send to [email protected]

Widows Search not working with Data Deduplication?

Hi,
I noticed that many files were missing when searching for them on my Server 2012 Fileserver.
After some troubleshooting I noticed that the ones missing, had a Size on Disk of 4KB and the "SparseFile" and "ReparsePoint" (PL) Flags set.
So it looks like they were processed by the enabled Data Deduplication.
Am I missing something here or is it really the case that deduplicated files cannot be indexed by windows search?

BTW! "... and mounted those file hierarcies below. 5 reparse points" so I my case I had consolidated the users Space into 5 overall concepts on 1 volume each. In that way I had already reduced my use of volumes.
So for now I can add a drive letter to each for trying to make Windows Search work.
But guess I am going to stumble into the same issue then with Windows Search not supporting resparse points again ....because some of the volumes are deduped.
Great experience as ever working with file virtualization in the Windows client, i.e. Liberaries. Just wonder why the heck MS did not improove that experience ...
Right now, with Server 2012, find myself "hacking" again this year 2013 ... to try and make Windows work on some basic issues. All the Flex is starting to go away now ... here comes reality ... NOT WORKING.
NTFS resparse points, long file names, etc. Great Flex! However in theory only. For real when everything is pulled together and the system receives its more or less complete configuration ... unsupported in Windows. Go hack it.
Please stop doing just components test ... and do complete flight tests also. More hollistic tests please across constraints! With a focus in each test ... from 0 to a fully configured system.
How can no support in Windows Search for resparse points in NTFS be a miss during introduction of libraries (Windows 7/2008 R2) and now during introduction of data deduplication (Windows 8/2012) ... during just internal tests at MS? And we all have to use
libraries (thus a remote Windows Search index on remote shares in a domain setup) ... and now Windows 8 WinRT is even more dependent on libraries? How can that be overlooked - or such a basic thing receive no priority - and just get skipped? How can that happen?

Data deduplication scrubbing job stuck on 30% for 6 days

Hi everyone
I have a LUN which has been stuck on "scrubbing" for around 6 days. It won't progress any further. I've run a scan disk and it doesn't report any errors. The second LUN on the box is running fine as well.
The logs under application and service logs -> microsoft -> windows -> deduplication doesn't report any errors.
Any ideas? Is it safe to reboot the machine while it is scrubbing?

Hi,
Based on your description, we can try to use the Get-DedupStatus and
Get-DedupVolume Windows PowerShell cmdlets to check the status of the deduplication process.
Regarding this point, the following article can be referred to for more information.
Monitor and Report for Data Deduplication
http://technet.microsoft.com/en-us/library/hh831505.aspx
Best regards,
Frank Shen

Data deduplication works vice versa

Hello. I have enabled data deduplication on one of my SSD's volumes and then run Start-DedupJob cmdlet with optimizastion parameter on this volume. After dedup completed i check Get-Dedupstatus and get this output:
FreeSpace SavedSpace OptimizedFiles InPolicyFiles Volume
81.25 GB 9.44 GB 2 11 D:
Howevere when i have checked disk space, i discovered that free space was not increased, but instead decreased. (I had 88Gb of free space before deduplication, and after i had 81.25Gb of free space). Also I have checked the only folder in this volume and
found it has the same size as before dedup.
Also i tried Start-dedupJob with GarbageCollection type after all, and now i have that output:
FreeSpace SavedSpace OptimizedFiles InPolicyFiles Volume
81.26 GB 0 B 2 11 D:
Why data deduplication worked so strange for me?

Hi,
For the total used space when you right click on your volume and choose Properties, it is not accurate as for those folders your current account do not have permission to access, they will not be counted.
For your current situation, you can test to first disable deduplication, run a check disk in CMD for used disk space, and enable deduplication again, when it finished, check the chkdsk result in CMD again to see if same thing still happens.
If you have any feedback on our support, please send to [email protected]

Data Deduplication - max memory allocation

Hi
i encountered this issue on a server:
2012R2
https://support.microsoft.com/en-us/kb/2891882/
i have configured the value from the KB, but i still get:
Data Deduplication detected job type "Optimization" on volume "\\?\Volume{8ce5ac3c-46df-4a37-8a9e-c4b28f1354d1}\" uses too much memory. 3179 MB is assigned. 3870 MB is used.
Data Deduplication cancelled job type "Optimization" on volume "\\?\Volume{8ce5ac3c-46df-4a37-8a9e-c4b28f1354d1}\". It uses too much memory than the amount assigned to it.
for the volume with most data.
can the value be increased?

Hi,
Please try to attempt a workaround by adding the following registry key with value 1000 to resolve the issue.
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ddpsvc\Settings\WlmMemoryOverPercentThreshold
Best Regards,
Mandy
Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact [email protected]

Data Deduplication and File Backup

Hello all,
This is my first question here in the forum. I just migrated one of our server from Server 2008 R2 to Server 2012 Standard R2. I use Freefilesync to backup a couple of network storages that was attached to the server 2008 and now it is attached to the Server
2012. I would like to know if I apply data deduplication on the network storage that are now attached to the Server 2012 would that affect the files on my backups? How would that work? Thanks

Thank you again. One last question: So, the storage savings I will have with dedupe won't be carried out to my backups? Thanks?
Space would be preserved it's just your data MSFT would "unpack" in the middle of the migration process. So sequence of actions looks like this:
1) Data generation tools puts bunch of files on dedupe-enabled volume.
2) MSFT scrubber process kicks in and "packs" data from 1) to now smaller amount of used blocks.
3) Copy process takes data from 2) and "unpacks" it in memory and copies it to destination volume, UNPACKED.
4) MSFT scrubber process kicks in on a destination volume to do basically the same it did in 2) for source volume.
So smart implementations like one done by HP and their StoreOnce would skip steps 3) and 4) as copy process would move indexes and hash<->data pairs. MSFT cannot do this (at least for now).
For reference:
HP StoreOnce Global Deduplication
http://h18006.www1.hp.com/storage/pdfs/hpstoreonce.pdf
Global Data Dedupe HOWTO
http://searchdatabackup.techtarget.com/definition/global-data-deduplication
Hope this helped :)
P.S. I'm NOT working for HP or whatever :)
StarWind VSAN [Virtual SAN] clusters Hyper-V without SAS, Fibre Channel, SMB 3.0 or iSCSI, uses Ethernet to mirror internally mounted SATA disks between hosts.

Data DeDuplication & ReFS

I saw this question partially answered over a year ago but since it is decision time for me to either go
Refs or NTFS, I'm wondering if deduplication will eventually be supported on ReFS volumes. Does anyone know if there are plans to, defiantly not, impossible, or MS is undecided at this time. I REALLY like ReFS, but the lack of deduplication
is holding me back at NTFS.

A very good overview is here:
http://kx.cloudingenium.com/microsoft/servers/windows-servers/windows-server-2012/windows-server-2012-file-system-resiliency-refs-data-deduplication-ntfs/
Why choose ReFS over NTFS?
Integrity offered by automatic correction of data corruption.
Designed to stay ONLINE as long as possible – if data corruption occurs, only that sector is ‘corrected’ or taken offline. Typically with NTFS volumes, corruption means running a CHKDSK which
can take many hours to days even.
Salvage – “a feature that removes the corrupt
data from the namespace on a live volume”. What this means is that even if there is corruption on the volume which cannot be repaired, the file system will savage those sectors so that the volume still remains online.
ReFS works with Storage
Spaces to Better Detect and Repair Problems. The one particular feature that stands out is that if you have a mirrored storage pool if one of the drives becomes corrupted ReFS would find that information on another disk and restore the information
automatically. During the entire process you do not suffer any downtime.
ReFS is Much Better at Handling Power Outages: Instead of writing over the existing metadata when you perform a change, it writes it on an empty space and once the operation is completed the change
takes place preventing a sudden loss of power to result in half-written metadata which can happen in NTFS.
ReFS Supports Long File Names and File
Path. The limitation on full path size has also been updated from 255 characters for the total path size to 32K (32,768).
Why choose NTFS over ReFS?
Been around for almost 20 years (July 2013 will be its 20th anniversary).
If you need or use any one of the following as they are no longer available on the ReFS: named streams, object IDs, short names, compression, file
level encryption (EFS), user data transactions, sparse, hard-links, extended
attributes, and quotas.
Data deduplication.
Replication using DFS.
Some programs might use file locks or other features that might prevent proper functioning on ReFS. I already had to move folders back to NTFS because of that. I’ve read SQL
databases don’t work well with ReFS as well as Virtual Machines. So you probably want NTFS to run those two among other applications.

Data deduplicated volumes. What happens if they are moved to another server?

I have come across a potential issue with a server setup.
I have a server with it's OS disk in a Raid1. It's data deduplicated volumes are in a raid 10 via a separate raid controller. What happens if the OS disks fail? can the OS be re-installed without the data being lost on the deduplicated volumes?

I have come across a potential issue with a server setup.
I have a server with it's OS disk in a Raid1. It's data deduplicated volumes are in a raid 10 via a separate raid controller. What happens if the OS disks fail? can the OS be re-installed without the data being lost on the deduplicated volumes?
Yes, deduplication is completely portable between the systems. See:
Data Deduplication in Windows Server 2012
http://blogs.technet.com/b/filecab/archive/2012/05/21/introduction-to-data-deduplication-in-windows-server-2012.aspx
3) Portability: A
volume that is under deduplication control is an atomic unit. You can back up the volume and restore it to another server. You can rip it out of one Windows 2012 server and move it to another. Everything that is required to access your data is located on the
drive. All of the deduplication settings are maintained on the volume and will be picked up by the deduplication filter when the volume is mounted. The only thing that is not retained on the volume are the schedule settings that are part of the task-scheduler
engine. If you move the volume to a server that is not running the Data Deduplication feature, you will only be able to access the files that have not been deduplicated.
Hope this helped :)
StarWind VSAN [Virtual SAN] clusters Hyper-V without SAS, Fibre Channel, SMB 3.0 or iSCSI, uses Ethernet to mirror internally mounted SATA disks between hosts.

Dynamic Memory and Data Deduplication

Hi,
I'm currently testing W2K12 Deduplication feature on some servers, and I was wondering if I could enable "Dynamic Memory" on the VM in Hyper-V? Because the job only uses percentage of memory for the dedup job, I guess it's not possible.
Thanks in advance!

I know it old thread, but I ran into this and could not find anything on it but here, so I am adding my experience.
I can concur with 3magroup...
In my environment Windows Server 2012 with Dynamic Memory and Deduplication enabled, the HyperV does not release the driver locked ram for the dedup optimization. What resulted was an insane amount of paging and a painfully slow VM. Increasing the memory
buffer for the dynamic memory did not cause the HyperV to release the memory for the dedup. Instead, it still "driver locked" nearly all of the ram, which did not make sense to me based on my understanding of the Memory Buffer setting.
I can verify the VM poor performance and dedup using the paging file with the Resource Monitor, disk section, and the Driver locked RAM with RamMap.

[Forum FAQ] Date Deduplication frequently asked questions and solutions

Q1: Sometimes we deleted a folder or a file on a deduped volume. The Free space is not released. How to do a refresh to show the correct result?
A1: The following cmdlet could help to perform a refresh (Figure 1):
Update-DedupStatus
Figure 1: Force a refresh
Q2: How to remove a file on a volume from Deduplication list after the whole volume is already deduped?
A2: Run the following cmdlet:
Expand-DedupFile –path: “X:\folder\file”.
Note: It does not support wildcards in the -Path parameter (Figure 2).
Figure 2: Error when use wildcards
We can do standard PowerShell pipelining to automate for multiple files such as:
dir X:\folder | expand-dedupfile
Here is an example:
Figure 3: Example
Please click to vote if the post helps you. This can be beneficial to other community members reading the thread.

4. How to clear the SBOMailer queue?
As a workaround, please use dummy or wrong SMTP server and send all those mails to that address.
After sending those mail to dummy SMTP, you can see the mail status changed to "E" (Error), then after setting up the correct smtp mailer
would not try to send the mails with status "E".
5. How to use dialer in BP?
1.How does this work?
If you have Microsoft#s automatic phone dialer installed, you can press CTRL+TAB to automatically dial the numbers in the telephone fields.
2. Are there settings I can change? In the System Initialization - General Settings - Services, to place calls directly through SAP Business One if a phone system or modem is installed on your computer or in your network, in these fields specify the area code and the number to dial for an outside line.
3. How can I troubleshoot when this doesn't work? Try to check if the number is correct, the best way to check it is to use the MS Phone Dialer and to enter this number exactly as it is written in B1. If the dialer succeed dialing this number so B1 should succeed too, if the dialer failed check what is the problem with the number or with the Microsoft Phone Dialer.
6. Iu2019m unable to set exchange rate in credit note
To set an exchange rate in a Credit Note is only possible to if the credit note is not based on another document. In this case the exchange rate of the base document will be considered. Please see also SAP Notes [686224|http://service.sap.com/sap/bc/bsp/spn/smb_searchnotes/display2.htm?note_langu=E&note_numm=686224] und [703028|http://service.sap.com/sap/bc/bsp/spn/smb_searchnotes/display2.htm?note_langu=E&note_numm=703028]

Hyper-V Server Manager -Error occurred during enumeration of Data Deduplication volumes

Hi Experts,
I having this problem in my Hyper-V Server Manager. Can somebody tell me why this happening? Inside my Hyper-V, I has been installed Microsoft DPM 2012 and both Hyper-V is cluster. Please advice.

Hi Tyler,
I had the same problem until I unlocked (BitLocker) the HDD causing the error message.

Data Protection Manager 2012 - Inconsistent when backing up Deduplicated File Server

Protected Server
Server 2012 File Server with Deduplication running on Data drive
DPM Server
Server 2012
Data Protection Manager 2012 Service Pack 1
We just recently upgraded our DPM server from DPM 2010 to DPM 2012 primarily because it is supposed to support Data Deduplication. Our primary File server that holds our home directories etc. is limited on space and was quickly running low so just after
we got DPM 2012 in place we optimized the drive on the file server which compressed the data about 50%. Unfortunately shortly after enabling deduplication the protected shares on the deduplicated volume are getting a Replica is Inconsistent error.
I continually get Replica is Inconsistent for the Server that has deduplication running on it. All of the other protected servers are being protected as they should be. I have run a consistency check multiple times probably about 10 times and it keeps going
back to Replica is inconsistent. The replica volume shows that it is using 3.5 TB and the Actual protect volume is 4TB in size and has about 2.5 TB of data on it with Deduplication enabled.
This is the details of the error
Affected area:   G:\
Occurred since: 1/12/2015 4:55:14 PM
Description:        The replica of Volume G:\ on E****.net is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with
consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent.
For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)
More information
Recommended action:
Synchronize with consistency check.
Run a synchronization job with consistency check...
Resolution:
To dismiss the alert, click below
Inactivate
Steps taken to resolve - I’ve spent some time doing some searches and haven’t found any solutions to what I am seeing. I have the data deduplication role installed on the DPM server which has been the solution for many people seeing similar issues. I have
also removed that role and the added it back. I have also removed the protected server and added it back to the protection group. It synchronizes and says consistent then after a few hours it goes back to inconsistent. When I go to recovery it shows that I
have recovery points and it appears that I can restore but because the data is inconstant I don’t feel I can trust the data in the recovery points. Both the protected server and the DPM servers’ updates are managed via a WSUS server on our network.
You may suggest I just un-optimize the drive on the protected server however after I have optimized the drive it takes a large amount more of space to un-optimize it (Anyone know why that is) anyways the drive isn’t large enough to support un-optimization.
If anyone has any suggestions I would appreciate any help. Thanks in advanced.

Ok I ran a consistency check and it completed successfully with the following message. However after a few minutes of it showing OK it now shows Replica is Inconsistent again.
Type: Consistency check
Status: Completed
Description: The job completed successfully with the following warning:
An unexpected error occurred while the job was running. (ID 104 Details: Cannot create a file when that file already exists (0x800700B7))
More information
End time: 2/3/2015 11:19:38 AM
Start time: 2/3/2015 10:34:35 AM
Time elapsed: 00:45:02
Data transferred: 220.74 MB
Cluster node -
Source details: G:\
Protection group members: 35
Details
Protection group: E*
Items scanned: 2017709
Items fixed: 653
There was a log for a failed synchronization job from yesterday here are the details of that.
Type: Synchronization
Status: Failed
Description: The replica of Volume G:\ on E*.net is not consistent with the protected data source. (ID 91)
More information
End time: 2/2/2015 10:04:01 PM
Start time: 2/2/2015 10:04:01 PM
Time elapsed: 00:00:00
Data transferred: 0 MB
Cluster node -
Source details: G:\
Protection group members: 35
Details
Protection group: E*

Data Deduplication

Similar Messages

Maybe you are looking for