2008 R2: OS(systemState/C: Drive) backup issues with VSS - 0x80042316 & 0x81080101 & 0x80080005 & 2155348001

Question

Hi-
Looked thru the thread related to my questions and found nothing that applies yet.
Example:
 http://social.technet.microsoft.com/Forums/en-US/4e4472b9-8e6f-4fdc-a287-1cc8c0f6de09/windows-2008-r2-backup-fails-with-event-521-error-code-2155348001?forum=windowsbackup
http://social.technet.microsoft.com/Forums/en-US/e8c910f7-f0dd-4613-beae-b98894fb6f23/windows-server-2008-std-backup-fails-constantly-with-error-code-2155348001?forum=windowsbackup
Here are details of the issues....
Backup - OS error - system image:
External disk - USB
Mon - Fri @ 6PM.
Windows SystemState
OS(C:)
Last backup failure...
Start time: 8/11/2014 6:01:32 PM
End time: 8/12/2014 4:51:29 AM
Duration: 10:49:56.4414784
[Critical error WINIMAGE_UNABLE_VSS] Unable to create VSS snapshot
  - Additional information: Refer to the log for more details
  - Troubleshooting link:
http://kb.backupassist.com/errormap.php?BA2512
The backup operation stopped before completing.
Detailed error: ERROR - The shared restore point operation failed with error (0x81000101)
The creation of a shadow copy has timed out. Try this operation again.
Log of files successfully backed up:
C:\Windows\Logs\WindowsServerBackup\Backup-12-08-2014_04-41-00.log
Log of files for which backup failed:
C:\Windows\Logs\WindowsServerBackup\Backup_Error-12-08-2014_04-41-00.log
NOTE:
The errors logs are not there?
Windows Backup timed-out before the shared protection point was created.
ERROR - The shared restore point operation failed with error (0x81000101)
The creation of a shadow copy has timed out. Try this operation again.
Event Viewer Log:
Sequence of Events...
1.
The Block Level Backup Engine service has successfully started.
2.
Backup operation that started...
Log Name:      Application
Source:        Microsoft-Windows-Backup
Date:          8/12/2014 4:51:06 AM
Event ID:      521
Task Category: None
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      mdnfile.Auxiant.local
Description:
The backup operation that started at '?2014?-?08?-?12T09:41:00.492000000Z' has failed because the
Volume Shadow Copy Service operation to create a shadow copy of the volumes being backed up failed
with following error code '2155348001'.
Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.
--> Suggestion are to check Virus scanner or consult DPM forum...
3.
OS Job failed with errors...( Backup Assist is 3rd party backup - wrapper round Wbadmin/Win Backup)
Log Name:      Application
Source:        BackupAssist
Date:          8/12/2014 4:51:51 AM
Event ID:      5634
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      mdnfile.Auxiant.local
Description:
Backup job OS backup failed with errors.
Critical error: Unable to create VSS snapshot
Destination: External disk
Start time: 8/11/2014 6:01:32 PM
End time: 8/12/2014 4:51:29 AM
Duration: 10:49:56.4414784
Destination: External disk
Start time: 8/11/2014 6:01:32 PM
End time: 8/12/2014 4:51:29 AM
Duration: 10:49:56.4414784
4.
Backup Assist reports via Win backup - Snapshot in progress
Log Name:      Application
Source:        BackupAssist
Date:          8/12/2014 4:52:20 AM
Event ID:      5634
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      mdnfile.Auxiant.local
Description:
Backup job Data backup failed with errors.
Ultra critical error: Volume Shadow Copy Error 0x80042316 - snapshot set in progress
Destination: Local directory
Start time: 8/12/2014 4:52:10 AM
End time: 8/12/2014 4:52:12 AM
Duration: 00:00:02.7300000
5.
The Block Level Backup Engine service has stopped.
6.
Later...we tried manually stopping VSS and SWPRV services - only VSS had the issue.
WBengine was not running...
C:\>sc stop swprv
SERVICE_NAME: swprv
        TYPE               : 10  WIN32_OWN_PROCESS
        STATE              : 3  STOP_PENDING
                                (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x0
C:\>sc query swprv
SERVICE_NAME: swprv
        TYPE               : 10  WIN32_OWN_PROCESS
        STATE              : 1  STOPPED
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x0
C:\>sc query wbengine
SERVICE_NAME: wbengine
        TYPE               : 10  WIN32_OWN_PROCESS
        STATE              : 1  STOPPED
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x0
C:\>sc query vss
SERVICE_NAME: vss
        TYPE               : 10  WIN32_OWN_PROCESS
        STATE              : 3  STOP_PENDING
                                (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x0
C:\>sc stop vss
[SC] ControlService FAILED 1061:
The service cannot accept control messages at this time.  
Log Name:      Application
Source:        VSS
Date:          8/12/2014 9:20:33 AM
Event ID:      8225
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      mdnfile.Auxiant.local
Description:
The VSS service is shutting down due to shutdown event from the Service Control Manager.
7.
VSS still having issues here...not stopping!
Log Name:      Application
Source:        VSS
Date:          8/12/2014 9:42:33 AM
Event ID:      11
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      mdnfile.Auxiant.local
Description:
Volume Shadow Copy Service information:
The COM Server with CLSID {e579ab5f-1cc4-44b4-bed9-de0991ff0623} and name IVssCoordinatorEx2 cannot be started.
Most likely the CPU is under heavy load. [0x80080005, Server execution failed]
8.
Not sure why this is started again?
Plus a DCOM timeout...
Log Name:      Application
Source:        Microsoft-Windows-Backup
Date:          8/12/2014 9:31:13 AM
Event ID:      753
Task Category: None
Level:         Information
Keywords:     
User:          SYSTEM
Computer:      mdnfile.Auxiant.local
Description:
The Block Level Backup Engine service has successfully started.
Event Xml:
Log Name:      System
Source:        Microsoft-Windows-DistributedCOM
Date:          8/12/2014 9:31:43 AM
Event ID:      10010
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      mdnfile.Auxiant.local
Description:
The server {E579AB5F-1CC4-44B4-BED9-DE0991FF0623} did not register with DCOM within the required timeout.
9.
Lots more errors about VSS...repeating Event ID's 11 and 8193...
Log Name:      Application
Source:        VSS
Date:          8/12/2014 9:42:33 AM
Event ID:      8193
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      mdnfile.Auxiant.local
Description:
Volume Shadow Copy Service error: Unexpected error calling routine CoCreateInstance. 
hr = 0x80080005, Server execution failed
. Operation:
   Instantiating VSS server
10.
Then, finally stopping again...
Log Name:      Application
Source:        Microsoft-Windows-Backup
Date:          8/12/2014 9:51:13 AM
Event ID:      754
Task Category: None
Level:         Information
Keywords:     
User:          SYSTEM
Computer:      mdnfile.Auxiant.local
Description:
The Block Level Backup Engine service has stopped.
11.
And still repeating 11 & 8193 at this point...as we write up this post.
12.
This was suggested by the Backup Assist - software vendor....
A.)
...for 0x81080101:
"ERROR - The shared restore point operation failed with error (0x81000101)
The creation of a shadow copy has timed out. Try again this operation."
If you receive the above error in your report, we have recently found a registry key which
contains the timeout value set by Microsoft for VSS and found that you can edit this to extend it.
To get to the registry key, open Regedit (Start > Run > Regedit) and browse to:
"HKLM\Software\\Microsoft\\Windows NT\\CurrentVersion\\SPP\\CreateTimeout".
Change value to 1200000 (2*10*60*1000 = 20 mins).
The default for the timeout period is 10 mins so hopefully doubling this will give enough time to
complete the snapshot.
If this isn't present, you'll need to manually enter the registry key. It's a DWORD 32-bit key.
--> NOTE: HAve not done this yet.
B.)
And for 0x80080005...
http://support.microsoft.com/kb/870655/en-gb
Our server does have high CPU load at times...
13.
Another post from technet mentions system volume size - here is our layout below:
A.)
OS/system Disk...
DISKPART> list disk
  Disk ###  Status         Size     Free     Dyn  Gpt
  Disk 0    Online          278 GB      0 B    (System Vol = C: Drive - OS w/ Programs and batch jobs)
  Disk 1    Online         2791 GB      0 B        *    (Data drive - DFS, rsync,etc. run on D: Drive)
  Disk 2    Online         1863 GB      0 B    (USB 2/3.0 - Enclosure w/ SATA disk = 2GB formatted w/ NTFS.)
DISKPART> detail disk
DELL PERC H710P SCSI Disk Device
Disk ID: CA0DEEA5
Type   : RAID
Status : Online
Path   : 1
Target : 0
LUN ID : 0
Location Path : PCIROOT(0)#PCI(0100)#PCI(0000)#RAID(P01T00L00)
Current Read-only State : No
Read-only  : No
Boot Disk  : Yes
Pagefile Disk  : Yes
Hibernation File Disk  : No
Crashdump Disk  : Yes
Clustered Disk  : No
  Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
  Volume 1         RECOVERY     NTFS   Partition   3072 MB  Healthy    System
  Volume 2     C   OS           NTFS   Partition    275 GB  Healthy    Boot
B.)
Backup disk...
ST320006 41AS USB Device
Disk ID: 801AF48C
Type   : USB
Status : Online
Path   : 0
Target : 0
LUN ID : 0
Location Path : UNAVAILABLE
Current Read-only State : No
Read-only  : No
Boot Disk  : No
Pagefile Disk  : No
Hibernation File Disk  : No
Crashdump Disk  : No
Clustered Disk  : No
  Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
  Volume 4     F   BACKUPS      NTFS   Partition   1863 GB  Healthy
C.)
NOTE:
Also on disk 0(system Disk) - we have a small OEM partition = 39MB and a Recovery Partition = 3GB.
D.)
Issues with WBAdmin now...
C:\>wbadmin get disks
wbadmin 1.0 - Backup command-line tool
(C) Copyright 2004 Microsoft Corp.
ERROR - A Volume Shadow Copy Service operation error has
occurred: (0x80042302)
A Volume Shadow Copy Service component encountered an unexpected error.
Check the Application event log for more information.
14.
Ran "Check-Disk" -no errors...
C:\>chkdsk
The type of the file system is NTFS.
The volume is in use by another process. Chkdsk
might report errors when no corruption is present.
Volume label is OS.
WARNING!  F parameter not specified.
Running CHKDSK in read-only mode.
CHKDSK is verifying files (stage 1 of 3)...
  116224 file records processed.
File verification completed.
  654 large file records processed.
  0 bad file records processed.
  0 EA records processed.
  76 reparse records processed.
CHKDSK is verifying indexes (stage 2 of 3)...
  156676 index entries processed.
Index verification completed.
  0 unindexed files scanned.
  0 unindexed files recovered.
CHKDSK is verifying security descriptors (stage 3 of 3)...
  116224 file SDs/SIDs processed.
Security descriptor verification completed.
  20227 data files processed.
CHKDSK is verifying Usn Journal...
  35628352 USN bytes processed.
Usn Journal verification completed.
Windows has checked the file system and found no problems.
 289233919 KB total disk space.
 100241492 KB in 85477 files.
     57380 KB in 20228 indexes.
         0 KB in bad sectors.
    226627 KB in use by the system.
     65536 KB occupied by the log file.
 188708420 KB available on disk.
      4096 bytes in each allocation unit.
  72308479 total allocation units on disk.
  47177105 allocation units available on disk.
15.
There are no backup error logs here - only .etl files...
C:\Windows\Logs\WindowsServerBackup\
Wbadmin.1.etl = 9:42AM on 8/12
Wbadmin.2.etl = 4:51AM on 8/12
Per windows Win troubleshooting:
http://technet.microsoft.com/en-us/library/ee692290(WS.10).aspx
http://blogs.technet.com/b/filecab/archive/2009/09/16/diagnosing-failures-in-windows-server-backup-part-1-vss-spp-errors.aspx
--> Related to Windows backup and not VSS?
2155348001 - 0x80780021 - Windows Backup timed-out before the shared protection point was created.
Should we run the "FIXVSS08.BAT" script - restarts services and re-registers DLL's - does this ever help?
Will this server need a reboot since VSS is in a state and we can't even test backups?
Is there a tool we can use to debug VSS or troubleshoot or OS backups further?
MSReport/Diag, VSS debugging, or another way of helping fix our OS(SystemState/C: drive) backup issues?
Any help is greatly appreciated - thx.
-SP
Scot Pichelman

Answer 1

Hi Scot,
Sometimes a third party backup application will change the default VSS provider adding VSS writers which causes Windows Server Backup using an incorrect component in doing backup, that's why I suggest to delete it for a test. Anyway you can reinstall it
after testing.
And as you said a high-load CPU could also be the cause of the issue so you may need to first rule hardware effective out.
If issue still exists, please try the steps below as well:
1. Please check DCOM Server Process Launcher service is started.
2. If not, try to change RPC service to run as SYSTEM account. To do this please modify the following key:
Note: Backup the key before edit incase it cause any issue.
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Servic es\RpcSs\
Find "ObjectName" and change the value to LocalSystem.
Reboot to see if DCOM back to work and if Windows Server Backup starts working. If not, please provide the result if there is any new error occurs.
If DCOM still not work, please compare the content of the following key with a working server:
HKEY_LOCAL_MACHINE\Software\Microsoft\OLE
Edit if there is any difference and make sure you created a backup of the key before editing. 
If you have any feedback on our support, please send to [email protected]