DPM 2012 UR3 Agent Update Error: HRESULT E_FAIL...DCOM Error

Question

My DPM Server is Windows 2008 R2 SP1 with a locally installed SQL and DPM Server and fully patched. My Workstations are all Windows 7 Ultimate x64, fully patched. I have manually installed the DPM RTM Agent and pushed the UR2 update from the DPM Server
without any challenges or issues. I have SCEP running with process exclusions for the DPM services on the WS and DPM Server. I’ve been running DPM 2012 RTM and then UR2 without any issues or challenges to-date.
I have recently updated my DPM 2012 Server from UR2 to UR3 and pushed out all the Agent Updates, which appeared to go successfully from the DPM Console perspective. I spot checked a few systems, 1 workstation (WS) and 1 server, everything looked good. A
couple of days after this I noticed 2 of my workstations failed backing up to Tape. I noticed the recovery options available for these WS’s have stopped updating hours after pushing the UR3 Agent update. The 2 WS’s I'm troubleshooting have a little different
scenario. I have 2 other WS’s that were updated in the same manner and are working perfectly.
Both WS cannot connect to the DPM Server and displays this DCOM Error:
"ERROR:ERROR HRESULT E_FAIL has been returned from a call to a COM component."
The 1st WS DPM Client has a Yelow Sign over-lay and "Administrator has not configured backups." when you hover over it.
The 2nd WS does not the Yellow Sign over-lay.
All the properties of the Local DPM Client Console 'Summary and Protected item'  tabs are not updated on the 1st WS but are on the 2nd WS.
1st WS process:
The Recovery Tab shows the available recover dates to choice from.
I can expand one of the backup dates, folders and copy a file to this system.
I can view my previous recovery Points from the console dating back to when I updated the Agents.
I’ve validated the DCOM settings on both WS's compared to a WS that is working, they are the same.
I uninstalled the agent manually, then copied and reinstalled the new UR3 Agent Version with the same outcome.
I uninstalled the new version and reinstalled the previous version, same error.
I then proceeded with upgrading UR2 agent version installing the UR3 version over the top, same error.
I logged on to this WS as the Domain Admin, removed and reinstalled the UR3 version, same error.
I removed, applied and then re-added the DPM server to the local WS DPM and DCOM Groups.
I removed, applied and then re-added the WS from the Groups on the DPM Server.
I’m able to see, ping and view other backup folders presented on the DPM server on this WS.
In the DPM Console I am able to connect to this Workstation successfully from the Administrative, Agent Space.
The DPMClientProtectionCurr.errlog file shows this repeatedly:
0FC0   
065C    10/13   05:02:32.499 
03        service.cpp(298)         
[000000000021FD90]            
ACTIVITY         
CService::AnnounceServiceStatus
0FC0   
0D18    10/13   05:02:33.059 
33        dlsclientprotection.cpp(778)           
[00000000002BC850]            
NORMAL          
Connecting to DPM Server: (<Server.FQDN>)
0FC0   
0D18    10/13   05:02:33.171 
33        dlsclientprotection.cpp(861)           
[00000000002BC850]            
WARNING         Failed: Hr: = [0x80004005] : Failed to set client OS details - ClientName:<WS.FQDN>, ClientOSVersion:6.1.7601, OSType:2, AgentVersion: 4.0.1920.0
0FC0   
0D18    10/13   05:02:33.171 
33        dlsclientprotection.cpp(1490)           
[00000000002BC850]            
WARNING         Failed: Hr: = [0x80004005] : Failed to get connection to DPM server: lVal : ConnectToDPMServer(bstrClientMachineName, &pIUnknown, &fConnectionSuccessful, &dCurrentStatus,
exceptionResult)
The
DPMRACurr.eerlog show this for both WS's:
0EF0 1110 10/13 04:05:02.157 22 genericthreadpool.cpp(824) [000000000093C360]  NORMAL Hr: = [0x80070002] CGenericThreadPool::m_dwMaximumNumberOfThreads[20]
0EF0 1110 10/13 04:05:02.157 03 miscellaneousutils.cpp(918)   NORMAL Error:ERROR_UNKNOWN_PRODUCT, While detecting DPM version. Assuming DPM isnt installed.
And this one repeatedly:
0BE8 08EC 10/13 13:59:40.404 29 radefaultsubtask.cpp(382) [00000000002F1640] 4F801526-72F2-4460-8760-F9D6E127FECB WARNING Failed: Hr: = [0x809909b0] : CRADefaultSubTask: WorkitemID does not exist, {DB3F138D-DE37-414D-BADC-5A161C6E7E1C}
0BE8 08EC 10/13 13:59:40.404 05 defaultsubtask.cpp(546) [00000000002F1640] 4F801526-72F2-4460-8760-F9D6E127FECB WARNING Failed: Hr: = [0x809909b0] : Encountered Failure: : lVal : CommandReceivedSpecific(pCommand,
pOvl)
0BE8 08EC 10/13 13:59:40.404 05 defaultsubtask.cpp(751) [00000000002F1640] 4F801526-72F2-4460-8760-F9D6E127FECB WARNING Failed: Hr: = [0x809909b0] : Encountered Failure: : lVal : CommandReceived(pAgentOvl)
The 2nd WS Process:
I did not remove or touch the local agent installation because the information is populated on this client’s agent properties fields.
I can see the folder selections I’ve chosen to be backed up by the protection group configuration.
When i open the Agent UI I receive the sam DCOM error.
I also receive a different error message on this WS when I try and do a Sync Now from the General Tab: Failed to trigger synchronization.
I was able to perform a consistency check on this WS that completed successfully from the DPM server.
I’ve validated the networking configuration settings are correct.
I flushed local DNS cache.
Thanks,
Paul

Answer 1

Hi,
I have a Windows 7 x64 machine that exhibits this problem. It seems that it is not caused on the client side but on the server side.
In our case the server logs the following every time the client tries to connect:
WARNING    Verify disconnected client failed with exception: System.ArgumentException: Das Element wurde bereits hinzugefügt. Schlüssel im Wörterbuch: "<machinename>". Hinzuzufügender Schlüssel: "<machinename>".
(Translation: Element has already been added. Key in Table: "<machinename>". Key to be added: "<machinename>".
I have uninstalled the agent on the client several times. One time I have seen the following error message on the server side:
Deployment.cs(2815)            NORMAL    Removing all agent records for server[<machinename>] from table InstalledAgent.
ntlmutils.cpp(185)            WARNING    Failed: Hr: = [0x80070002] : Error trying to open key [HKLM\Software\Microsoft\Microsoft Data Protection Manager\Agent\2.0\NtlmAuthData\<machinename>]
SchedulerImpl.cs(166)            NORMAL    Entering DeRegister, JobDefID: bafe2b8b-53b2-4694-aa28-9e2c6c152d51
Catalog.cs(1006)            WARNING    No retry on exception The specified @name ('') does not exist. while executing sp_delete_alert
Catalog.cs(1013)            WARNING    SqlException encountered, SqlRetryCommand diag details - SqlCommandText  => Name=msdb.dbo.sp_delete_alert, CommandType=StoredProcedure
Catalog.cs(1013)            WARNING    CommandDiagInfo => CanRetry=False, CommandTimeout=3600
Catalog.cs(1013)            WARNING    CommandParams   => Count=2, InTx=True
Catalog.cs(1013)            WARNING         Param[0]   => ParameterName=@name | Value= | Size=0 | DbType=NVarChar | Direction=Input | IsNullable=False
Catalog.cs(1013)            WARNING         Param[1]   => ParameterName=@RETURN_VALUE | Value=1 | Size=0 | DbType=Int | Direction=ReturnValue | IsNullable=False
EventManager.cs(98)            NORMAL    Publishing event from SchedulerImpl.cs(838): JobScheduleChange, [JobDefID=bafe2b8b-53b2-4694-aa28-9e2c6c152d51]
Errors on the client side appear the moment the agent is re-added to the DPM server (I use "connect agent" instead of "install agent"). After that moment I get the message "Error connection to the DPM server. Error: a COM-component returned E_FAIL" every time
I open the DPM client. This is mirrored by the Verify error mentioned above on the server side.
To me it seems that parts of the machine config are still in the MSDB after removal (maybe caused by the registry error). If this client is connected again, the orphaned database entries cause the Verify error
If that's true, how could we get rid of those entries? Would it help to rename the client?
Uwe