DPM Secondary Server Failure

Question

Hello,
We've had an issue with our secondary DPM server where a planned update to SCDPM (from 2012 to 2012R2) caused some of the protected clients to not successfully create replicas. At this point I decided to restore the database which seemed like a good idea
at the time as the backups were working, right up until they stopped working.
After restoring the database, none of the issues were solved and I had gained a new problem -- I could no longer edit the protection groups on the secondary server at all. 
At this point I was stuck. I couldn't get the backups to run, and I couldn't edit anything on the server to make it work. I was left with one option as far as I or Google could see -- Blow the Secondary server away and start from scratch. The secondary is
only used as a DR option, so this wasn't a big deal. I removed the agent from the Primary server, uninstalled DPM from the Secondary and deleted the database. I then installed DPM2012 R2 fresh, installed the agent on the Primary from the Secondary and created
a new protection group.
And herein lies the problem: I cannot see the node for "Protected Servers" listed underneath the Primary server.
Now, I know there can be all sorts of drama's related to moving protected servers around in a Primary-Secondary scenario as DPM databases tend to hold onto information. However everything I've read so far would indicate that the issue would lie on the DB
of the Secondary server, which I've completely blown away and as such shouldn't be a problem. 
I have noticed that the Primary server maintains the Server ID of the Secondary server, even after it was uninstalled and removed from the environment, under the tbl_AM_Server table of it's database. I'd have thought that this should be removed upon removing
the agent from the server, however it is still there. Can I manually delete this row from the database, and reinstall the agent to try and force it onto a new (and potentially correct) ID?
I've also noticed that when I install the agent onto the Primary (from the Secondary), the DPMServerID of the Primary server, as listed in the database on the Primary, does not change to point to the Secondary -- is this expected behavior?
If I do a dance from within the database and point the DPMServerID field of the two servers to each other, I can then expand the node from within the Secondary and the Primary will list the "Protected Servers" node, showing the Secondary Server. Obviously
that's not what I want, as I don't want to back the Secondary up to itself, but it does show that the group can be displayed, and may be related to the Primary not believing that it has any protected servers to back up. Is there a table or a field in the DB
that lists secondary servers for protection points, or any other reason why the Primary would believe that there are no valid servers for the Secondary to look after? 
Now, i thought maybe I'm missing some files or folders, so I've verified the folder and service permissions as being correct. The MTATEMP$\PSInfo folder was missing from the Primary server, so I recreated it copying the settings from the Secondary. Still
no change in behavior.
I also get an enumeration error on a share when the Secondary expands to the Primary from within the protection group. Not sure if this is related? It's been happening from before the server died, so I'm thinking not.
Any suggestions on what I can look at would be greatly appreciated. It's pretty awful that we have a product that allows itself to get into this position. I know what we have done isn't exactly a "Supported Scenario", but I wouldn't have expected
this much trouble.
Thanks,
Nick.

Answer 1

Microsoft found the issue.
DPM 2012 R2 – No “Protected Servers” listed on secondary DPM server
Issue was caused by a ##temporarytable left behind from the old DPM server in the stored procedure dbo.prc_AM_PCInfor_TempProtectedServersList_Finalize. Both the _Create and _Insert were correct. Microsoft replaced the incorrect temporary table name for
the correct one.
dbo.prc_AM_PCInfor_TempProtectedServersList_Create
dbo.prc_AM_PCInfor_TempProtectedServersList_Finalize
dbo.prc_AM_PCInfor_TempProtectedServersList_Insert
How to find the correct table name
Event Log – Application. The following event log on the Primary DPM server showed the error and contained the incorrect SQL table name “Invalid object name '##62007ceb_6949_4a55_837e_4d8bf93c5973tmp_tbl_ProtectedPS'.”
You can then look at the stored procedure dbo.prc_AM_PCInfor_TempProtectedServersList_Create for the correct SQL temporary table name.
Replace the incorrect SQL temporary table name in stored procedure dbo.prc_AM_PCInfor_TempProtectedServersList_Finalize with the correct one.
Event log Data
Log Name:      Application
Source:        MSDPM
Date:          27/11/2014 12:31:25
Event ID:      945
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      PrimaryDPM.company.com.au
Description:
The description for Event ID 945 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
If the event originated on another computer, the display information had to be saved with the event.
The following information was included with the event:
Unable to connect to the DPM database because of a general database failure.  Make sure that SQL Server is running and that it is configured correctly.
Problem Details:
<FatalServiceError><__System><ID>19</ID><Seq>19850</Seq><TimeCreated>27/11/2014 1:31:25 AM</TimeCreated><Source>DpmThreadPool.cs</Source><Line>163</Line><HasError>True</HasError></__System><ExceptionType>SqlException</ExceptionType><ExceptionMessage>Invalid
object name '##62007ceb_6949_4a55_837e_4d8bf93c5973tmp_tbl_ProtectedPS'.</ExceptionMessage><ExceptionDetails>System.Data.SqlClient.SqlException (0x80131904): Invalid object name '##62007ceb_6949_4a55_837e_4d8bf93c5973tmp_tbl_ProtectedPS'.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean&amp; dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task&amp; task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task&amp; task, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlRetryCommand.ExecuteNonQuery()
   at Microsoft.Internal.EnterpriseStorage.Dls.AgentManager.DB.AMServerInfo.GenerateProtectedPSServerIDs(SqlContext ctx)
   at Microsoft.Internal.EnterpriseStorage.Dls.AgentManager.DB.ProtectedPSs.GeneratePSInfoXml(Object obj)
ClientConnectionId:d409310a-43bb-48e3-b663-ac4b8cab74c0</ExceptionDetails></FatalServiceError>
the message resource is present but the message is not found in the string/message table
Event Xml:
<Event xmlns=">
  <System>
    <Provider Name="MSDPM" />
    <EventID Qualifiers="0">945</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2014-11-27T01:31:25.000000000Z" />
    <EventRecordID>224715</EventRecordID>
    <Channel>Application</Channel>
    <Computer>PrimaryDPM.company.com.au</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Unable to connect to the DPM database because of a general database failure.  Make sure that SQL Server is running and that it is configured correctly.
Problem Details:
&lt;FatalServiceError&gt;&lt;__System&gt;&lt;ID&gt;19&lt;/ID&gt;&lt;Seq&gt;19850&lt;/Seq&gt;&lt;TimeCreated&gt;27/11/2014 1:31:25 AM&lt;/TimeCreated&gt;&lt;Source&gt;DpmThreadPool.cs&lt;/Source&gt;&lt;Line&gt;163&lt;/Line&gt;&lt;HasError&gt;True&lt;/HasError&gt;&lt;/__System&gt;&lt;ExceptionType&gt;SqlException&lt;/ExceptionType&gt;&lt;ExceptionMessage&gt;Invalid
object name '##62007ceb_6949_4a55_837e_4d8bf93c5973tmp_tbl_ProtectedPS'.&lt;/ExceptionMessage&gt;&lt;ExceptionDetails&gt;System.Data.SqlClient.SqlException (0x80131904): Invalid object name '##62007ceb_6949_4a55_837e_4d8bf93c5973tmp_tbl_ProtectedPS'.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean&amp;amp; dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task&amp;amp; task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task&amp;amp; task, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlRetryCommand.ExecuteNonQuery()
   at Microsoft.Internal.EnterpriseStorage.Dls.AgentManager.DB.AMServerInfo.GenerateProtectedPSServerIDs(SqlContext ctx)
   at Microsoft.Internal.EnterpriseStorage.Dls.AgentManager.DB.ProtectedPSs.GeneratePSInfoXml(Object obj)
ClientConnectionId:d409310a-43bb-48e3-b663-ac4b8cab74c0&lt;/ExceptionDetails&gt;&lt;/FatalServiceError&gt;
</Data>
  </EventData>
</Event>