Virtual Machine Manager 2012 R2 migrate from standalone to cluster - no starting

Question

Hello,
we have migrated our Virtual Machine Manager 2012 R2 UR1 installatin from a standalone machine to a clustered version.
But now the cluster instance won't startup.
- uninstall the standalone virtual machine manager 2012 with the retain database option.
- created failover cluster
- installed vmm into cluster and pointed to existing database.
- added second node
The error we now get is out of the report.txt:
------------------- Error Report -------------------
Error report created 17.04.2014 19:38:57
CLR is not terminating
--------------- Bucketing Parameters ---------------
EventType=VMM20
P1(appName)=vmmservice.exe
P2(appVersion)=3.2.7620.0
P3(assemblyName)=ImgLibEngine.dll
P4(assemblyVer)=3.2.7620.0
P5(methodName)=Microsoft.VirtualManager.DB.Adhc.StoredCertificate.CacheVMConnectCertificate
P6(exceptionType)=System.AggregateException
P7(callstackHash)=7b6a
SCVMM Version=3.2.7620.0
SCVMM flavor=C-buddy-RTL-AMD64
Default Assembly Version=3.2.7620.0
Executable Name=vmmservice.exe
Executable Version=3.2.7510.0
Base Exception Target Site=140721289766296
Base Exception Assembly name=mscorlib.dll
Base Exception Method Name=System.Security.Cryptography.CryptographicException.ThrowCryptographicException
Exception Message=One or more errors occurred.
EIP=0x00007ffc469c43c8
Build bit-size=64
------------ exceptionObject.ToString() ------------
System.AggregateException: One or more errors occurred. ---> System.Security.Cryptography.CryptographicException: The specified network password is not correct.
at System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr)
at System.Security.Cryptography.X509Certificates.X509Utils._LoadCertFromBlob(Byte[] rawData, IntPtr password, UInt32 dwFlags, Boolean persistKeySet, SafeCertContextHandle& pCertCtx)
at System.Security.Cryptography.X509Certificates.X509Certificate.LoadCertificateFromBlob(Byte[] rawData, Object password, X509KeyStorageFlags keyStorageFlags)
at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(Byte[] rawData, SecureString password, X509KeyStorageFlags keyStorageFlags)
at Microsoft.VirtualManager.DB.Adhc.StoredCertificate.CacheVMConnectCertificate(StoredCertificate cert)
at Microsoft.VirtualManager.DB.Adhc.StoredCertificate.ImportCertificates(List`1 certificates, ReportCertImportFailure ReportImportFailure)
at Microsoft.VirtualManager.DB.Adhc.StoredCertificate.ImportAllCertificates(ReportCertImportFailure ReportImportFailure)
at Microsoft.VirtualManager.Engine.VirtualManagerService.LoadCertificates()
at Microsoft.VirtualManager.Engine.VirtualManagerService.TimeStartupMethod(String description, TimedStartupMethod methodToTime)
at System.Threading.Tasks.Task.Execute()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at Microsoft.VirtualManager.Engine.VirtualManagerService.WaitForStartupTasks()
at Microsoft.VirtualManager.Engine.VirtualManagerService.TimeStartupMethod(String description, TimedStartupMethod methodToTime)
at Microsoft.VirtualManager.Engine.VirtualManagerService.ExecuteRealEngineStartup()
at Microsoft.VirtualManager.Engine.VirtualManagerService.TryStart(Object stateInfo)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.TimerQueueTimer.CallCallback()
at System.Threading.TimerQueueTimer.Fire()
at System.Threading.TimerQueue.FireNextTimers()
---> (Inner Exception #0) System.Security.Cryptography.CryptographicException: The specified network password is not correct.
at System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr)
at System.Security.Cryptography.X509Certificates.X509Utils._LoadCertFromBlob(Byte[] rawData, IntPtr password, UInt32 dwFlags, Boolean persistKeySet, SafeCertContextHandle& pCertCtx)
at System.Security.Cryptography.X509Certificates.X509Certificate.LoadCertificateFromBlob(Byte[] rawData, Object password, X509KeyStorageFlags keyStorageFlags)
at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(Byte[] rawData, SecureString password, X509KeyStorageFlags keyStorageFlags)
at Microsoft.VirtualManager.DB.Adhc.StoredCertificate.CacheVMConnectCertificate(StoredCertificate cert)
at Microsoft.VirtualManager.DB.Adhc.StoredCertificate.ImportCertificates(List`1 certificates, ReportCertImportFailure ReportImportFailure)
at Microsoft.VirtualManager.DB.Adhc.StoredCertificate.ImportAllCertificates(ReportCertImportFailure ReportImportFailure)
at Microsoft.VirtualManager.Engine.VirtualManagerService.LoadCertificates()
at Microsoft.VirtualManager.Engine.VirtualManagerService.TimeStartupMethod(String description, TimedStartupMethod methodToTime)
at System.Threading.Tasks.Task.Execute()<---
Maybe someone has an idea where to look at.
Best Regards,
Marcus
Marcus Lehmann

Answer 1

Hi,
nope, we solved it.
It seems that this behavior occurs only under special circumstances.
Scenario is:
1. RDP Gateway connection to our Hyper-V Hosts (We need it for our Windows Azure Pack installation). A certificate is needed to encrypt the connection between RDPGW and Hyper-V host. This certificate is distributed by the VMM to the Hyper-V Hosts with the
private key.
2. Migrating from standalone installation to High Availability therefore migrating from DPAPI to DKM
It looks like that the key or the password to unencrypt the private key, which is stored in the database, is itself stored in the DPAI/DKM.
So when you try to get the migrated Service online the error report in my first post is generated.
What we did was fiddeling around in the database. Make sure you have a backup !!
You need two tables:
tbl_ADHC_Host
tbl_VMM_CertificateStore
1. Go to the table "tbl_ADHC_Host" and edit the key fk_ADHC_Host_VMM_CertificateStore and set "Enfore Foreign Key Constraint" to "No".
2. get the value "VMConnectCertficateID" from the table "tbl_ADHC_Host"
3. Go to the table "tbl_VMM_CertificateStore" and delete the Certficate with the value which equals the "VMConnectCertificateID". Note: Corresponding Certificate in the CErtificatestore table should have something in the column "PrivatekeyPassword"
and an ObjectType of 6.
4. The service should now start and you can reconfigure the certificate used for the encryption between your RDPGW and Hyper-V Hosts, like you did before.
5.  Go to the table "tbl_ADHC_Host" and edit the key fk_ADHC_Host_VMM_CertificateStore and set "Enfore Foreign Key Constraint" back to "Yes".
Hope that helps. ;)
Best Regards,
Marcus
Marcus Lehmann