SCOM 2012 R2 agent upgrade fails crippling agents

Question

Running into a large amount of SCOM agents that are failing the upgrade from 2012 SP1 to R2 and would appreciate any feedback from my SCOM community colleagues.  Warning this issue is not for the faint of heart. 
Plan:
I am in the process of upgrading 1900 manually installed SCOM 2012 SP1 agents to R2.  I am using SCCM to deliver the upgrade using the standard sanctioned upgrade parameters. 
Momagent.msi /qn /l*v %SystemDrive%\SCOM2012AgentUpgrade.log AcceptEndUserLicenseAgreement=1
Problem:
I have run into a problem where on a larger group of systems, 165 servers, where the upgrade fails and leaves the agent in a crippled state.  At this point the agent cannot be removed cleanly via add remove, nor can a straight install of the agent
be done.  What is required is a manual removal of registry keys and then a clean install of the agent can be performed to remediate.  This is no problem and I am able to do this.  (Note: all other methods of agent removal did not work ie:
Cleanmom.exe utility or add remove)
Task:
Seeing that we need to do agent upgrades in the future it would be great to know why this happened so we can plan for this in the future.
Notes:
Seems like a random sample of servers (2003/2008) with different applications running on them so that doesn't help in narrowing things down. 
Looking at the MSI log, I see a common issue among systems that had this problem.
******* Product: {387306D9-78CE-4E0E-B952-28A50CC8B3EE}
           ******* Action:
           ******* CommandLine: **********
MSI (s) (7C:7C) [08:08:43:668]: User policy value 'SearchOrder' is 'nmu'
MSI (s) (7C:7C) [08:08:43:668]: User policy value 'DisableMedia' is 0
MSI (s) (7C:7C) [08:08:43:668]: Machine policy value 'AllowLockdownMedia' is 0
MSI (s) (7C:7C) [08:08:43:668]: SOURCEMGMT: Media enabled only if package is safe.
MSI (s) (7C:7C) [08:08:43:668]: SOURCEMGMT: Looking for sourcelist for product {387306D9-78CE-4E0E-B952-28A50CC8B3EE}
MSI (s) (7C:7C) [08:08:43:668]: SOURCEMGMT: Adding {387306D9-78CE-4E0E-B952-28A50CC8B3EE}; to potential sourcelist list (pcode;disk;relpath).
MSI (s) (7C:7C) [08:08:43:668]: SOURCEMGMT: Now checking product {387306D9-78CE-4E0E-B952-28A50CC8B3EE}
MSI (s) (7C:7C) [08:08:43:668]: SOURCEMGMT: Media is enabled for product.
MSI (s) (7C:7C) [08:08:43:668]: SOURCEMGMT: Attempting to use LastUsedSource from source list.
MSI (s) (7C:7C) [08:08:43:668]: SOURCEMGMT: Trying source C:\WINDOWS\SysWOW64\CCM\Cache\LFG00446.1.System\i386\.
MSI (s) (7C:7C) [08:08:43:668]: Note: 1: 2203 2: C:\WINDOWS\SysWOW64\CCM\Cache\LFG00446.1.System\i386\MOMAgent.msi 3: -2147287037
MSI (s) (7C:7C) [08:08:43:668]: SOURCEMGMT: Source is invalid due to missing/inaccessible package.
MSI (s) (7C:7C) [08:08:43:668]: Note: 1: 1706 2: -2147483647 3: MOMAgent.msi
MSI (s) (7C:7C) [08:08:43:668]: SOURCEMGMT: Processing net source list.
MSI (s) (7C:7C) [08:08:43:668]: Note: 1: 1706 2: -2147483647 3: MOMAgent.msi
MSI (s) (7C:7C) [08:08:43:668]: SOURCEMGMT: Processing media source list.
MSI (s) (7C:7C) [08:08:44:744]: Note: 1: 2203 2:  3: -2147287037
MSI (s) (7C:7C) [08:08:44:744]: SOURCEMGMT: Source is invalid due to missing/inaccessible package.
MSI (s) (7C:7C) [08:08:44:744]: Note: 1: 1706 2: -2147483647 3: MOMAgent.msi
MSI (s) (7C:7C) [08:08:44:744]: SOURCEMGMT: Processing URL source list.
MSI (s) (7C:7C) [08:08:44:744]: Note: 1: 1402 2: UNKNOWN\URL 3: 2
MSI (s) (7C:7C) [08:08:44:744]: Note: 1: 1706 2: -2147483647 3: MOMAgent.msi
MSI (s) (7C:7C) [08:08:44:744]: Note: 1: 1706 2:  3: MOMAgent.msi
MSI (s) (7C:7C) [08:08:44:744]: SOURCEMGMT: Failed to resolve source
MSI (s) (7C:8C) [08:08:44:744]: Note: 1: 1714 2: Microsoft Monitoring Agent 3: 1612
CustomAction  returned actual error code 1612 (note this may not be 100% accurate if translation happened inside sandbox)
MSI (s) (7C:8C) [08:08:44:744]: Product: Microsoft Monitoring Agent -- Error 1714.The older version of Microsoft Monitoring Agent cannot be removed. Contact your technical support group. System Error 1612.
Error 1714.The older version of Microsoft Monitoring Agent cannot be removed. Contact your technical support group. System Error 1612.
Action ended 8:08:44: RemoveExistingProducts. Return value 3.
Action ended 8:08:44: INSTALL. Return value 3.
Ok so the obvious is that the installer is looking for original source installation files and not able to find them. What is surprising to me however is that the product references a 32 bit scom agent guid
387306D9-78CE-4E0E-B952-28A50CC8B3EE, however this is a 64 bit machine.  Our build process dictates that a 64 bit machine only receive a 64 bit SCOM agent. 
Doing a search on this product guid I realized I skipped some other references at the top of the MSI log that might offer some more explanation...
MSI (s) (7C:8C) [08:08:11:896]: PROPERTY CHANGE: Adding OM_OM12_SP1_AGENT_FOUND property. Its value is '{8B21425D-02F3-4B80-88CE-8F79B320D330}'.
MSI (s) (7C:8C) [08:08:11:896]: PROPERTY CHANGE: Modifying OM_OM12_SP1_AGENT_FOUND property. Its current value is '{8B21425D-02F3-4B80-88CE-8F79B320D330}'. Its new value: '{8B21425D-02F3-4B80-88CE-8F79B320D330};{387306D9-78CE-4E0E-B952-28A50CC8B3EE}'.
MSI (s) (7C:8C) [08:08:11:896]: Skipping action: _StopCoreServices.80B659D9_F758_4E7D_B4FA_E53FC737DCC9 (condition is false)
MSI (s) (7C:8C) [08:08:11:896]: Skipping action: _KillOMProcesses.80B659D9_F758_4E7D_B4FA_E53FC737DCC9 (condition is false)
MSI (s) (7C:8C) [08:08:11:896]: Doing action: _Set_OM_AGENT_FOUND
Action ended 8:08:11: FindRelatedProducts. Return value 1.
MSI (s) (7C:8C) [08:08:11:896]: PROPERTY CHANGE: Adding OM_AGENT_FOUND property. Its value is '{8B21425D-02F3-4B80-88CE-8F79B320D330};{387306D9-78CE-4E0E-B952-28A50CC8B3EE}'.
Action start 8:08:11: _Set_OM_AGENT_FOUND.
MSI (s) (7C:8C) [08:08:11:896]: Doing action: _Set_MOMV3_AGENT_FOUND
Action ended 8:08:11: _Set_OM_AGENT_FOUND. Return value 1.
MSI (s) (7C:8C) [08:08:11:896]: PROPERTY CHANGE: Adding MOMV3_AGENT_FOUND property. Its value is '{8B21425D-02F3-4B80-88CE-8F79B320D330};{387306D9-78CE-4E0E-B952-28A50CC8B3EE}'.
Action start 8:08:11: _Set_MOMV3_AGENT_FOUND.
Well that is interesting, seems to me that in the first line the MSI installer reads the OM_OM12_SP1_AGENT_FOUND property and identifies this with a  64 bit agent guid (8B21425D-02F3-4B80-88CE-8F79B320D330), but then appends
the 32 bit guid at the end (387306D9-78CE-4E0E-B952-28A50CC8B3EE).  This is the point of my confusion and my suspicion as the cause of the problem I am having. 
Concluding thoughts:
Why does the installer seems to first recognizes a 64 bit agent, but then later changes it's property to include a 32 bit agent guid.  Could a 32 bit agent have got on this 64 bit server based on the details of this log?  Would this
cause my agents to fail the upgrade?  I think so, but looking to bounce this over to another fellow SCOM colleague who may have wrestled with this before.
Thanks in advance if you took the time to read this and think about it.  Extra points if you have any extra thoughts!
Keith

Answer 1

Hi,
Have you used Requirement to limit the platform when you deploy application with SCCM.
We
are trying to better understand customer views on social support experience, so your participation in this
interview project would be greatly appreciated if you have time.
Thanks for helping make community forums a great place.