ACS forwarding from untrusted domain 0x80090325 SEC_E_UNTRUSTED_ROOT

Question

I have SCOM 2012 R2 Update Rollup 4 installed with 2 management servers running WS12R2 in a single management group in my main AD domain. One of the management servers is also an ACS collector. I have an untrusted AD domain, with a SCOM gateway server in
it, and I used the gateway to install a SCOM agent on a domain controller in that domain. Now I am trying to configure an ACS forwarder on that untrusted domain controller to talk to the ACS collector back on the management server.
However, when I restart the
Microsoft Monitoring Agent Audit Forwarding service on that domain controller, I get this error in its
Event Viewer > Apps and Services > Operations Manager:
1/23/2015 5:08:01 PM Source AdtAgent Event ID 4369 Forwarder unsuccessfully tried to connect to the following collector(s):
<acsCollectorFQDN>:51909, status: 0x80090325 (TCP connect), source:registry addresses tried: <IP>:51909. If the list of collectors is blank, then AdtAgent was unable to locate a collector. Common reasons for this message are: The machinef(s)
listed is not online. AdtServer is not running on the machine(s) listed. AdtServer on the machine(s) listed is not listening on the specified port. TCP connectivity to the AdtServer machine is blocked by firewall, IPSec, or other filtering mechanism AdtServer
on the machine(s) listed actively refused the connection (due to policy or current activity load). For detailed failure information, enable trace logging using the TraceFlags registry key and examine the AdtAgent.log in the \temp subdirectory of the Windows
directory.
I followed these two articles in order to set up the ACS forwarder on the DC in the untrusted domain: "How to configure security events collection by using Audit Collection Services from computers in untrusted environment?" {1/3/12}https://gefufna.wordpress.com/2012/01/03/how-to-configure-security-events-collection-by-using-audit-collection-services-from-computers-in-untrusted-environment/ "Forwarder
is unable to connect to collector Event id 4369 in forwarder event view" {5/5/14}
http://jimmy-scom.blogspot.com/2014/05/forwarder-is-unable-to-connect-to.html
EXTRA INFO Here are the detailed steps that I took (sorry for all this, but there are an awful number of steps!):
1) I confirmed that the agent for the DC shows as Healthy in OM Console > Monitoring > Operations Manager > Agent Details > Agent Health State > Agent State (right) pane.
2) On the ACS collector, I stopped
Operations Manager Audit Collection Service, then from Admin cmd prompt I did this:
c:> cd \windows\system32\security\adtserver
c:> adtserver –c
} 1 certificates found for server authentication usage.
Enter the number of the certificate you want AdtServer to use for authenticating to AdtAgent or 0 to quit without saving: 1
Certificate 1 selected. Attempting to save thumbprint to registry ...
success.
Then I started
Operations Manager Audit Collection Service.
3) On the DC in the untrusted domain, from Admin cmd prompt I did this:
c:> cd c:\windows\system32
c:> adtagent -c
} No  Issued To                   Issued By                   Expires   
Thumbprint
 1: <untrustedDCfqdn> <untrustedDomainCA>             2015-11-30 02:44:58    <thumbprint>
2 certificates found for client authentication usage.
Enter the number of the certificate you want AdtAgent to use for authenticating to AdtServer or 0 to quit without saving: > 1
} Certificate 1 selected. Attempting to save thumbprint to registry… success.
4) On the DC in the untrusted domain, I opened mmc > Certificates > Local Computer > Personal > Certificates > I exported the certificate from step 3 to a DER encoded binary X.509 (.CER) file.
5) I also looked at the Certification Path for the certificate, and figured out which certificate is its Root CA certificate. I copied that certificate to a DER encoded binary X.509 (.CER) file.
6) I copied the first .CER file to a computer in my main domain, which is at 2012 R2 level. From AD Users and Computers, I created a "dummy" computer object using the NetBios name of the DC back on the untrusted domain. I right clicked the computer
object > Named Mappings > I added the .CER file, and left "Use Subject for alternate identity" checked. I unchecked "Use Issuer for alternate security identity".
7) I copied the Root CA certificate .CER file over to the SCOM management server that doubles as my ACS collector, and from there I did mmc > Certificates > Local Computer > Trusted Root Certificates > Certificates > I imported the Root
CA certificate.
8) I also went to my CA server on my main domain, I ran pkiview.msc > right clicked “Enterprise PKI” > Manage AD Containers > NTAuthCertificates tab > and I imported the Root CA certificate there as well.
9) I ran telnet from the DC on the untrusted domain, and confirmed that port 51909 is open from there to the ACS collector on the main domain.
10) I enabled audit collection fot the DC on the untrusted domain. I did this from OM Console > Monitoring > Operations Manager > Agent Details > Agent Health State > Agent State (second column in middle pane) > I selected the Healthy <untrustedDCfqdn>
> I clicked Enable Audit Collection.
Then under "Task Parameters" > i clicked [Override] > for New Value I specified <ACScollectorFQDN>. For task credentials I specified Other account, and specified a domain admin account in the untrusted domain. The result was "The
task completed successfully. Enable Audit Collection, status:Success".
11) On the ACS collector, I restarted Operations Manager Audit Collection Service. On the DC in the untrusted domain I restarted Microsoft Monitoring Agent Audit Forwarding service.
12) Result was this error on the DC in the untrusted domain, in its
Event Viewer > Apps and Services > Operations Manager
1/23/2015 5:08:01 PM Source AdtAgent Event ID 4369 Forwarder unsuccessfully tried to connect to the following collector(s):
<acsCollectorFQDN>:51909, status: 0x80090325 (TCP connect), source:registry addresses tried: 10.1.1.91:51909. If the list of collectors is blank, then AdtAgent was unable to locate a collector. Common reasons for this message are: The machinef(s)
listed is not online. AdtServer is not running on the machine(s) listed. AdtServer on the machine(s) listed is not listening on the specified port. TCP connectivity to the AdtServer machine is blocked by firewall, IPSec, or other filtering mechanism AdtServer
on the machine(s) listed actively refused the connection (due to policy or current activity load). For detailed failure information, enable trace logging using the TraceFlags registry key and examine the AdtAgent.log in the \temp subdirectory of the Windows
directory.
13) On the DC in the untrusted domain I created DWORD reg value
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AdtAgent\Parameters\TraceFlags and set it to 524420 decimal. The resulting c:\windows\temp\AdtAgent.log file only confirmed that I'm getting 0x80090325 errors.
After all this, why am I getting 0x80090325, which translates to SEC_E_UNTRUSTED_ROOT ??? Did I do something wrong in steps 5, 7 and 8? Thanks for reading all the way through :)
Marko

Answer 1

Thanks Yan Li, you gave me an idea. I got the ACS forwarder in the untrusted domain to work (!), by analyzing the setup on the SCOM gateway that I set up in the untrusted domain. I issued the ACS forwarder a certificate from the domain that SCOM is in, INSTEAD
of configuring the ACS forwarder to use the certificate that it already had from its own domain.
So the new procedure is: do steps 1 and 2, then instead of step 3 I did this…
2B) I issued a certificate from the AD domain containing SCOM to the domain controller in the untrusted domain that is my ACS forwarder. I did this from the AD Certificate Services web site, and asked it to use certificate template that I created for the
SCOM gateway server in the untrusted domain.
2C) The new certificate appeared in the Personal store of the domain controller. I exported it, then ran the MomCertImport utility so that I would not get an error in the next step (per
http://www.systemcentercentral.com/scom-deployment-across-multiple-networks/)
3) On the domain controller in the untrusted domain, I re-ran "adtserver -c", and selected the new certificate.
3B) I then ran “MomCertImport /Remove”, since I already have a SCOM gateway in the untrusted domain.
Then I proceeded with steps 4, skipped 5, did 6, skip 7-8, did 9-11, result was this on the DC in the untrusted domain, in its Event Viewer > Apps and Services > Operations Manager
2/3/2015 12:20:01 PM Source AdtAgent Event ID 4368 Forwarder successfully connected to the following collector:
<ACScollectorFQDN>:51909, status: 0x0 (success), source: registry
addresses tried: <IPaddress>:51909
ACS forwarding works now! I will confirm by repeating the procedure for another domain controller in the untrusted forest.
Marko