WLST/start AdminServer - problems with trusted cert key store

Question

Hello,
I have clustered environment. Machine1: AdminServer and odi_server1. Machine2: odi_server2. There is NodeManager running on each machine. This is my nodemanager.properties for NodeManager on Machine1:
#Thu Dec 19 13:18:30 CET 2013
#Thu Dec 19 11:29:43 CET 2013
#Thu Dec 19 11:17:53 CET 2013
#Tue Dec 11 11:40:20 CET 2012
DomainsFile=/home/oracle/Oracle/Middleware/wlserver_10.3/common/nodemanager/nodemanager.domains
LogLimit=0
PropertiesVersion=10.3
DomainsDirRemoteSharingEnabled=false
javaHome=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64
AuthenticationEnabled=true
NodeManagerHome=/home/oracle/Oracle/Middleware/wlserver_10.3/common/nodemanager
JavaHome=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre
LogLevel=INFO
DomainsFileEnabled=true
StartScriptName=startWebLogic.sh
ListenAddress=
NativeVersionEnabled=true
ListenPort=5556
LogToStderr=true
SecureListener=true
LogCount=1
DomainRegistrationEnabled=false
StopScriptEnabled=false
QuitEnabled=false
LogAppend=true
StateCheckInterval=500
CrashRecoveryEnabled=false
StartScriptEnabled=true
LogFile=/home/oracle/Oracle/Middleware/wlserver_10.3/common/nodemanager/nodemanager.log
LogFormatter=weblogic.nodemanager.server.LogFormatter
ListenBacklog=50
KeyStores=CustomIdentityAndCustomTrust
CustomIdentityKeystoreType=jks
CustomIdentityKeyStoreFileName=/home/oracle/Oracle/Middleware/user_projects/domains/odi_cluster/keystore.jks
CustomIdentityKeyStorePassPhrase={3DES}VRCBXCfDocQ=
CustomTrustKeystoreType=jks
CustomTrustKeyStoreFileName=/home/oracle/Oracle/Middleware/user_projects/domains/odi_cluster/cacerts.jks
CustomTrustKeyStorePassPhrase=
CustomIdentityAlias=keyAlias
CustomIdentityPrivateKeyPassPhrase={3DES}VRCBXCfDocQ=
As you can see, I have my custom trust (cacerts,jks) and identity (keystore.jks) keystores and they are set for node manager in this file. Next, nodemanager is started via wlst, like this:
bea_home = '/home/oracle/Oracle/Middleware';
pathseparator = '/';
listen_port = '5556';
listen_address = 'eb-etl1';
node_manager_home = bea_home + pathseparator + 'wlserver_10.3' + pathseparator + 'common' + pathseparator + 'nodemanager';
startNodeManager(verbose='true', NodeManagerHome=node_manager_home, ListenPort=listen_port, ListenAddress=listen_address);
I want to start my AdminServer via wlst (by connectiong to nodemanager), like this:
bea_home = '/home/oracle/Oracle/Middleware';
pathseparator = '/';
admin_username = 'weblogic';
admin_password = '1q2w3e1q2w3e';
listen_address = 'eb-etl1';
listen_port = '5556';
admin_server_url='t3://eb-etl1:7005'
domain_name = 'odi_cluster';
domain_home = bea_home + pathseparator + 'user_projects' + pathseparator + 'domains' + pathseparator + domain_name;
print 'CONNECT TO NODE MANAGER';
nmConnect(admin_username, admin_password, listen_address, listen_port, domain_name, domain_home, 'ssl');
print 'START ADMIN SERVER ONLY ON THE MACHINE WHERE THE ADMIN SERVER IS PRESENT';
nmStart('AdminServer');
print 'CONNECT TO ADMIN SERVER';
connect(admin_username, admin_password, admin_server_url);
print 'START MANAGED SERVERS ON THE MACHINE';
start('odi_server1','Server');
But I can't even connect to node manager:
CONNECT TO NODE MANAGER
Connecting to Node Manager ...
<2013-12-19 13:48:23 CET> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<2013-12-19 13:48:23 CET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
<2013-12-19 13:48:24 CET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=Entrust Root Certification Authority - G2,OU=(c) 2009 Entrust\, Inc. - for authorized use only,OU=See www.entrust.net/legal-terms,O=Entrust\, Inc.,C=US". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<2013-12-19 13:48:24 CET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=thawte Primary Root CA - G3,OU=(c) 2008 thawte\, Inc. - For authorized use only,OU=Certification Services Division,O=thawte\, Inc.,C=US". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<2013-12-19 13:48:24 CET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=T-TeleSec GlobalRoot Class 3,OU=T-Systems Trust Center,O=T-Systems Enterprise Services GmbH,C=DE". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<2013-12-19 13:48:24 CET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=T-TeleSec GlobalRoot Class 2,OU=T-Systems Trust Center,O=T-Systems Enterprise Services GmbH,C=DE". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<2013-12-19 13:48:24 CET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=GlobalSign,O=GlobalSign,OU=GlobalSign Root CA - R3". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<2013-12-19 13:48:24 CET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "OU=Security Communication RootCA2,O=SECOM Trust Systems CO.\,LTD.,C=JP". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<2013-12-19 13:48:24 CET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=VeriSign Universal Root Certification Authority,OU=(c) 2008 VeriSign\, Inc. - For authorized use only,OU=VeriSign Trust Network,O=VeriSign\, Inc.,C=US". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<2013-12-19 13:48:24 CET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=KEYNECTIS ROOT CA,OU=ROOT,O=KEYNECTIS,C=FR". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<2013-12-19 13:48:24 CET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=GeoTrust Primary Certification Authority - G3,OU=(c) 2008 GeoTrust Inc. - For authorized use only,O=GeoTrust Inc.,C=US". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<2013-12-19 13:48:24 CET> <Warning> <Security> <BEA-090542> <Certificate chain received from eb-etl1 - 172.18.0.106 was not trusted causing SSL handshake failure. Check the certificate chain to determine if it should be trusted or not. If it should be trusted, then update the client trusted CA configuration to trust the CA certificate that signed the peer certificate chain. If you are connecting to a WLS server that is using demo certificates (the default WLS server behavior), and you want this client to trust demo certificates, then specify -Dweblogic.security.TrustKeyStore=DemoTrust on the command line for this client.>
This Exception occurred at Thu Dec 19 13:48:24 CET 2013.
javax.net.ssl.SSLKeyException: [Security:090542]Certificate chain received from eb-etl1 - 172.18.0.106 was not trusted causing SSL handshake failure. Check the certificate chain to determine if it should be trusted or not. If it should be trusted, then update the client trusted CA configuration to trust the CA certificate that signed the peer certificate chain. If you are connecting to a WLS server that is using demo certificates (the default WLS server behavior), and you want this client to trust demo certificates, then specify -Dweblogic.security.TrustKeyStore=DemoTrust on the command line for this client.
Problem invoking WLST - Traceback (innermost last):
  File "/home/oracle/Oracle/Middleware/deploy/scripts/startBiatelbit_puw.py", line 12, in ?
  File "<iostream>", line 123, in nmConnect
  File "<iostream>", line 648, in raiseWLSTException
WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : [Security:090542]Certificate chain received from eb-etl1 - 172.18.0.106 was not trusted causing SSL handshake failure. Check the certificate chain to determine if it should be trusted or not. If it should be trusted, then update the client trusted CA configuration to trust the CA certificate that signed the peer certificate chain. If you are connecting to a WLS server that is using demo certificates (the default WLS server behavior), and you want this client to trust demo certificates, then specify -Dweblogic.security.TrustKeyStore=DemoTrust on the command line for this client.
Use dumpStack() to view the full stacktrace
So - it seems my trust keystore is not even used, why? Why still demo key store is used??
If I remove this:
KeyStores=CustomIdentityAndCustomTrust
CustomIdentityKeystoreType=jks
CustomIdentityKeyStoreFileName=/home/oracle/Oracle/Middleware/user_projects/domains/odi_cluster/keystore.jks
CustomIdentityKeyStorePassPhrase={3DES}VRCBXCfDocQ=
CustomTrustKeystoreType=jks
CustomTrustKeyStoreFileName=/home/oracle/Oracle/Middleware/user_projects/domains/odi_cluster/cacerts.jks
CustomTrustKeyStorePassPhrase=
CustomIdentityAlias=keyAlias
CustomIdentityPrivateKeyPassPhrase={3DES}VRCBXCfDocQ=
from my nodemanager.properties, there is no exception while connecting to node manager and I can start admin server. But - I can't start odi_server1 (weblogic console says that node manager for Machine1 is unreachable). From other hand, when I run AdminServer via startWebLogic script (with above keystore definitions), I can start my odi_server1 via weblogic administration console without any problems.
Also, NodeManager for Machine2 is always unreachable, no matter what I do (with or without keystore definitions).
Do you have any idea what am I doing wrong?

Answer 1

Hi,
If the admin URL is specified with the https protocol, then http tunneling must be enabled for the server from the console -> servers -> AdminServer ->Protocols -> http.
Moreover we also need to add following java options to the stopWebLogic.cmd or setDomainEnv.cmd:
set JAVA_OPTIONS=$JAVA_OPTIONS$ -Dweblogic.security.IdentityKeyStore=CustomIdentity -Dweblogic.security.CustomIdentityKeyStoreFileName=identity.jks -Dweblogic.security.CustomIdentityKeyStorePassPhrase=password -Dweblogic.security.Identity.KeyStoreType=JKS -Dweblogic.security.TrustKeyStore=CustomTrust -Dweblogic.security.CustomTrustKeyStoreFileName=trust.jks -Dweblogic.security.CustomTrustKeyStoreType=JKS -Dweblogic.security.CustomTrustKeyStorePassPhrase=password -Dweblogic.security.IgnoreHostNameVerification=true -Dweblogic.security.SSL.ignoreHostnameVerification=true
Regards,
Kal