Error configuring and starting Managed Server in WebLogic 10.3

Question

Hi,
I am able to configure and start the Managed Server on windows environment. I am trying to configure Managed Server on linux environment. I am configuring the Managed Server with custom keystore. When I configured the Managed Server I specified the keystore/trustsote information in the Keystores tab of the Admin Console for Managed Server settings. Specified Private Key Alias, Password in SSL tab.But when I am starting the Managed Server from command line, I see the process looking for demo trust files and failing.
Here is the command I run to start the Managed Server:
./startManagedWebLogic.sh MyManagedServer1 https://localhost.localdomain:8443
Below is the console output:
<Mar 11, 2015 12:37:44 PM PDT> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 20.45-b01 from Sun Microsystems Inc.>
<Mar 11, 2015 12:37:45 PM PDT> <Info> <Security> <BEA-090065> <Getting boot identity from user.>
Enter username to boot WebLogic server:beaadmin
Enter password to boot WebLogic server:
<Mar 11, 2015 12:37:53 PM PDT> <Info> <Security> <BEA-090908> <Using default WebLogic SSL Hostname Verifier implementation.>
<Mar 11, 2015 12:37:53 PM PDT> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /opt/wlserver_10.3/server/lib/DemoTrust.jks.>
<Mar 11, 2015 12:37:53 PM PDT> <Error> <Security> <BEA-090132> <Could not open the keystore file /opt/wlserver_10.3/server/lib/DemoTrust.jks for read access. Exception: java.io.FileNotFoundException: /opt/wlserver_10.3/server/lib/DemoTrust.jks (No such file or directory)>
<Mar 11, 2015 12:37:53 PM PDT> <Warning> <Security> <BEA-090164> <Failed to load trusted certificates from keystore /opt/wlserver_10.3/server/lib/DemoTrust.jks of type jks>
<Mar 11, 2015 12:37:53 PM PDT> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /opt/jre1.6.0_45/lib/security/cacerts.>
<Mar 11, 2015 12:37:53 PM PDT> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=Entrust Root Certification Authority - G2,OU=(c) 2009 Entrust\, Inc. - for authorized use only,OU=See www.entrust.net/legal-terms,O=Entrust\, Inc.,C=US". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Mar 11, 2015 12:37:53 PM PDT> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=thawte Primary Root CA - G3,OU=(c) 2008 thawte\, Inc. - For authorized use only,OU=Certification Services Division,O=thawte\, Inc.,C=US". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Mar 11, 2015 12:37:53 PM PDT> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=T-TeleSec GlobalRoot Class 3,OU=T-Systems Trust Center,O=T-Systems Enterprise Services GmbH,C=DE". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Mar 11, 2015 12:37:53 PM PDT> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=T-TeleSec GlobalRoot Class 2,OU=T-Systems Trust Center,O=T-Systems Enterprise Services GmbH,C=DE". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Mar 11, 2015 12:37:53 PM PDT> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=GlobalSign,O=GlobalSign,OU=GlobalSign Root CA - R3". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Mar 11, 2015 12:37:53 PM PDT> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "OU=Security Communication RootCA2,O=SECOM Trust Systems CO.\,LTD.,C=JP". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Mar 11, 2015 12:37:53 PM PDT> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=VeriSign Universal Root Certification Authority,OU=(c) 2008 VeriSign\, Inc. - For authorized use only,OU=VeriSign Trust Network,O=VeriSign\, Inc.,C=US". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Mar 11, 2015 12:37:53 PM PDT> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=KEYNECTIS ROOT CA,OU=ROOT,O=KEYNECTIS,C=FR". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Mar 11, 2015 12:37:53 PM PDT> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=GeoTrust Primary Certification Authority - G3,OU=(c) 2008 GeoTrust Inc. - For authorized use only,O=GeoTrust Inc.,C=US". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Mar 11, 2015 12:37:53 PM PDT> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.6.0  Tue Nov 15 08:52:36 PST 2011 1441050 >
<Mar 11, 2015 12:37:54 PM PDT> <Warning> <Security> <BEA-090477> <Certificate chain received from localhost.localdomain - 127.0.0.1 was not trusted causing SSL handshake failure.>
<Mar 11, 2015 12:37:54 PM PDT> <Warning> <Security> <BEA-090477> <Certificate chain received from localhost.localdomain - 127.0.0.1 was not trusted causing SSL handshake failure.>
<Mar 11, 2015 12:37:54 PM PDT> <Warning> <Security> <BEA-090477> <Certificate chain received from localhost.localdomain - 127.0.0.1 was not trusted causing SSL handshake failure.>
<Mar 11, 2015 12:37:54 PM PDT> <Warning> <Security> <BEA-090477> <Certificate chain received from localhost.localdomain - 127.0.0.1 was not trusted causing SSL handshake failure.>
<Mar 11, 2015 12:37:54 PM PDT> <Warning> <Security> <BEA-090477> <Certificate chain received from localhost.localdomain - 127.0.0.1 was not trusted causing SSL handshake failure.>
<Mar 11, 2015 12:37:55 PM PDT> <Warning> <Security> <BEA-090477> <Certificate chain received from localhost.localdomain - 127.0.0.1 was not trusted causing SSL handshake failure.>
<Mar 11, 2015 12:37:55 PM PDT> <Warning> <Security> <BEA-090477> <Certificate chain received from localhost.localdomain - 127.0.0.1 was not trusted causing SSL handshake failure.>
<Mar 11, 2015 12:37:55 PM PDT> <Emergency> <Management> <BEA-141151> <The admin server could not be reached at https://localhost.localdomain:8443.>
<Mar 11, 2015 12:37:55 PM PDT> <Info> <Configuration Management> <BEA-150018> <This server is being started in managed server independence mode in the absence of the admin server.>
<Mar 11, 2015 12:37:55 PM PDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Mar 11, 2015 12:37:55 PM PDT> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<Mar 11, 2015 12:37:55 PM PDT> <Notice> <LoggingService> <BEA-320400> <The log file /h/data/local/base_domain/servers/MyManagedServer1/logs/MyManagedServer1.log will be rotated. Reopen the log file if tailing has stopped. This can happen on some platforms like Windows.>
<Mar 11, 2015 12:37:55 PM PDT> <Notice> <LoggingService> <BEA-320401> <The log file has been rotated to /h/data/local/base_domain/servers/MyManagedServer1/logs/MyManagedServer1.log00015. Log messages will continue to be logged in /h/data/local/base_domain/servers/MyManagedServer1/logs/MyManagedServer1.log.>
<Mar 11, 2015 12:37:55 PM PDT> <Notice> <Log Management> <BEA-170019> <The server log file /h/data/local/base_domain/servers/MyManagedServer1/logs/MyManagedServer1.log is opened. All server side log events will be written to this file.>
<Mar 11, 2015 12:37:56 PM PDT> <Error> <Security> <BEA-000000> <[Security:090837]The configured keystore location for the PKI Credential Mapper is either null or empty.>
<Mar 11, 2015 12:37:56 PM PDT> <Error> <Security> <BEA-000000> <[Security:090815]The password provided for the keystore configured for the PKICredential Mapper was null.>
<Mar 11, 2015 12:37:57 PM PDT> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Mar 11, 2015 12:37:57 PM PDT> <Critical> <Security> <BEA-090403> <Authentication for user beaadmin denied>
<Mar 11, 2015 12:37:57 PM PDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication for user beaadmin denied
weblogic.security.SecurityInitializationException: Authentication for user beaadmin denied
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:966)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)
        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
        at weblogic.security.SecurityService.start(SecurityService.java:141)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        Truncated. see log file for complete stacktrace
Caused By: javax.security.auth.login.FailedLoginException: [Security:090303]Authentication Failed: User beaadmin weblogic.security.providers.authentication.LDAPAtnDelegateException: [Security:090295]caught unexpected exception
        at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:251)
        at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
        at java.security.AccessController.doPrivileged(Native Method)
        at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        Truncated. see log file for complete stacktrace
>
<Mar 11, 2015 12:37:57 PM PDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Mar 11, 2015 12:37:57 PM PDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<Mar 11, 2015 12:37:57 PM PDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
Why is it looking for demotrust.jks file to start the Managed Server. We don't use the demo trust and cannot copy to the location it is referring to. I followed the solution provided at the below link:
http://serverfault.com/questions/106499/custom-trust-and-custom-identity-keystore-in-weblogic-10-3
There are 2 options given in the link. To update the boot.properties of the Admin Server and appending JAVA_OPTIONS to DomainEnv file. Both of these caused the failure of Admin Server. I couldn't bring up the server after these changes.I also came across posts talking about boot.properties for Managed Server. I don't see the security folder that has boot.properties for my Managed Server (verified it on both windows and linux boxes).
Thanks in advance.
-nn.

Answer 1

Hi Faisal,
Here is the config.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<domain xsi:schemaLocation="http://xmlns.oracle.com/weblogic/security/wls http://xmlns.oracle.com/weblogic/security/wls/1.0/wls.xsd http://xmlns.oracle.com/weblogic/domain http://xmlns.oracle.com/weblogic/1.0/domain.xsd http://xmlns.oracle.com/weblogic/security http://xmlns.oracle.com/weblogic/1.0/security.xsd http://xmlns.oracle.com/weblogic/security/xacml http://xmlns.oracle.com/weblogic/security/xacml/1.0/xacml.xsd"
        xmlns="http://xmlns.oracle.com/weblogic/domain" xmlns:sec="http://xmlns.oracle.com/weblogic/security"
        xmlns:wls="http://xmlns.oracle.com/weblogic/security/wls" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <name>JCRMDomain</name>
    <domain-version>10.3.4.0</domain-version>
    <security-configuration xmlns:xacml="http://xmlns.oracle.com/weblogic/security/xacml">
        <name>JCRMDomain</name>
        <realm>
            <sec:authentication-provider xsi:type="wls:default-authenticatorType">
                <sec:control-flag>SUFFICIENT</sec:control-flag>
            </sec:authentication-provider>
            <sec:authentication-provider xmlns:ext="http://xmlns.oracle.com/weblogic/security/extension" xsi:type="ext:jcrm-authenticatorType">
                <sec:name>JcrmAuthenticator</sec:name>
                <sec:control-flag>SUFFICIENT</sec:control-flag>
            </sec:authentication-provider>
            <sec:role-mapper xmlns:xac="http://xmlns.oracle.com/weblogic/security/xacml" xsi:type="xac:xacml-role-mapperType"></sec:role-mapper>
            <sec:authorizer xmlns:xac="http://xmlns.oracle.com/weblogic/security/xacml" xsi:type="xac:xacml-authorizerType"></sec:authorizer>
            <sec:adjudicator xsi:type="wls:default-adjudicatorType"></sec:adjudicator>
            <sec:credential-mapper xsi:type="wls:default-credential-mapperType"></sec:credential-mapper>
            <sec:credential-mapper xsi:type="wls:pki-credential-mapperType">
                <sec:name>PKICredentialMapper</sec:name>
            </sec:credential-mapper>
            <sec:cert-path-provider xsi:type="wls:web-logic-cert-path-providerType"></sec:cert-path-provider>
            <sec:cert-path-builder>WebLogicCertPathProvider</sec:cert-path-builder>
            <sec:user-lockout-manager>
                <sec:lockout-threshold>3</sec:lockout-threshold>
                <sec:lockout-duration>15</sec:lockout-duration>
                <sec:lockout-reset-duration>99999</sec:lockout-reset-duration>
            </sec:user-lockout-manager>
            <sec:deploy-role-ignored>false</sec:deploy-role-ignored>
            <sec:deploy-policy-ignored>false</sec:deploy-policy-ignored>
            <sec:fully-delegate-authorization>true</sec:fully-delegate-authorization>
            <sec:security-dd-model>DDOnly</sec:security-dd-model>
            <sec:combined-role-mapping-enabled>true</sec:combined-role-mapping-enabled>
            <sec:name>myrealm</sec:name>
            <sec:delegate-m-bean-authorization>false</sec:delegate-m-bean-authorization>
            <sec:password-validator xmlns:pas="http://xmlns.oracle.com/weblogic/security/providers/passwordvalidator"
                                    xsi:type="pas:system-password-validatorType">
                <sec:name>JCRMPasswordValidator</sec:name>
                <pas:max-password-length>31</pas:max-password-length>
                <pas:min-password-length>15</pas:min-password-length>
                <pas:min-numeric-characters>2</pas:min-numeric-characters>
                <pas:min-lowercase-characters>2</pas:min-lowercase-characters>
                <pas:min-uppercase-characters>2</pas:min-uppercase-characters>
                <pas:min-non-alphanumeric-characters>2</pas:min-non-alphanumeric-characters>
            </sec:password-validator>
            <sec:deployable-provider-synchronization-enabled>false</sec:deployable-provider-synchronization-enabled>
        </realm>
        <default-realm>myrealm</default-realm>
        <credential-encrypted>{AES}cpwhv2peqwl0Z8Ma1Jk7KlmC6d1gtCAqxnzrmR5ssybHpM9AUzbBRJ7rE7m4/7U16IHgsTMkcOr/5/abZJ4NeuHR45jdDK1EjFmnUuPssISmGkan3EIpCKHmPPIynVxh</credential-encrypted>
        <node-manager-username>wdUXiuwL5A</node-manager-username>
        <node-manager-password-encrypted>{AES}UDiZjaLOrm0y+/I9noQVURudpzmRKMBOQR2B8ofNIt8=</node-manager-password-encrypted>
    </security-configuration>
    <server>
        <name>AdminServer</name>
        <max-http-message-size>10000000</max-http-message-size>
        <ssl>
            <name>AdminServer</name>
            <enabled>true</enabled>
            <hostname-verifier xsi:nil="true"></hostname-verifier>
            <hostname-verification-ignored>true</hostname-verification-ignored>
            <client-certificate-enforced>false</client-certificate-enforced>
            <listen-port>8443</listen-port>
            <two-way-ssl-enabled>true</two-way-ssl-enabled>
            <server-private-key-alias>localhost.localdomain</server-private-key-alias>
            <server-private-key-pass-phrase-encrypted>{AES}eZnL/NM4/xJaFj8rZAE4oB2htDm3srPUsTWYtDXblvk=</server-private-key-pass-phrase-encrypted>
            <ssl-rejection-logging-enabled>true</ssl-rejection-logging-enabled>
            <allow-unencrypted-null-cipher>false</allow-unencrypted-null-cipher>
            <use-server-certs>true</use-server-certs>
            <jsse-enabled>true</jsse-enabled>
        </ssl>
        <listen-port-enabled>false</listen-port-enabled>
        <web-server>
            <keep-alive-secs>60</keep-alive-secs>
            <https-keep-alive-secs>60</https-keep-alive-secs>
        </web-server>
        <listen-address/>
        <java-compiler>javac</java-compiler>
        <tunneling-enabled>true</tunneling-enabled>
        <tunneling-client-ping-secs>45</tunneling-client-ping-secs>
        <tunneling-client-timeout-secs>40</tunneling-client-timeout-secs>
        <client-cert-proxy-enabled>false</client-cert-proxy-enabled>
        <key-stores>CustomIdentityAndCustomTrust</key-stores>
        <custom-identity-key-store-file-name>certstores/keystore</custom-identity-key-store-file-name>
        <custom-identity-key-store-type>JKS</custom-identity-key-store-type>
        <custom-identity-key-store-pass-phrase-encrypted>{AES}eZnL/NM4/xJaFj8rZAE4oB2htDm3srPUsTWYtDXblvk=</custom-identity-key-store-pass-phrase-encrypted>
        <custom-trust-key-store-file-name>certstores/truststore</custom-trust-key-store-file-name>
        <custom-trust-key-store-type>JKS</custom-trust-key-store-type>
        <custom-trust-key-store-pass-phrase-encrypted>{AES}uSkLzfC74bMCEjomgVdHtVw47iZg8BN71g7cdKr+XHo=</custom-trust-key-store-pass-phrase-encrypted>
    <overload-protection>
      <panic-action>system-exit</panic-action>
      <failure-action>force-shutdown</failure-action>
      <server-failure-trigger>
        <max-stuck-thread-time>600</max-stuck-thread-time>
        <stuck-thread-count>3</stuck-thread-count>
      </server-failure-trigger>
    </overload-protection>
    </server>
    <server>
      <name>MyManagedServer1</name>
      <ssl>
        <enabled>true</enabled>
        <listen-port>8445</listen-port>
        <server-private-key-alias>localhost.localdomain</server-private-key-alias>
        <server-private-key-pass-phrase-encrypted>{AES}eZnL/NM4/xJaFj8rZAE4oB2htDm3srPUsTWYtDXblvk=</server-private-key-pass-phrase-encrypted>
      </ssl>
      <machine xsi:nil="true"></machine>
      <listen-port>8444</listen-port>
      <listen-port-enabled>true</listen-port-enabled>
      <cluster xsi:nil="true"></cluster>
      <web-server>
        <web-server-log>
          <number-of-files-limited>false</number-of-files-limited>
        </web-server-log>
      </web-server>
      <listen-address></listen-address>
      <key-stores>CustomIdentityAndCustomTrust</key-stores>
      <custom-identity-key-store-file-name>certstores/keystore</custom-identity-key-store-file-name>
      <custom-identity-key-store-type>JKS</custom-identity-key-store-type>
      <custom-identity-key-store-pass-phrase-encrypted>{AES}eZnL/NM4/xJaFj8rZAE4oB2htDm3srPUsTWYtDXblvk=</custom-identity-key-store-pass-phrase-encrypted>
      <custom-trust-key-store-file-name>certstores/truststore</custom-trust-key-store-file-name>
      <custom-trust-key-store-type>JKS</custom-trust-key-store-type>
      <custom-trust-key-store-pass-phrase-encrypted>{AES}uSkLzfC74bMCEjomgVdHtVw47iZg8BN71g7cdKr+XHo=</custom-trust-key-store-pass-phrase-encrypted>
      <data-source>
        <rmi-jdbc-security xsi:nil="true"></rmi-jdbc-security>
      </data-source>
    </server>
    <production-mode-enabled>false</production-mode-enabled>
    <embedded-ldap>
        <name>JCRMDomain</name>
        <credential-encrypted>{AES}LNAnq4qCQQmy9rCOMN+uIZVPq1bVvPrwQqqtOScZrnYmwESt9aruHNVy4IjqMWz7</credential-encrypted>
    </embedded-ldap>
    <administration-port-enabled>true</administration-port-enabled>
    <configuration-version>10.3.4.0</configuration-version>
    <admin-server-name>AdminServer</admin-server-name>
</domain>
It has 'CustomIdentityAndCustomTrust' for both Admin Server as well as managed Server in it.
Thanks,
nn