OMS doesn't start - can't connect to database

Question

Hi all,
My dev/test environment won't start OMS anymore. This really has us stopped in our tracks...
It will start the WebTier, but OMS won't come up:
[oracle@ora-emcc log]$ $OMS_HOME/bin/emctl status oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is not functioning because of the following reason:
Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up.
Trying to check oms details results in an error as well:
[oracle@ora-emcc config]$ $OMS_HOME/bin/emctl status oms -details -sysman_pwd $SYSMAN_PWD
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Exception in thread "main" java.util.MissingResourceException: Can't find bundle for base name oracle.security.jps.internal.common.resources.keystore.KeyStoreResources, locale
        at java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1427)
        at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1250)
        at java.util.ResourceBundle.getBundle(ResourceBundle.java:952)
        at oracle.security.jps.util.JpsBundle.<init>(JpsBundle.java:70)
        at oracle.security.jps.util.JpsBundle.getResourceBundle(JpsBundle.java:151)
        at oracle.security.jps.util.JpsBundle.getResourceBundle(JpsBundle.java:106)
        at oracle.security.jps.JpsLocalizableException.getResourceBundle(JpsLocalizableException.java:157)
        at oracle.security.jps.JpsLocalizableException.getLocalizedMessage(JpsLocalizableException.java:140)
        at oracle.security.jps.JpsLocalizableException.getLocalizedMessage(JpsLocalizableException.java:131)
        at java.lang.Throwable.toString(Throwable.java:343)
        at java.lang.String.valueOf(String.java:2826)
        at java.lang.StringBuilder.append(StringBuilder.java:115)
        at java.lang.Throwable.printStackTraceAsCause(Throwable.java:536)
        at java.lang.Throwable.printStackTraceAsCause(Throwable.java:545)
        at java.lang.Throwable.printStackTraceAsCause(Throwable.java:545)
        at java.lang.Throwable.printStackTrace(Throwable.java:516)
        at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:59)
        at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:342)
        at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:304)
        at org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:234)
        at org.apache.log4j.WriterAppender.append(WriterAppender.java:159)
        at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
        at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
        at org.apache.log4j.Category.callAppenders(Category.java:203)
        at org.apache.log4j.Category.forcedLog(Category.java:388)
        at org.apache.log4j.Category.error(Category.java:319)
        at oracle.sysman.emctl.oms.PrintOMSSetupInfo.main(PrintOMSSetupInfo.java:240)
In the WebLogic Admin Console, I can see that BIP, EMGC_ADMINSERVER and EMGC_OMS1 are all in state Running and health Ok.
Our OMS and OMR servers are (and have always been) 2 different machines. The database on OMR has always been (and is currently) up and available.
Checking that I can connect to the repository database on OMR from the OMS server:
[oracle@ora-emcc ~]$ sqlplus sysman/$SYSMAN_PWD@ora-emccdb:1521/emccdb
SQL*Plus: Release 11.1.0.7.0 - Production on Thu Oct 10 23:34:53 2013
Copyright (c) 1982, 2008, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL>
Any suggestions will be much appreciated.
Ed

Answer 1

Hi all,
Many thanks for responding. I have opened an SR with Oracle 24 days ago. However, I just noticed I haven't heard from them for 13 days!
Must ping them to see what's happening...
I will check out your suggestions. The db and the listener are all fine and up anyway.
After looking in a lot of logfiles (and uploading them to the SR), it looks as if it could have to do with credentials in a wallet:
[oracle@ora-emcc log]$ cat secure.log
2013-10-15 20:07:36,265 [main] INFO oms.ListReposDetails logp.251 - Listing repository details from CredStore
2013-10-15 20:07:36,269 [main] INFO oms.AdminCredsWalletUtil setInstanceHome.177 - Getting credentials from wallet
2013-10-15 20:07:36,728 [main] INFO oms.AdminCredsWalletUtil setInstanceHome.192 - Read the credentials from wallet
2013-10-15 20:07:36,728 [main] INFO util.EmctlUtil logp.251 - Connecting over t3s to: ora-emcc.localdomain/7102 using id: weblogic
2013-10-15 20:07:38,639 [main] INFO mas.CredStoreUtil logp.251 - getCredential : Got creds for mapName -EM keyName - REPOS_DETAILS
2013-10-15 20:07:38,639 [main] INFO oms.ListReposDetails logp.251 - Exception thrown when getting repos details from CredStore: javax.management.RuntimeMBeanException: oracle.security.jps.service.keystore.KeyStoreServiceException: JPS-06516: Failed to get credential store. Reason oracle.security.jps.service.credstore.CredStoreException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOException: PKI-02002: Unable to open the wallet. Check password.
I will ping Oracle Support again (it's now with the security team) and see what the feedback is.
Regards,
Ed