ESSSVR.exe crashes after service start

Question

Greeting everyone,
We have a dedicated Essbase server. Today we started getting a TCP/IP protocol timeout error when trying to connect to EAS.
The first thing we noticed was the "OPMN service for Essbase" was actually running --this is the only Essbase service on this server.
At first we tried to restart the service, but the problem persisted. Then we tried to restart the server, but this didn't help neither.
Finally, we noticed that when you launch the service start (by using the oracle provided .bat files), while the service is starting, an ESSSVR.exe process shows up in the task manager, but once the service is started, the ESSSVER.exe process crashes and disappears from the task manager.
What can be wrong?
Thank you.
Regards,
G.Sanchez

Answer 1

Thank you, John.
Looking at the OPMN logs, I can see the following the problem started after a scheduled service restart. Prior to it and up to 11-20-2012 at 4:00 AM, everything was fine. When the service was automatically restarted at 4:40 AM after perfoming an automated back up (which is performed daily), the logs started to show errors and warnings. I pasted them here --maybe you can give me some ideas:
opmn.log (nothing odd appears to be here)
[2012-11-20T04:01:56][opmn][TRACE:1][668][OPMN]Request 3 Completed. Command: /shutdown
[2012-11-20T04:01:56][opmn][TRACE:1][][OPMN][code:pm-requests]Http 3
EPM_epmsystem3~Essbase1~EssbaseAgent~AGENT~161747104:6416
  Status: Stopped
  Operation: request (time out while waiting for a managed process to stop)
  ErrFile: C:\Oracle\Middleware\user_projects\epmsystem3\diagnostics\logs\OPMN\opmn\console~Essbase1~EssbaseAgent~AGENT~1.log
  String: second stop attempted
EPM_epmsystem3~Essbase1~EssbaseAgent~AGENT~161747104:6416
  Status: Stopped
  Operation: request (success)
  ErrFile: C:\Oracle\Middleware\user_projects\epmsystem3\diagnostics\logs\OPMN\opmn\console~Essbase1~EssbaseAgent~AGENT~1.log
[2012-11-20T04:01:56][opmn][TRACE:1][666][OPMN][code:pm-process]Process Stopped: Essbase1~EssbaseAgent~AGENT~1 (161747104:6416)
[2012-11-20T04:40:15][opmn][NOTIFICATION:1][90][OPMN][code:ons-internal]ONS server initiated
[2012-11-20T04:40:15][opmn][NOTIFICATION:1][522][OPMN][code:pm-internal]PM state directory exists: C:\Oracle\Middleware\user_projects\epmsystem3\config\OPMN\opmn\states
[2012-11-20T04:40:15][opmn][NOTIFICATION:1][675][OPMN][code:pm-internal]OPMN server ready.  Request handling enabled.
[2012-11-20T04:40:15][opmn][TRACE:1][667][OPMN][code:pm-requests]Request 2 Started.  Command: /start
[2012-11-20T04:40:15][opmn][TRACE:1][662][OPMN][code:pm-process]Starting Process: Essbase1~EssbaseAgent~AGENT~1 (161747105:0)
[2012-11-20T04:40:35][opmn][TRACE:1][662][OPMN][code:pm-process]Starting Process: Essbase1~EssbaseAgent~AGENT~1 (161747105:0)
[2012-11-20T04:40:46][opmn][TRACE:1][662][OPMN][code:pm-process]Starting Process: Essbase1~EssbaseAgent~AGENT~1 (161747105:0)
[2012-11-20T04:40:54][opmn][TRACE:1][668][OPMN][code:pm-requests]Request 2 Completed. Command: /start
*service.log*
12/11/20 04:00:48 shutdown
12/11/20 04:40:16 startproc
================================================================================
opmn id=DESHYPDB02.codere.com:6712
    0 of 1 processes started.
ias-instance id=EPM_epmsystem3
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ias-component/process-type/process-set:
    Essbase1/EssbaseAgent/AGENT/
Error
--> Process (index=1,uid=161747105,pid=6612)
    failed to start a managed process after the maximum retry limit
    Log:
    C:\Oracle\Middleware\user_projects\epmsystem3\diagnostics\logs\OPMN\opmn\console~Essbase1~EssbaseAgent~AGENT~1.log
*console~Essbase1~EssbaseAgentÃGENT~1.log* (I copy what seems to be the most relevant information)
Read Error:Fatal Error. C:\Oracle\Middleware\user_projects\epmsystem3\EssbaseServer\essbaseserver1\bin\*TEMPESSBASE.SEC is corrupt*
Fatal Error: Essbase did not shutdown cleanly. Please remove TEMPESSBASE.SEC and restart Essbase with the ESSBASE.SEC file. Some security changes could have been lost
According to this last error, shall I just remove (or back up) the TEMPESSBASE.SEC file, leave ESSBASE.SEC and try a restart?
Many thanks,
G.Sanchez