Database restarting daily, why?

Question

My RDBMS is Enterprise Edition Release 11.2.0.3.0 - 64bit Production running on RHEL 6 x86_64 running on a virtual server. Each morning at 05:58 the database restarts. Looking at the various trace files generated at that time I see messages as follows -
CELL communication is configured to use 0 interface(s):
kcrfwy: minimum sleep is 9974 usecs (overhead is 8974 usecs)
kgfmIsAppliance=0 (due to init)
There is a vktm trace file:
VKTM running at (1)millisec precision with DBRM quantum (100)ms
[Start] HighResTick = 1383573491638525
kstmrmtickcnt = 0 : ksudbrmseccnt[0] = 1383573491
kwqmnich: current time:: 13: 58: 21: 0
kwqmnich: instance no 0 repartition flag 1
kwqmnich: initialized job cache structure
*** 2013-11-04 05:58:23.677
kwqinfy: Call kwqrNondurSubInstTsk
PQQ: Skipping service checks
kgsksysstop: blocking mode (2) timestamp: 1383573500394276
kgsksysstop: successful
kgsksysresume: successful
*** 2013-11-04 05:59:14.924
PQQ: Active Services changed
PQQ: Old service table
SvcIdx  SvcId Active ActDop
PQQ: New service table
SvcIdx  SvcId Active ActDop
     1      1      1      0
     2      2      1      0
     3      3      1      0
     4      4      1      0
Control file enqueue hold time tracking dump at time: 1470029
  1: 1140ms (rw) file: kct.c line: 2522 count: 140733193388033 total: 1140ms time: 1469406
The alert log shows the shutdown/startup and all looks normal. The audit trail file shows a connection as sysdba at this time (that is to be expected I suppose).
The vktm trace file indicates to me that the problem may be due to the virtual keeper of the time and I wondered if it is perhaps getting out of sync with the time clock on the virtual server. I spoke with one of the system administrators who suggested some checks. One is that the entry tinker panic 0 be in the /etc/ntp.conf file but I don't find that entry. The VM tools appear to be in place -
-bash-3.2$ grep "buildNr =" /usr/bin/vmware-config-tools.pl
$buildNr = '8.6.11 build-1180212';
-bash-3.2$ /sbin/lsmod | grep vm
vmmemctl 46424  0
vmci            83804  1 vsock
vmxnet                55556  0
vmxnet3               88544  0
At this point I am stuck and not sure what else to look at. I have done a bit of searching but have not found anything which seems to fit my situation. If anyone has encountered this or has any suggestions I would be most grateful.
Thank you.
Bill Wagman

Answer 1

I understand your desire to 'plan' and why that might normally make sense but if planning is needed before your org can change their most important passwords then you have a large, gaping security hole.
One of the key tasks that ANY organization must be able to perform AT A MOMENTS NOTICE is changing key passwords. Implicit in that task is the requisite documentation that shows the EXACT steps the need to be taken and clearly identifies ALL possible impacts (i.e. other password changes needed, batch process impacts, etc).
The IT manager, or lead DBA, should be able to put their hands on security documentation that identifies that process and the steps to implement it.
As part of the process of resolving your current issue I suggest you broach that LARGER deficiency to your management and make immediate plans to address it.