Deciphering Cron log to try to determin system date change issue

Question

Solaris 9 box, Sparc Sun V1280.
We had two events the past day or so where the system time of a server changed 3 hours forward and then changed back 3 hours later that day. We manually corrected the time after the first shift was noticed around 4pm.
Each time the time shifts took place, there were no logged in users and no root users at all during the shift. So, the assumption is that a script with root access did it.
However, I have yet to find any scripts that would do that yet.
The one place that I can see when the time changed is in the cron log.
It is a bit confusing looking at the log because I'm not exactly sure what I am seeing.
You can see when a command takes place with the CMD: entry, but what follows is not clear if it actually belongs to that command or is possibly from an entry above that was still running... It doesn't help that several of the entries take place at the same times, typically 00, 15,30 and 45 minutes after the hour.
I've changed that now, but only have the logs of when it happened to investigate with.
Of course, all parties involved say that they changed nothing that would cause this :-)
Below is a number of entries surrounding the inital time shift where the servers time moved forward 3 hours...
. > CMD: /usr/lib/sa/sa1
. > sys 20943 c Tue Oct 27 11:00:00 2009
. > CMD: /app/oraprod/admin/dba/scripts/check_concurrent_mgr.sh
. > oraprod 20944 c Tue Oct 27 11:00:00 2009
. > CMD: /usr/local/system/TOSCO/system-monitoring/filesysck.sh > /dev/null 2>&1
. > root 20945 c Tue Oct 27 11:00:00 2009
. < root 20945 c Tue Oct 27 11:00:21 2009
. < oraprod 20942 c Tue Oct 27 11:00:00 2009 rc=1
. < prodmgr 20941 c Tue Oct 27 11:00:00 2009
. < sys 20943 c Tue Oct 27 11:00:00 2009
. < oraprod 20944 c Tue Oct 27 11:00:01 2009
. > CMD: /usr/lib/sa/sa1
. > sys 29747 c Tue Oct 27 11:15:00 2009
. > CMD: /app/oraprod/admin/dba/scripts/check_concurrent_mgr.sh
. > oraprod 29748 c Tue Oct 27 11:15:00 2009
. > CMD: /usr/local/system/TOSCO/system-monitoring/filesysck.sh > /dev/null 2>&1
. > root 29749 c Tue Oct 27 11:15:00 2009
. < sys 29747 c Tue Oct 27 11:15:01 2009
. < oraprod 29748 c Tue Oct 27 11:15:01 2009
. < root 29749 c Tue Oct 27 *11:15:14* 2009
Note the time shift after this point...
. > CMD: /app/oraprod/admin/dba/scripts/check_workflow_mailer.sh
. > oraprod 3809 c Tue Oct 27 *15:06:32* 2009
. < oraprod 3809 c Tue Oct 27 15:06:32 2009 rc=1
. > CMD: /usr/local/system/TOSCO/system-monitoring/filesysck.sh > /dev/null 2>&1
. > root 4491 c Tue Oct 27 15:15:00 2009
. > CMD: /usr/lib/sa/sa1
. > sys 4492 c Tue Oct 27 15:15:00 2009
So, I'm trying to see if it is possible to know which (if any) entries might be the cause of the time shift. The entries appear to be mixed between the commands (CMD: lines) making it hard to know which actually belongs to which command.
I have actually looked at every script or function listed and don't see anything in them to cause this however.
This is a Solaris 9 box, Sparc Sun V1280.
It has been basically static for years and has 2 related servers that have not been effected. There are also three 3 identical servers used in test that are also uneffected. Each has ntpd which keeps them in sync except of course when the one box leaps ahead 3 hours or back 3 hours.
The time shifts happened both on the same day, once around 11:15am and again about 5:15pm
Server was rebooted the next day and has not done this since.

Answer 1

krelvinaz wrote:
Solaris 9 box, Sparc Sun V1280.
We had two events the past day or so where the system time of a server changed 3 hours forward and then changed back 3 hours later that day. We manually corrected the time after the first shift was noticed around 4pm.I don't understand. Did the time shift back on it own after 3 hours, or did you correct the time after you noticed the shift.
>
Each time the time shifts took place, there were no logged in users and no root users at all during the shift. So, the assumption is that a script with root access did it. That is possible. It is also possible that this is a hardware error. The clock on the motherboard can fail in ways that drive the clock to different points in time. It also gives you a way to test.
If you can afford the trouble it would cause, can you change the time to just before the first shift and see if it shifts on its own?
>
However, I have yet to find any scripts that would do that yet.
The one place that I can see when the time changed is in the cron log.
It is a bit confusing looking at the log because I'm not exactly sure what I am seeing.
You can see when a command takes place with the CMD: entry, but what follows is not clear if it actually belongs to that command or is possibly from an entry above that was still running... It doesn't help that several of the entries take place at the same times, typically 00, 15,30 and 45 minutes after the hour.Each job launch starts with two lines (that are very difficult to display on this forum, even in tags)
{code}> CMD: /this/is/the/job{code}
{code}>  <user> <PID> c <start time>{code}
There will be a matching line with a less-than sign and the same PID to mark when the job completes.
If it is the hardware clock, my experience tells me that this will get worse until the motherboard fails completely.   Hopefully this is a script and you can find it.
If nothing else, you might consider writing a stupid job monitor.  Just something in perl that will sleep 1 second, then check the time.  If it notices a large jump in time, it can do a quick 'ps' and write down the time and what's running.
Darren
Edited by: Darren_Dunham on Oct 29, 2009 1:57 PM