What to do when updatemanager won't install ....

Question

Forum Moderator et al:
I periodically see instances where updatemanager will hang when it is starting the "Install phase".
In other words, after I check for patches, select the patches to be installed, and it has downloaded the patches. It will often simply sit there at the final "Analyzing system" or at the "Installing updates" section .... but not actual be installing anything.
When it works properly, prstat shows that it is running a series of "pkginstall" and "pkgadd" commands ... but when it hangs, these are not running.
While I trust that this means that something it needs is no longer running, I can't seem to figure out what it is missing and, more often than not, my only way to recover is to reboot. In fact, when I try to hit "Cancel", it seems to sit forever at the "Canceling ..." window but never actually closes down updatemanager. So, I do a "ps -ef | grep update" and kill the processes that it lists (/usr/bin/updatemanager, the java process running swupdate.jar, and two "smpatch update" processes and then I also "rm /var/sadm/spool/sunucLog/job_history.log.lck".). But that normally doesn't fix anything so I have to reboot to "restore" updatemanager functionality. That's not a very pleasant thing to have to do on a production machine and always makes me cringe as being the Windows way of solving problems.
Is there a list of things like processes that need to be running, lock files that either should or should not be present, etc. Ideally, it would be nice to be able to run a little script named something like "um_check" that could check for all of these things.
Note: I'm confident that I've got all applicable patches installed and I have three identical machines ... periodically, however, one of them will show this "updatemanager hang" behavior and which machine of the three may hang never seems predictable.
What should I be looking for? /var/adm/messages rarely shows anything. Is there another log file that I can be checking?
Thanks,
John

Answer 1

Forum Moderator:
I just ran an experiment that, I think, provides some additional information. This morning, one of my machines thinks that it needs to install the following 4 patches (this is a Solaris 10u4 x86 system). They are:
127112-11 (kernel .... requiring immediate reboot)
119758-12 (Samba .... also requiring immediate reboot)
124873-04 (Sun Studio 12 stuff ...)
128345-01 (time.h)
If I select all 4 of these, it appears to download everything and then hangs at the "Installing ..." step. When I check the log that I generated with the debug flag, it appears to have determined that the order of the patches to be applied as: 128345-01, 119758-12, 124873-04, and 127112-11. And, as shown by the last few lines of the updatemanager debug file, it tried to start installing 128345-01 and just sat there.
Status: STATUS DOWNLOAD_PATCHDB BEGIN
Status: STATUS DOWNLOAD_DETECTORS BEGIN
Status: STATUS DOWNLOAD_DETECTORS END DOWNLOAD_DETECTORS.0 "Download of {0} was
successful." "detectors"
Status: STATUS DOWNLOAD_PATCHDB END DOWNLOAD_PATCHDB.0 "Download of {0} was successful." "current"
Status: STATUS ANALYZE BEGIN
Status: STATUS PATCH_SEQUENCER BEGIN 128345-01
Status: STATUS PATCH_SEQUENCER END 128345-01 PATCH_SEQUENCER.0 "Update {0} passed sequencer check." "128345-01"
Status: STATUS PATCH_SEQUENCER BEGIN 119758-12
Status: STATUS PATCH_SEQUENCER END 119758-12 PATCH_SEQUENCER.0 "Update {0} passed sequencer check." "119758-12"
Status: STATUS PATCH_SEQUENCER BEGIN 124873-04
Status: STATUS PATCH_SEQUENCER END 124873-04 PATCH_SEQUENCER.0 "Update {0} passed sequencer check." "124873-04"
Status: STATUS PATCH_SEQUENCER BEGIN 127112-11
Status: STATUS PATCH_SEQUENCER END 127112-11 PATCH_SEQUENCER.0 "Update {0} passed sequencer check." "127112-11"
Status: STATUS ANALYZE END ANALYZE.0 "Request to analyze the system was successful
ul."
Status: STATUS INSTALL BEGIN 128345-01
So, I killed the various update processes and did a 'rm /var/sadm/spool/sunucLog/job_history.log.lck
and started up updatemanger a second time with the debug flag.
This time I only selected on patch to be applied at a time .... but with no other changes.
Starting with 127112-11 (the Kernel) .... it tried to install it, realized that it was a "reboot immediate" and came back and reported that it would be installed at the next reboot and asked if I wanted to reboot now. I answered "No" and continued.
The next patch in the list was 119758-12 (Samba) and behaved the same way ... an immediate reboot patch that can only be installed at the next reboot and asked if I wanted to reboot now. I answered "No" and continued.
The next patch was 124873-04 (Some of the Sun Studio 12 debug tools, I think). Interestingly, this installed just fine. This seems to indicate that there is nothing intrinsically broken with my updatemanager setup and would also seem to indicate that the underling call to smpatch also succeeded.
Finally, I selected 128345-01 (time.h) from the list. As when I was trying to install all 4 patches simultaneously, this hung and the final line of the updatemanager debug log was:
Status: STATUS INSTALL BEGIN 128345-01
where it tried to start installing this patch ..... but nothing appeared to happen.
So, this would seem to imply that there are certain patches that cause updatemanager to hang. While I don't have a clue as to what is different about 128345-01 (that hung up) as compared to 124873-04 that installed successfully during the same updatemanager session .... it sure seems that there is something that is patch-specific, don't you think?
Now, I can anticipate your next question: "Does 128345-01 install properly when run explicitly from the command line as: 'smpatch add -i 128345-01'?
The answer is "Yes", here is the result of that command:
root@flare # smpatch add -i 128345-01
add patch 128345-01
Transition old-style patching.
Patch 128345-01 has been successfully installed.
So, while I'm still befuddled by this, here is what I think I have learned:
1. There is nothing intrinsically wrong with my updatemanager setup.
2. Certain patches seem to cause updatemanager to hang after it tries to install them and that patch will seem to hang where it is installed as a series of patches selected for install or if it is the only patch selected for installation in updatemanager.
3. That same patch, however, can be successfully be installed by manually executing 'smpatch add -i <patch-id>'.
I hope that provides some useful information that will help to unravel this.
Thanks,
John