OEL 56 and Oracle validated rpm server will not reboot.

Question

[root@stage-rac21 admin]# up2date -i oracle-validated
Fetching Obsoletes list for channel: ol5_x86_64_latest...
Fetching Obsoletes list for channel: el5_x86_64_oracle...
Fetching rpm headers...
Name Version Rel
oracle-validated 1.1.0 7.el5 x86
Testing package set / solving RPM inter-dependencies...
kernel-uek-2.6.32-100.28.9. ########################## Done.
kernel-uek-firmware-2.6.32- ########################## Done.
kexec-tools-1.102pre-126.0. ########################## Done.
oracle-validated-1.1.0-7.el ########################## Done.
ql2xxx-firmware-1.01.01-0.2 ########################## Done.
busybox-1.2.0-7.el5.x86_64. ########################## Done.
Preparing ########################################### [100%]
Installing...
1:busybox ########################################### [100
2:kexec-tools ########################################### [100
3:ql2xxx-firmware ########################################### [100
4:kernel-uek-firmware ########################################### [100
5:kernel-uek ########################################### [100
6:oracle-validated ########################################### [100
The following packages were added to your selection to satisfy dependencies:
Name Version Release
kernel-uek 2.6.32 100.28.9.el5
kernel-uek-firmware 2.6.32 100.28.9.el5
kexec-tools 1.102pre 126.0.1.el5_6.5
ql2xxx-firmware 1.01.01 0.2.el5
busybox 1.2.0 7.el5
Reboot and will not come up.
KVM shows:
mount: could not find filesystem '/dev/root'
setuproot: moving /dev failed: No such file or directory
no fstab.sys, mounting internal defaults
setuproot: error mounting /proc: No such file or directory
switchroot: mount failed : No such file or directory
Kernel Panic - not syncing: Attempted to kill init!
Pid: 1, comm: init Not tainted 2.6.32-100.28.9.el5 #1
Call Trace:
Is anyone else having trouble with 2.6.32-100.28.9.el5?

Answer 1

One of our sys admins has some input on this. Starting to make sense.
Looks like maybe this kernel is missing the device mapper info.
*.203 + *.205:
Now regarding the issue with this and the *.205 server post kernel
updates + reboot (kernel panic due to the system not being able to load
/), I think this was due to some needed device mapper modules being
missing from the initrd of the new kernel (see below for the details).
As per lenny, I've updated the grub configs on *.205 to load the old
kernel (default=1 in /boot/grub/grub.conf; the system was restored by
booting back to the previous kernel) for now:
[root@stage-rac22 ~]# mkdir /tmp/initrd_diff
[root@stage-rac22 ~]# cp -v /boot/initrd-2.6.32-100.28.9.el5.img
/boot/initrd-2.6.18-238.5.1.0.1.el5.img /tmp/initrd_diff/
`/boot/initrd-2.6.32-100.28.9.el5.img' ->
`/tmp/initrd_diff/initrd-2.6.32-100.28.9.el5.img'
`/boot/initrd-2.6.18-238.5.1.0.1.el5.img' ->
`/tmp/initrd_diff/initrd-2.6.18-238.5.1.0.1.el5.img'
[root@stage-rac22 ~]# chmod -v go+r /tmp/initrd_diff/*
mode of `/tmp/initrd_diff/initrd-2.6.18-238.5.1.0.1.el5.img' changed to
0644 (rw-r--r--)
mode of `/tmp/initrd_diff/initrd-2.6.32-100.28.9.el5.img' changed to
0644 (rw-r--r--)
[root@stage-rac22 ~]# chown -vR admin. /tmp/initrd_diff/
[root@stage-rac22 ~]# su - admin
[admin@stage-rac22 ~]$ cd /tmp/initrd_diff/
[admin@stage-rac22 initrd_diff]$ mkdir -v old new
mkdir: created directory `old'
mkdir: created directory `new'
[admin@stage-rac22 initrd_diff]$ (cd old/; zcat
../initrd-2.6.18-238.5.1.0.1.el5.img|cpio -idv)
[admin@stage-rac22 initrd_diff]$ (cd new/; zcat
/tmp/initrd-2.6.32-100.28.9.el5.img|cpio -idv)
[admin@stage-rac22 initrd_diff]$ diff -qr old/ new/
diff: old/dev/ram: No such file or directory
diff: new/dev/ram: No such file or directory
Files old/init and new/init differ
Files old/lib/cciss.ko and new/lib/cciss.ko differ
Only in old/lib: dm-log.ko
Only in old/lib: dm-mem-cache.ko
Only in old/lib: dm-message.ko
Only in old/lib: dm-mirror.ko
Only in old/lib: dm-mod.ko
Only in old/lib: dm-raid45.ko
Only in old/lib: dm-region_hash.ko
Only in old/lib: dm-snapshot.ko
Only in old/lib: dm-zero.ko
Files old/lib/ehci-hcd.ko and new/lib/ehci-hcd.ko differ
Only in old/lib: ext3.ko
Only in old/lib: jbd.ko
Files old/lib/ohci-hcd.ko and new/lib/ohci-hcd.ko differ
Only in old/lib: scsi_mod.ko
Only in old/lib: sd_mod.ko
Files old/lib/uhci-hcd.ko and new/lib/uhci-hcd.ko differ