[SOLVED] Upgrade from SLES10 SP3 & OES2 SP2 to SP4/SP3 breaks NCS

Question

Hi folks,
(This post was originally meant to be a rant and a request for help, but
while writing the final paragraph i found the solution. It's still a
rant, but i figured i'd post my solution here in case someone else runs
into the same issue.)
I've just spent several hours banging my head against a broken cluster
node. My system is a 32-bit SLES 10 VM running on VMware ESX 3.5.x.
I upgraded from SLES10 SP3 and OES2 SP2 to the next service packs for
each (using the move-to-oes-sp3 script in yast2 online_update).
Everything went well for the first few update/reboot sequences, then
after the final reboot on SLES10 SP4 & OES2 SP3, cluster services would
not load or join the cluster on restart.
I checked dmesg and found errors about "Loading module compiled for
kernel version 2.6.16.60-0.54.5-vmi" into a previous kernel version, so
i tried downgrading to that kernel version, only to find that it was
older than the one i had just upgraded from (it's the original SLES10
SP3 kernel). So i tried upgrading back to the same kernel which is
running on the other cluster node (2.6.16.60-0.77.1-vmi), but that did
not work any better.
<preaching>
I have to say that i'm not impressed that OES2 SP3 isn't even compiled
against the appropriate kernel, and because of SUSE's kernel RPM
overwrite policy there's no way i can select to boot from a previous
kernel to see if that fixes things. Note to SUSE and other distro
builders: if you're not doing kernel package upgrades like Red Hat or
Ubuntu (so that we can select to boot from the previous kernel from the
boot menu), you're doing it *WRONG*.
</preaching>
I then upgraded again to the latest recommended kernel for SLES10 SP4,
and still no joy. Dmesg shows this error before the rot starts:
allocation failed: out of vmalloc space - use vmalloc=<size> to increase
size.
When searching for this error i stumbled across
http://ubuntuforums.org/showthread.php?t=1613132
which pointed me to
http://www.mythtv.org/wiki/Common_Pr...lloc_too_small
Adding vmalloc=192M to /boot/grub/menu.lst and rebooting solved the
problem for me.
Regards,
Paul

Answer 1

Originally Posted by Paul Gear
On 13/10/11 07:36, magic31 wrote:
> ...
> Not sure if it's related to the issue you've hit, but along the lines
> of what Kevin already mentioned, there are issues that can arries when
> using the VMI kernel (that I've seen) if having multiple flavors of the
> kernel installed along with it (as in having both kernel-vmi as also
> kernel-smp packages installed) ...
> Curious, is that also the case with your setup?
The system in question has kernel-bigsmp and kernel-vmi installed. We
only ever boot from kernel-vmi.
> I've moved to only using the smp kernel on VMware (along with the
> clock=pit boot option to avoid time drift issues, or pmtr I think in
> your case when also running NCS services in the vm).
When we installed the system (on OES2 SP1, i believe) it was a while
ago, and kernel-vmi with clocksource=acpi_pm on the kernel command line
was the only solution we could find to get reliable time. If there are
updated best-practice documents, i'd be happy to hear about them.
However, this is a production cluster and my boss is (rightly) rather
reticent to make major changes.
Paul
the official Vmware paper on time is that IF you're using SLES 10.x 32-bit you will use VMI and NO kernel params (ie, get rid of the clockpit and clocksource=blah).
But if 64-bit then you're okay.
The easiest, IMO (especially with vmware) is to use the miggui (migration utility). That's how I'm converting all my 32-bit servers in vmware to 64-bit. Works quite well.
--Kevin