OL5.11 device null's permissions

Question

Fresh OL5.11 installation, updated from Oracle's Public Yum. [required as as OL6 and OL7 do not support the older Mellanox Cougar HCA card in the servers]
After boot, /dev/null permissions are incorrectly set:
crw-rw---- 1 root root 1, 3 Jan 21  2015 /dev/null
udev rules file seems correct:
[root@xxxx ~]# grep null  /etc/udev/rules.d/50-udev.rules
KERNEL=="null",                 MODE="0666", OPTIONS="last_rule"
KERNEL=="null",                 SYMLINK+="XOR"
And rule is executed correctly according to this test:
[root@xxxx ~]# udevtest /class/mem/null
main: looking at device '/devices/virtual/mem/null' from subsystem 'mem'
udev_rules_get_name: no node name set, will use kernel name 'null'
udev_db_get_device: no db file to read /dev/.udev/db/devices@virtual@mem@null: No such file or directory
udev_node_add: creating device node '/dev/null', major = '1', minor = '3', mode = '0666', uid = '0', gid = '0'
So it would seem that something resets the permissions afterwards. But what and where?
The only "non-standard" service that runs, that a vanilla OL5.11 will not have, is openibd (Infiniband service, renamed rdma on later kernel versions). However it should not touch /dev/null. i have a couple of OL5.7 and OL5.4 servers too with openibd enabled, but without the permissions issue around /dev/null. That said, there are a number of changes I spotted between OL5.7 and OL5.11 ito service startup with the openibd service. [btw,  the Infiniband service loads fine and IPoIB works, and in bonded and connected modes too, on these OL5.11 servers]
Searching via google turned up a bunch of related but mostly useless info, some dating back to 2007. No, I do not want to hack this via rc.local. Yes, I do want to know where and why it is caused, and fixed the root problem.
RHEL5 has a support note logged for this exact issue - which contents are only visible to RHN users. Did not find a similar note on Oracle's My Support.
Any ideas, comments and suggestions on how to troubleshoot and fix this, will be much appreciated.

Answer 1

The following is the output of a default system installation from the OL 5.11 x64 iso image in VirtualBox.
# lsb_release -d
Description:    Enterprise Linux Enterprise Linux Server release 5.11 (Carthage)
# ls -l /dev/null
crw-rw-rw- 1 root root 1, 3 Jan 21  2015 /dev/null
/dev/null looks fine.
The /dev device tree is populated by udev at every system restart. The procedure that creates /dev/null is /sbin/start_udev script which reads the content of /etc/udev/makedev.d/50-udev.nodes, which contains "null".
Perhaps the following can help to determine whether the problem is caused by udev:
# rm -f /dev/null
# sh -x /sbin/start_udev
# ls -l /dev/null
crw-rw-rw- 1 root root 1, 3 Jan 21 21:45 /dev/null
The command that gets executed is:
/sbin/MAKEDEV -x /dev/null
If your /dev/null looks fine after that, than udev is most likely not the problem. If it does not, do you have any other udev.rules file that may overwrite 50-udev.rules. Can you check all udev rules?
# grep "null" /etc/udev/rules.d/*
/etc/udev/rules.d/50-udev.rules:KERNEL=="null",            MODE="0666", OPTIONS="last_rule"
/etc/udev/rules.d/50-udev.rules:KERNEL=="null",            SYMLINK+="XOR"
If it is not udev, I have no idea. But perhaps you can spot the problem by adding "set -vx" to /etc/rc, restart and watch the screen.