Q: How many Users do you serve w. your OES11-Infrastructure?

Question

Reason for this - at first glance - stupid- question is the following:
We migrated (or Novell forced us to) from our low power (two HL DL360 G3 and 2 DL360 g4 one G5) 6.5 Cluster to a very juicy (6xDL360 G8, tons of proccessors) OES 11 cluster. We serve
approx. 1600 - 1700 Clients. Today everything stopped some of the Cluster Nodes didn't respond. This never happend in our old Cluster!
The reason was or IS by "Enterprise" Design:
DMESG: ipv4: Neighbour table overflow. !!!! Are we the only ones in the (small Novell) World who serves more than 1000 Clients???? WHY is there a limitation in the IP Cluster stack???
Did one EVER tested that. Should we better migrate to M$???
ip neigh show | wc -l --> shows 1021 entrys
sysctl net.ipv4.neigh.default.gc_thresh3
net.ipv4.neigh.default.gc_thresh3 = 1024
Means one cluster Node can handle 1024 connections. What if on one Node runs the DNS server? Only 1020 of 1600 Clients get an answer! Cool enterprise solution boys!
VERY angry!

Answer 1

On 03/25/2014 04:56 AM, jottschi wrote:
>
> Reason for this - at first glance - stupid- question is the following:
> We migrated (or Novell forced us to) from our low power (two HL DL360 G3
> and 2 DL360 g4 one G5) 6.5 Cluster to a very juicy (6xDL360 G8, tons of
> proccessors) OES 11 cluster. We serve
> approx. 1600 - 1700 Clients. Today everything stopped some of the
> Cluster Nodes didn't respond. This never happend in our old Cluster!
> The reason was or IS by "Enterprise" Design:
> DMESG: ipv4: Neighbour table overflow. !!!! Are we the only ones in the
> (small Novell) World who serves more than 1000 Clients???? WHY is there
> a limitation in the IP Cluster stack???
I'm sorry to hear about the lack of availability of the cluster nodes. I
am sure it is frustrating to have a system configured to be
highly-reliable (via clustering) to fail due to default settings. With
that said, this is a default Linux setting, and you're welcome to change it.
> Did one EVER tested that. Should we better migrate to M$???
I do not understand what you're asking.... did anybody test this on large
broadcast domains?
> ip neigh show | wc -l --> shows 1021 entrys
Wow, well that's quite a few. I think this number indicates something you
do not realize, though. While something like 'ss' or 'netstat' will tell
you how many layer three (IP) or four (TCP/UDP) connections your system
has, that number is not reflected by 'ip neigh', so usually the number of
clients your system has is limited in the list of neighbors by quite a
bit. For example, if I make a connection on my laptop (openSUSE, but same
default for my neighboring table) to Google, I can see that I have a fair
number of network connections total as shown:
Code:
me@mybox:~/Desktop> /usr/sbin/ss -planeto | grep -c ESTAB
61
but that's only tangentially related to the number of neighbors my laptop
sees:
Code:
me@mybox:~/Desktop> ip neigh
192.168.1.20 dev eth0 lladdr 00:1d:09:03:54:02 STALE
192.168.1.254 dev eth0 lladdr 00:1e:2a:74:66:35 STALE
192.168.1.1 dev eth0 lladdr f8:8f:ca:40:7a:1c REACHABLE
192.168.255.50 dev br8 lladdr ac:d3:58:ae:8e:ac STALE
How can that be? I have at least eight connections to unique remote
machines, so shouldn't I have at least eight listings in 'ip neigh'
output? The answer, of course, is 'no' because what you see from 'ip
neigh' are the cached addresses found via ARP, and ARP only matters within
the current broadcast domain, meaning on your local network before you hit
any routers. Even though I'm connected out to eight machines, all my
local routing table needs to care about (despite being VPN'd to a few
places and therefore having a few extra networks considered semi-local)
are the server, router, another router, and printer. Also, this number,
as you can see above, has entries become stale pretty quickly. I think
that time period is, again by default, something like thirty seconds,
since there is no guarantee from one second to the next that a given
server, workstation, or other node on the local network will still be
there. After something is stale the system just uses ARP again to see
what's still out there.
> sysctl net.ipv4.neigh.default.gc_thresh3
> net.ipv4.neigh.default.gc_thresh3 = 1024
Yes, this is a default for the Linux kernel in general. You're welcome to
tune it using sysctl, or probably via Yast so that it is stored to be used
after rebooting (and for the nice simple UI, if you're into that kind of
thing).
> Means one cluster Node can handle 1024 connections. What if on one Node
> runs the DNS server? Only 1020 of 1600 Clients get an answer! Cool
> enterprise solution boys!
No, it means that a given box, by default, can handle 1024 connections
from the local network. A typical Class C network, for example, only has
254 possible nodes, so only 253 connections. I've seen enterprises do
some supernetting to get things like 1022 nodes possible in a network, and
sometimes even fill that, but that's it. I have been on networks that had
addresses handed out from a Class B-like range (172.16.x.x) or even a
class A range (10.x.x.x) but those networks never had 65k or 10M boxes on
them because when you get too far beyond a thousand nodes you typically
suffer a bit of slowness. In your case you not only have more than 1024
nodes, but all of them are talking to the same server at the same time on
that same network.
Can Linux handle more? Sure, you're dealing with the same OS that runs
most supercomputers in the world, many of them being large clusters of
computers all working together in a single network, so obviously the
potential is there; however, there are reasons that one does not allow the
kernel to chew up all memory for a setting like "how many neighbors do you
have cached right now" when, for 99.9999% of the world, that max number
is around 200 since this specifically deals with neighbors on the same
segment or broadcast domain, not neighbors meaning anything reachable via
IP. Doing otherwise, for example setting it to 1,000,000, means that
somebody who was clever could more-easily waste your system's memory via a
denial of service (DoS) attack by flooding the network with bogus ARP
entries. Every one takes some memory, and every one needs to be managed
by the kernel until it is expired and removed, and again those networks
needing more than 1024 to be remembered within any single thirty-second
period are pretty rare. Your network must be pretty awesome to handle
that number of concurrent systems in the same logical network without
degradation.
Regarding running a DNS server, that is a good possibility as a service
that would experience this problem assuming all of the clients were on the
same network as the DNS server itself. While possible, the number of
networks where I've seen that has been pretty small overall since usually
those machines (servers) end up in a DMZ somewhere to prevent
unauthorized, or at least unaudited, access.
> VERY angry!
You're right in that this limitation should be documented, and made quite
obvious wherever it is documented. The migration guide would make sense,
as that would be a place where a lot of assumptions about defaults would
be questioned due to the OS change. If you have a specific page in the
documentation where you read through and expected limitations to be called
out, please either submit feedback, perhaps linking to this thread to
provide the background that you experienced, or post a link here an I'll
do the same.
If there are any other defaults that you feel should be changed,
particularly for your environment, sharing those may help as well.
In the meantime, you can change your current settings by running the
following as 'root' if you have not done so already:
Code:
cp /etc/sysctl.conf /etc/sysctl.conf-`date +%s`
cat << EOM >> /etc/sysctl.conf
# Setup higher threshold for arp
net.ipv4.neigh.default.gc_thresh3 = 4096
net.ipv4.neigh.default.gc_thresh2 = 2048
net.ipv4.neigh.default.gc_thresh1 = 1024
EOM
To apply the changes either run 'sysctl -p' (from memory I think that's
the right command) or else reboot the node, then perform again on other nodes.
Good luck.
If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...