Listener Load Balancing Becomes Unbalanced!!

Question

Hi all, Yesterday we observed that for no apparent reason, one instance in a RAC database appeared to be left out of the listener load balancing equation. This resulted in a very high load on the remaining instances. The only way we could repair this was to restart the un(der)utilized instance.
Before restart:
SQL> select count(*),inst_id from gv$session group by inst_id;
COUNT(*) INST_ID
216 7
41 12
220 9
After restart:
SQL> select count(*),inst_id from gv$session group by inst_id;
COUNT(*) INST_ID
222 7
218 12
224 9
Has anyone else seen this? Is there any way to tell what is going on? Is there any way to fix this without restarting the instance? We have tried restarting various listeners without success. Thanks in advance, Mike

Answer 1

Actually, we have the problem in effect on one of our databases right now. The ocr status for all the instances is good:
ora.S.S1.inst ONLINE ONLINE on x911
ora.S.S2.inst ONLINE ONLINE on x912
ora.S.S9.inst ONLINE ONLINE on x919
ora.S.db ONLINE ONLINE on x927
But when we look at the number of sessions per instance:
SQL> select inst_id,count(*) from gv$session group by inst_id;
INST_ID COUNT(*)
1 50
2 237
9 230
They are not balanced. We look at the load average on each host:
oracle@x912:~> ssh x911 uptime
3:44pm up 224 days 6:48, 15 users, load average: 1.07, 1.00, 0.99
oracle@x912:~> ssh x912 uptime
3:44pm up 223 days 11:46, 12 users, load average: 5.26, 5.18, 5.35
oracle@x912:~> ssh x919 uptime
3:44pm up 223 days 11:42, 10 users, load average: 3.64, 4.01, 6.47
x911 for instance S1 is the lowest (which we would expect with no connections)
But when we connect through SQL*PLUS:
oracle@x912:~> sqlplus dbamb@s
SQL*Plus: Release 10.2.0.2.0 - Production on Wed Dec 19 15:47:06 2007
Copyright (c) 1982, 2005, Oracle. All Rights Reserved.
Enter password:
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
SQL> select instance_name from v$instance;
INSTANCE_NAME
S2
We get connected to a higher utilized instance. When we connect to the instance by name:
SQL> conn dbamb@s1
Enter password:
Connected.
SQL> /
INSTANCE_NAME
S1
We get it. Isn't the RAC db connection is supposed to go to the host with the lowest load average? What can I do?
Thanks in advance, Mike