Multiple DB's and multiple listeners - sanity check

Question

In setting up a 3 node cluster to provide HA Oracle failover to a minimum of 4 databases for starters, just need a bit of confirmation on listeners...
4 seperate RG's for each DB.
Each instance has it's own LHN + IPdefined in /etc/hosts.
Each node has Oracle home on local disk. DB files on shared disk, each on their own HASP.
HASP has weak dependencie on local disk/Ora-homes and that all works fine.
I setup the listeners for each logical server to be dependent on the HASP as well.
So, 4 RGs, 4 listeners, 4 LHN's, etc. Seems pretty straightforward, and the oracle server resources all start up fine, however when failing a RG over to another node, it takes a real long time and eventually gets it done but the listeners on all the DB's restart before the failover completes. DBA left the listener.ora file as default so it does not have all the SID's of the DB's we have in the cluster. Does listener.ora need to be edited to reflect 4 seperate listeners or just SIDs? Wondering if there is an example of this somewhere, I've found references to RAC listener.ora files but that's a tad bit different.
thanks again,
Dave
Aug 19 11:03:02 SC[SUNWscor.oracle_listener.start]:db480t1-rg:db480t1_listener-rs: Starting listener LISTENER3.
Aug 19 11:03:02 SC[SUNWscor.oracle_listener.start]:db480t1-rg:db480t1_listener-rs: Using method 'run_setuid_prog' to execute start commands
LSNRCTL for Solaris: Version 10.2.0.2.0 - Production on 19-AUG-2008 11:03:02
Copyright (c) 1991, 2005, Oracle.  All rights reserved.
TNS-01106: Listener using listener name LISTENER1 has already been started
Executing command: /opt/SUNWscor/oracle_listener/bin/oracle_listener_manage start
Aug 19 11:03:06 SC[SUNWscor.oracle_listener.start]:db480t1-rg:db480t1_listener-rs: Failed to start listener LISTENER3.
Aug 19 11:04:12 SC[SUNWscor.oracle_listener.stop]:db480t1-rg:db480t1_listener-rs: listener LISTENER3 is not running.
Aug 19 11:06:08 SC[SUNWscor.oracle_listener.start]:db480t1-rg:db480t1_listener-rs: Starting listener LISTENER3.
Aug 19 11:06:08 SC[SUNWscor.oracle_listener.start]:db480t1-rg:db480t1_listener-rs: Using method 'run_setuid_prog' to execute start commands
LSNRCTL for Solaris: Version 10.2.0.2.0 - Production on 19-AUG-2008 11:06:08
Copyright (c) 1991, 2005, Oracle.  All rights reserved.
TNS-01106: Listener using listener name LISTENER1 has already been started
Executing command: /opt/SUNWscor/oracle_listener/bin/oracle_listener_manage start
Aug 19 11:06:12 SC[SUNWscor.oracle_listener.start]:db480t1-rg:db480t1_listener-rs: Failed to start listener LISTENER3.Edited by: 2wheel on Aug 19, 2008 9:13 AM
Edited the listener.ora to reflect the 4 seperate listeners, LISTENER1, 2, 3, 4. I edited the properties of LISTENER_NAME for each listener resource. Now if I enable one listener on one node, works fine. Enable 2nd listener (RG happens to be on different node) works fine. Try to enable 3rd or 4rth listeners on the same nodes and it pukes....

Answer 1

messages snippet from attempt to create a 3rd listener on a node where all the RG's are online, two listeners already running. Will not allow it:
Aug 20 07:07:41 devdsm01nd03 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_validate> for resource <db480t1_listener-rs>, resource group <db480t1-rg>, node <devdsm01nd03>, timeout <120> seconds
Aug 20 07:07:41 devdsm01nd03 Cluster.RGM.rgmd: [ID 285716 daemon.notice] 20 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_validate>:tag=<db480t1-rg.db480t1_listener-rs.2>: Calling security_clnt_connect(..., host=<devdsm01nd03>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Aug 20 07:07:42 devdsm01nd03 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_validate> completed successfully for resource <db480t1_listener-rs>, resource group <db480t1-rg>, node <devdsm01nd03>, time used: 0% of timeout <120 seconds>
Aug 20 07:07:42 devdsm01nd03 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_init> for resource <db480t1_listener-rs>, resource group <db480t1-rg>, node <devdsm01nd03>, timeout <30> seconds
Aug 20 07:07:42 devdsm01nd03 Cluster.RGM.rgmd: [ID 252072 daemon.notice] 50 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_init>:tag=<db480t1-rg.db480t1_listener-rs.4>: Calling security_clnt_connect(..., host=<devdsm01nd03>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Aug 20 07:07:42 devdsm01nd03 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_init> completed successfully for resource <db480t1_listener-rs>, resource group <db480t1-rg>, node <devdsm01nd03>, time used: 0% of timeout <30 seconds>
Aug 20 07:07:42 devdsm01nd03 Cluster.CCR: [ID 973933 daemon.notice] resource db480t1_listener-rs added.
Aug 20 07:07:43 devdsm01nd03 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_start> for resource <db480t1_listener-rs>, resource group <db480t1-rg>, node <devdsm01nd03>, timeout <300> seconds
Aug 20 07:07:43 devdsm01nd03 Cluster.RGM.rgmd: [ID 252072 daemon.notice] 50 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_start>:tag=<db480t1-rg.db480t1_listener-rs.0>: Calling security_clnt_connect(..., host=<devdsm01nd03>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Aug 20 07:07:48 devdsm01nd03 SC[SUNWscor.oracle_listener.start]:db480t1-rg:db480t1_listener-rs: [ID 847065 daemon.error] Failed to start listener LISTENER3.
Aug 20 07:07:48 devdsm01nd03 Cluster.RGM.rgmd: [ID 938318 daemon.error] Method <bin/oracle_listener_start> failed on resource <db480t1_listener-rs> in resource group <db480t1-rg> [exit code <1>, time used: 1% of timeout <300 seconds>]Listeners already running:
[devdsm01nd03:spgdas]: ps -ef |grep listener
    root    13   999   0 15:48:48 ?           0:17 /usr/bin/ksh /opt/SUNWscor/oracle_listener/bin/oracle_listener_monitor -R db480
    root   139   999   0 15:49:08 ?           0:17 /usr/bin/ksh /opt/SUNWscor/oracle_listener/bin/oracle_listener_monitor -R db480
[devdsm01nd03:spgdas]: ps -ef |grep LISTENER
  oracle 29953     1   0 15:48:44 ?           0:14 /oracle/app/product/10.2.0/bin/tnslsnr LISTENER -inheritMore from the listener logs:
Aug 20 07:07:44 SC[SUNWscor.oracle_listener.start]:db480t1-rg:db480t1_listener-rs: Starting listener LISTENER3.
Aug 20 07:07:44 SC[SUNWscor.oracle_listener.start]:db480t1-rg:db480t1_listener-rs: Using method 'run_setuid_prog' to execute start commands
LSNRCTL for Solaris: Version 10.2.0.2.0 - Production on 20-AUG-2008 07:07:44
Copyright (c) 1991, 2005, Oracle.  All rights reserved.
TNS-01106: Listener using listener name LISTENER has already been started
Executing command: /opt/SUNWscor/oracle_listener/bin/oracle_listener_manage start
Aug 20 07:07:48 SC[SUNWscor.oracle_listener.start]:db480t1-rg:db480t1_listener-rs: Failed to start listener LISTENER3.Trying to deduce this down to the lowest denominator. I agree, it's probably a configuration issue, so I removed all the server resources again to make the RGs a little leaner. When I do this (with each RG only having the HASP and LHN resources) I can move all 4 RG's to one node. That works fine. I add listeners one by one and that's when it derails.