Service not registered with listener on one of the nodes?

Question

We have an 11.2.0.1 oracle RAC with 2 nodes. Recently the listener died on node 1 and after a manual startup it seems like the service doesn't register with the listener on this node.
Here's what statusing the listener on both nodes shows:
[oracle@db01]$ $GRID_HOME/bin/lsnrctl status
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 17-NOV-2010 12:27:49
Copyright (c) 1991, 2009, Oracle. All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date 17-NOV-2010 07:14:40
Uptime 0 days 5 hr. 13 min. 9 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/app/11.2.0/grid/network/admin/listener.ora
Listener Log File /u01/app/oracle/diag/tnslsnr/db01/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.128.4)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.128.14)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM1", status READY, has 1 handler(s) for this service...
The command completed successfully
[oracle@db02 ~]$ $GRID_HOME/bin/lsnrctl status listener
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 17-NOV-2010 12:30:08
Copyright (c) 1991, 2009, Oracle. All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date 17-NOV-2010 07:14:40
Uptime 0 days 5 hr. 15 min. 28 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/app/11.2.0/grid/network/admin/listener.ora
Listener Log File /u01/app/oracle/diag/tnslsnr/db02/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.128.5)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.128.15)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "prd1.somehost.net" has 1 instance(s).
Instance "prd12", status READY, has 1 handler(s) for this service...
Service "prd1XDB.somehost.net" has 1 instance(s).
Instance "prd12", status READY, has 1 handler(s) for this service...
The command completed successfully
The prd12 instance shows as registered with the listener on db02 but the prd11 doesn't on db01. Right now all connections are routed to the prd12 instance. Also trying to connect to prd11 manually returns "ORA-12505: TNS:listener does not currently know of SID given in connect descriptor". The prd11 instance is actually running and I can connect to it locally via 'sqlplis "/ as sysdba"' but not via 'sqlplus user/pass@prd1'. I'm wondering what went wrong and how to register the prd11 with the listener on db01.
Here's some more information
[oracle@db01 ~]$ $GRID_HOME/bin/srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node db02
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node db01
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node db01
[oracle@db01 ~]$ $GRID_HOME/bin/srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node db02
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node db01
SCAN VIP scan3 is enabled
SCAN VIP scan3 is running on node db01
[oracle@db01 ~]$ $GRID_HOME/bin/srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): db01,db02
Thanks in advance,
Nick
Edited by: user639426 on Nov 17, 2010 4:44 AM

Answer 1

Seems like the LISTENER_SCAN2 and LISTENER_SCAN3 are running on node 1 but still I can't status them
[oracle@db01 ~]$ ps -ef | grep tns
oracle 6227 31486 0 08:24 pts/1 00:00:00 grep tns
oracle 18049 1 0 Nov17 ? 00:00:20 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
oracle 18120 1 0 Nov17 ? 00:00:16 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit
oracle 18131 1 0 Nov17 ? 00:00:05 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit
[oracle@db01 ~]$ $GRID_HOME/bin/lsnrctl status LISTENER_SCAN2
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 19-NOV-2010 08:25:11
Copyright (c) 1991, 2009, Oracle. All rights reserved.
TNS-01101: Could not find service name LISTENER_SCAN2
[oracle@db01 ~]$ $GRID_HOME/bin/lsnrctl status LISTENER_SCAN3
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 19-NOV-2010 08:25:18
Copyright (c) 1991, 2009, Oracle. All rights reserved.
TNS-01101: Could not find service name LISTENER_SCAN3
/u01/app/11.2.0/grid/network/admin/listener.ora contents:
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))) # line added by Agent
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))) # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON # line added by Agent
/u01/app/oracle/product/11.2.0/dbhome_1/network/admin/tnsnames.ora contents:
PRD1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = prdcluster-scan)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = prd1.somehost.net)
Everything worked just fine for quite some time until I found the listener down one morning and started it up manually. I wonder if I didn't start up the processes correctly. I did:
$GRID_HOME/bin/srvctl stop scan_listener
$GRID_HOME/bin/srvctl stop scan
$GRID_HOME/bin/srvctl stop listener
$GRID_HOME/bin/srvctl start listener
$GRID_HOME/bin/srvctl start scan
$GRID_HOME/bin/srvctl start scan_listener
Is this the correct way of starting up the listener processes?
Nick