Ora Fail Safe - under cluster ctl on one node but gives FS:10413  on other

Hi,
Trying to set up a single Fail Safe (3.3.3) instance (9.2) on a set of Windows 2003 servers running MSCS.
After creating the standalone DB on the first node, I was able to get it verified and added to the cluster control (while the second server was down - had to this as the error I'm about to mention kept rolling back the changes on both nodes, otherwise). Now that the first node is configured and the FS listener service and FS DB service are created, I tried setting the second node to match the first one (viz. getting listener.ora, sqlnet.ora and even tnsnames.ora).
The "verify group" operation kept failing on creation of the Windows FS service for the DB and listener.
Found out that there was an issue with picking up the Oracle_Home when even trying to invoke sqlplus manually from the command line (though the registry entries seemed to be right and match the ones set on the first node). Eventually on adding the ORACLE_HOME variable to the list of env variables, it started picking up the OH correctly and the next "verify group" operation successfully added the WIndows FS service for the DB (OracleService{SID}) on the second node. (The first node didn't have to encounter these problems as during the standalone DB install (using DBCA), this service was created).
Next, started the Windows FS DB service (on the second node with the problem) and tried starting the DB manually (out of cluster controll) and it worked fine.
However when trying to "verify group" again, I keep getting the error FS-10413 (and the subsequent FS:10046, FS:10111 and FS:10890) while trying to add the Windows FS listener service (OracleORA_HOMETNSListenerFsl{oradb_virtual_name}).
Could really do with extra eyes (and heads) helping to get to the bottom of this. Thanks.
<<< The gist of the output from the "verify group" action is below >>>
13 18:29:18 > FS-10300: Verifying Oracle Net listener resource OracleORA_HOME1TNSListenerFsloradbl
14 18:29:19 Oracle Net listener Fsloradbl for the group is not found
FS-10413: Error found in Oracle Net on node PANTHER; the group listener needs to be taken offline. Do you want it to be fixed? > Yes
15 18:29:19 >> FS-10600: Oracle Net configuration file updated: C:\ORACLE\PRODUCT\9.2\NETWORK\ADMIN\LISTENER.ORA
16 18:29:20 ** ERROR : FS-10066: Failed to start Windows service OracleORA_HOME1TNSListenerFsloradbl for the Oracle Net listener
<<<< >>>>>>>
Further to typing the above, read the section 7.10.8.2 in "Oracle Fail Safe Concepts and Admin guide" (http://download-uk.oracle.com/docs/html/B12070_01/ofs_db.htm#sthref985) and checked the output log. Bizarrely, the output file has a few "msg not found" errors (akin to when it doesn't find the correct OH).
Wonder where this has to be set in the registry!?!
The TNS-01151 (missing listener name in listener.ora) error is also a bit odd as this is the new listener OFS is trying to create and add to the new listener.ora file. Isn't it!?!?!
<<<< Fsloradbl.out >>>>>
LSNRCTL for 32-bit Windows: Version 9.2.0.7.0 - Production on 07-FEB-2007 21:50:46
Copyright (c) 1991, 2002, Oracle Corporation. All rights reserved.
Starting tnslsnr: please wait...
TNSLSNR for 32-bit Windows: Version 9.2.0.7.0 - Production
Message 279 not found; No message file for product=NETWORK, facility=NL
TNS-01151: Message 1151 not found; No message file for product=NETWORK, facility=TNS     [Fsloradbl]
Listener failed to start. See the error message(s) above...
<<<< >>>>
Thanks for your time (reading and hopefully responding to this) and any comments or suggestions.
Regards,
Ranjit

yes i did. i mean, the db starts corretly using the Enteprise Manager and listener.ora and tnsnames.ora are the ones created during db creation. they seem alright.
i'm trying to execute those commands from the node of the cluster where the db resides.
but... i have one big question: do i have to create the db instance on BOTH cluster nodes (both pointing to the same oradata in the shared disk) or is it enough to create my standalone db only on the nodeA and then let everything to the FailSafe management (adding the standalone db to the cluster group FailSafe creates all the services needed)???
thanks

Similar Messages

  • XML - ORA-19025: EXTRACTVALUE returns value of only one node

    Hi,
    I am new to XML DB. Can somebody help me with the below XML
    I use the following XML ... (I have pasted only a part of it coz i require data only till this section )
    XML
    <?xml version="1.0" encoding="UTF-8"?><SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-
    instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><SOAP-ENV:Body>
    <ns:PicklistWS_GetPicklistValues_Output xmlns:ns="urn:crmondemand/ws/picklist/">
    <ListOfParentPicklistValue xmlns="urn:/crmondemand/xml/picklist">
    <ParentPicklistValue>
    <Language>ENU</Language>
    <ParentFieldName>plProduct_Team</ParentFieldName>
    <ParentDisplayValue>Marketing On Demand</ParentDisplayValue>
    <ParentCode>Marketing On Demand</ParentCode>
    <Disabled>N</Disabled>
    <ListOfPicklistValue>
    <PicklistValue>
    <Code>OCP/SME Escalation</Code>
    <DisplayValue>OCP/SME Escalation</DisplayValue>
    <Disabled>N</Disabled>
    </PicklistValue>
    <PicklistValue>
    <Code>Fusion Request</Code>
    <DisplayValue>Fusion Request</DisplayValue>
    <Disabled>N</Disabled>
    </PicklistValue>
    Code
    SELECT distinct
    EXTRACTVALUE(VALUE(SR), '/ParentPicklistValue/ListOfPicklistValue/PicklistValue/Code','xmlns="urn:/crmondemand/xml/picklist"') AS Display,
    EXTRACTVALUE(VALUE(SR),'/ParentPicklistValue/ListOfPicklistValue/PicklistValue/DisplayValue','xmlns="urn:/crmondemand/xml/picklist"') AS Return,
    EXTRACTVALUE(VALUE(SR),'/ParentPicklistValue/ParentDisplayValue','xmlns="urn:/crmondemand/xml/picklist"') AS parent_display,
    EXTRACTVALUE(VALUE(SR),'/ParentPicklistValue/ParentCode','xmlns="urn:/crmondemand/xml/picklist"') AS parent_return
    FROM TABLE(XMLSEQUENCE(EXTRACT(
    WEB_SERVICE('<?xml version="1.0" encoding="UTF-8" standalone="no"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <soap:Body>
    <PicklistWS_GetPicklistValues_Input xmlns="urn:crmondemand/ws/picklist/">
    <FieldName>Type</FieldName>
    <RecordType>Service Request</RecordType>
    </PicklistWS_GetPicklistValues_Input>
    </soap:Body>
    </soap:Envelope>'
    ,'document/urn:crmondemand/ws/picklist/:GetPicklistValues', Sessionid),
    '/soap:Envelope/soap:Body/*/*/*','xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/'))) SR
    ERROR
    ORA-19025: EXTRACTVALUE returns value of only one node
    UNDERSTANDING
    As my Xpath points only until the node - ParentPicklistValue and not the child nodes under it. Hence when i try to query the child nodes - /ParentPicklistValue/ListOfPicklistValue/PicklistValue/Code, I recieve the above mentioned error.
    REQUIREMENT
    Can somebody help me to recieve the Parent values and also its child values based on the above query and xml.

    Hi,
    That's a classic ;)
    You need a second XMLSequence that will shred the collection of PicklistValue into relational rows :
    select extractvalue(value(sr2), '/PicklistValue/Code', 'xmlns="urn:/crmondemand/xml/picklist"') AS Display
         , extractvalue(value(sr2), '/PicklistValue/DisplayValue', 'xmlns="urn:/crmondemand/xml/picklist"') AS Return
         , extractvalue(value(sr1), '/ParentPicklistValue/ParentDisplayValue', 'xmlns="urn:/crmondemand/xml/picklist"') AS parent_display
         , extractvalue(value(sr1), '/ParentPicklistValue/ParentCode', 'xmlns="urn:/crmondemand/xml/picklist"') AS parent_return
    from table(
           xmlsequence(
             extract( WEB_SERVICE( ... )
                    , '/soap:Envelope/soap:Body/ns:PicklistWS_GetPicklistValues_Output/ListOfParentPicklistValue/ParentPicklistValue'
                    , 'xmlns="urn:/crmondemand/xml/picklist"
                       xmlns:ns="urn:crmondemand/ws/picklist/"
                       xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"' )
         ) sr1
       , table(
           xmlsequence(
             extract( value(sr1)
                    , '/ParentPicklistValue/ListOfPicklistValue/PicklistValue'
                    , 'xmlns="urn:/crmondemand/xml/picklist"' )
         ) sr2
    ;What's your database version BTW?
    On 10.2 and up, you may use XMLTable instead.

  • "ORA-19025 EXTRACTVALUE returns value of only one node" - Suggestion

    ORA-19025 EXTRACTVALUE returns value of only one node
    When this error occurs, would you please provide some or all of the following information:
    (1) row number
    (2) row id
    (3) the name of the offending column
    (5) the XQuery expression
    (6) the node name
    (7) the text value being parsed, in which an excess node was found
    and would you consider processing every good row, assuming it may be correct,
    because, in 13 thousand lines generated by my view, there is unlikely to even one more row in error.
    Thank you,
    Mike Rainville
    FYI,
    Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
    With the Partitioning, Real Application Clusters, OLAP and Data Mining options
    ojdbc6.jar 11.1.0.7.0

    Thanks for the suggestion...
    The problem is that in one row, information in a data dump about two different things was combined into one, due to a gap in the input file. The starting delimiter for the second thing and the ending delimiter for the first were missing. The result was that many entity tags that should have been unique were duplicated, ion that particular row.
    I was able to guard the view against future such errors with occurrences using this
    in the WHERE clause ...
    AND NOT ( XMLCLOB LIKE '%<TAG1>%<TAG1>%'
    OR XMLCLOB LIKE '%<TAG2>%<TAG2>%'
    OR XMLCLOB LIKE '%<TAG3>%<TAG3>%'
    /* ... Repeated once for each tag used with extractvalue */
    OR XMLCLOB LIKE '%<TAGN>%<TAGN>%'
    It filters out any row with two identical starting tags, where one is expected.
    I am pleased to see that the suggestion got through.

  • The tools on Adobe Photoshop Elements 9 seemed to have frozen. The crop symbol is 3 of the symbols nest to each other. The hand move tool is not one hand but 3 next to each other. Other tools won't work at all. Really nice!

    Ya, okay ! I use a number of PSE programs, one of which is PSE 9. The tools for some reason seem to have frozen. When I use the move tool I don't get just one hand but 3 hand symbols all next to each other. Same thing with the crop tool, I don't get one symbol but 3 all next to each other. There are a couple of tools that just don't work at all.  It's like the PSE 9 got hit with some virus or sorts. Oh well, live and learn.  Trying to get through to tech service is next to impossible.

    Have you seen this other thread you started about this problem?
    Photoshop Elements 9 is messed up.
    Did you use Barbara's fix?
    It would be less confusing if you only posted this question once, not three times:
    HOW DO I RESET ALL THE TOOLS IN PSE 9 ? THE CURSOR IS REALLY MESSED UP AND DOES NOT WORK WITH THE TOOLS. I TOUCH THE HAND AND I GET 3 HANDS, I TRY TO USE THE CROP AND I GET 3 SMALL CIRCLES.  THE CURSOR IS REALLY MESSED UP. DO I HAVE TO GET ANOTHER PSE 9 ?
    Photoshop Elements 9 is messed up.
    The tools on Adobe Photoshop Elements 9 seemed to have frozen. The crop symbol is 3 of the symbols nest to each other. The hand move tool is not one hand but 3 next to each other. Other tools won't work at all. Really nice!

  • RMAN Automatic backup/recovery with oracle fail safe, windows cluster

    Hello,
    I have question,
    1) Whether it is possible to do "RMAN Automatic Backup and recovery" in environment as ?
    Environment: -
    a. Windows clustering with windows server 2003 Enterprise Edition R2 (Two Node Clustering)
    b. Shared disk ( RAID )
    c. Oracle 10g standard edition one
    d. Oracle fail safe v3.3.3 (for redundancy)
    Here we have single oracle instance operating on single database whose files are located on shared disk.
    2) If answer to above question is yes please specify if there is some good documentation to it.
    Any help regarding this will be greatly appreciated.
    Thanks in advance,
    Rahul

    You just need to make sure that the RMAN scripts are always able to connect to the target database instance whether instance runs on cluster node 1 or cluster node 2
    here, If oracle services(resources) will shift from Node1 to Node2 (due to media or any failure) during RMAN Backup then there will be break in connection, will it destroy my backup or will it get started automatically without any harm?
    and also I want to know,
    Do we need to setup another server which will have RMAN backup script running?

  • Is there any standard way or testing utility for EJB fail safe in cluster env

              Hi
              I got to test the beans for complete fail safe like home
              fail overing and method calls (idempodentcy) fail over, is there
              any testing utility or standard approach for this or I got to do
              it manually??
              any help on this highly appriciated
              Thanks and regards
              Aruna
              

    Look for solutions here:
    https://discussions.apple.com/docs/DOC-3521
    https://discussions.apple.com/docs/DOC-3353
    Ciao.

  • Follwing query fails on one database, but the same works on other Database

    Hi,
    The following SQL fails on one database with ORA-01722 but the same works on other database.
    SQL> SELECT i.version, d.name, i.instance_name, i.host_name
    FROM v$instance i, v$database d, v$parameter p 2
    3 WHERE p.name = 'instance_number'
    4 AND DECODE(p.value, 0, 1, p.value) = i.instance_number;
    AND DECODE(p.value, 0, 1, p.value) = i.instance_number
    ERROR at line 4:
    ORA-01722: invalid number
    both databases are at same version (10.2.0.1) both are on same machine.
    Initailly I suspected NLS settings, I am surprised to see NLS settings are also same on both the databases.
    Any thoughts why is it happening

    Thanks for the reply
    Yes, i know that i am comparing varchar against number.
    bu there is no way that the database where the query is working has correct types on both sides as the tables are standard oracle dictionary tables.
    Yes, i tried this earlier, it works
    AND DECODE(p.value, '0', '1', p.value) = to_char(i.instance_number)
    it is very weired to see that error, i feel it something else that causing the trouble.

  • Session Failover not happening in SOA 11.1.1.5 Cluster when stop one Node

    Hi,
    We have a 2 Node SOA 11.1.1.5 Cluster. We are noticing one issue if when during a request processing if we bring down one Node the whole request fails and we loose the data. Other Node is not taking over the request to finish it. We have Hardware LoadBalancer that routes to one of OHS which internally routs to one of the SOA Servers. But when we shut down one Node right at that moment we are not able to access the service probably the request went to the server that is shutting down. This is the error we see
    oracle.fabric.common.FabricInvocationException: Unable to access the following endpoint(s): But when we try it again once the Node is down it works fine.
    Wondering if there's any setting we need to do to handle this type of scenario.
    Thanks

    I cut and paste the description of the note. There is a statement with the cause that states
    "This issue happens only when the display name of a NIC configured for migration in nodemanager is null"
    and it is a little ambigious. It implies there is a workaround of setting the display name, but doesn't discuss how to apply this workaround; the display name it refers to generated by a code call to the NetworkInterface.getDisplayName() method from the JDK.
    I assume that the display name here is the parameter called Interface within the nodemanger.properties, but there is nothing within the note that confirms this, nor if setting this to the NICE name will fix this issue. here is a full list of the nodemanger.propertes file -> http://download.oracle.com/docs/cd/E12840_01/wls/docs103/nodemgr/java_nodemgr.html#wp1068415
    I'd download and try the patch in your place; it's a WLS patch (as nodemanager is part of WLS) so shouldn't affect SOA Suite. Failing that, open a SR.

  • One node RAC pause/hang/block on other node shutdown

    Hi,
    We have a Java application running on Linux servers connecting to a 10.2.0.1 RAC cluster, also Linux. When the application starts it opens up a pool of connections to the databsae, and these are used throughout the life time of the application. One server connects to one RAC node.
    AppA - DBA
    AppB - DBB
    When we shutdown one node, the application connecting to that node stops, which is what we would expect in this configuration.
    What is strange is that the other application blocks for 63 seconds and then continues. So it is like the database is blocking, or the database connections are blocking.
    We are not using TAF, FAN, FCN, LB, VIPs or any special features, just simple lightweight JDBC from one server to one database. In fact I do not thing we are unwittingly using any of these features, we have them switched off.
    john

    user1788323 wrote:
    What is strange is that the other application blocks for 63 seconds and then continues. So it is like the database is blocking, or the database connections are blocking.How have you determined/diagnosed the 63s blocking? (more details in this regard may shed some light on the problem)
    Assuming that the block is server side, then two basic reasons comes to mind.
    Networking issue - the CRS on the surviving node has to perform certain functions, like switching the VIP of the node that left the cluster to a surviving cluster node. The listener may need to re-register services. A local firewall may need to be dynamically reconfigured for supporting the new failed-over VIP. Etc.
    Thus these could result in some kind of delay or issue in the network layer that you are seeing from the client side.
    Infrastructure issue. If the actual client request via JDBC reaches the server process, and it is slow in responding, then that is not a network issue - instead some underlying service or s/w layer that the server process needs to use to perform the client request is busy for those 63s.
    This could be related to the Interconnect, the shared I/O storage layer or something along those lines. For example, how does the Interconnect and/or SAN switch re-act when a server node is powered down or rebooted?
    There's not really sufficient information to make anything but a guesses.. You will need to isolate the problem with further testing.
    I have seen similar problems with 10.1.0.3 CRS and RAC when a node is evicted from the cluster. In this case the "hung" period was in excess of 15 minutes and only for new connections (Listener unable to hand off to dedicated servers or dispatchers). Existing connections worked fine however and were unaware of any problems. But part of the issue in this case was a poor (outdated) driver layer - and also the last time we used proprietary binary drivers (kernel modules) from 3rd party vendors that results in a tainted (and very fixed and rigid) Linux kernel. Today we're sticking with an OpenSource driver layer only for Linux.

  • Attachment won't open on one iPhone, but works fine on the other. Why?

    2 iPhones with the same iOS version receive 1 email with an attachment from an Outlook user: one iPhone can open the attachment, the other can't (it only shows as winmail.dat). Why is this and how can I get it to open? Thanks!

    He advised me to 'back up' everything and 'reformat' the computer.
    I have been looking at how to do these tasks - not 100% on both.
    The quickest way to do this is to create a clone copy of your existing internal Macintosh HD. You can create this copy using applications like Data Backup, Carbon Copy Cloner, or SuperDuper!.
    I use an external USB HDD for the clone copies.
    The following AppleInsider article will give you an idea on how to use Carbon Copy Cloner to make a clone backup of your Mac's hard drive.

  • 5s stopped accessing email for only one account but still accesses it for others on same server

    My 5s quit accessing one email account but continues to access the other account.  Both accounts use same provider and servers. Any ideas on what has happened? Have re-entered account several times as well as powered down the phone.

    Go into Settings>Mail, Contacts, Calendars and select the account you are having trouble with. Select the incoming mail server and double-check the username and password. Verify it is being used in all of the correct locations. If it is sending alright, then you do not have to worry about the SMTP server settings.

  • ORA-19025: EXTRACTVALUE returns value of only one nod- When creating index

    Hi,
    My table have 2 columns . Columns are
    Key varchar2(50)
    Attribute XMLType
    Below is my sample XML which is stored in Attribute column.
    <resource>
    <lang>
    <lc>FRE</lc>
    <lc>ARA</lc>
    </lang>
    <site>
    <sp>257204</sp>
    <sp>3664</sp>
    </site>
    <page>
    <pp>64272714</pp>
    <pp>64031775</pp>
    <pp>256635</pp>
    </page>
    <key>
    <kt>1</kt>
    </key>
    </resource>
    When i try to create a index on XML column and i am getting the above exception. kindly help me out becz i'm not familar with oracle.
    Query is
    create index XMLTest_ind on language_resource_wrapper
         (extractValue(attribute, '/resource/site/sp') )
    index on sp,pp elements in the above XML.

    Thanks for the suggestion...
    The problem is that in one row, information in a data dump about two different things was combined into one, due to a gap in the input file. The starting delimiter for the second thing and the ending delimiter for the first were missing. The result was that many entity tags that should have been unique were duplicated, ion that particular row.
    I was able to guard the view against future such errors with occurrences using this
    in the WHERE clause ...
    AND NOT ( XMLCLOB LIKE '%<TAG1>%<TAG1>%'
    OR XMLCLOB LIKE '%<TAG2>%<TAG2>%'
    OR XMLCLOB LIKE '%<TAG3>%<TAG3>%'
    /* ... Repeated once for each tag used with extractvalue */
    OR XMLCLOB LIKE '%<TAGN>%<TAGN>%'
    It filters out any row with two identical starting tags, where one is expected.
    I am pleased to see that the suggestion got through.

  • Sql Query is running fine in one database but running from long in other

    Hi All,
    Please advice me on below:
    One query is running fine on 11gr2 database with having 2GB SGA and cost of the running query is 15, but the same query is running from long time in other database having same 2GB SGA.
    Below is teh Query:
    SELECT CDU.USER_NAME, DECODE(:B2 , 'QUEUE','begin
    dbms_aqadm.grant_queue_privilege(''ALL'','''||:B1
    ||''','''||CDU.USER_NAME||''',true); end;', 'grant '||DECODE(:B2 ,
    'TABLE','SELECT,INSERT,UPDATE,DELETE'||DECODE(CDU.ADMIN,'Y',',REFERENCES
    ',''), 'VIEW',DECODE(:B3 ,NULL,'SELECT','SELECT,INSERT,UPDATE,DELETE'||D
    ECODE(CDU.ADMIN,'Y',',REFERENCES','')), 'TYPE','EXECUTE',
    'SEQUENCE','SELECT', 'EXECUTE')||' on
    '||PACK_UTILS.GET_SCHEMA_OWNER||'.'||:B1 ||' to
    "'||CDU.USER_NAME||'"'||DECODE(CDU.ADMIN,'Y',' WITH GRANT OPTION',''))
    AS GRANT_SOURCE, 'create or replace synonym
    "'||CDU.USER_NAME||'"."'||:B1 ||'" for
    "'||PACK_UTILS.GET_SCHEMA_OWNER||'"."'||:B1 ||'"' AS SYNONYM_SOURCE,
    NVL2( S.TABLE_NAME, 'Y', 'N' ) SYNONYM_EXISTS FROM CD_USERS CDU,
    ALL_SYNONYMS S, ALL_USERS U WHERE :B2 IN
    ('TYPE','TABLE','VIEW','FUNCTION','PROCEDURE','PACKAGE','SEQUENCE','QUEU
    E','SYNONYM') AND CDU.USER_TYPE = 'S' AND CDU.USER_NAME = U.USERNAME
    AND CDU.USER_NAME = S.OWNER AND :B1 = S.TABLE_NAME
    Please advise me as application team is chasing me like anything. what I can find out is that Cost is different in both databases.

    Sorry forr Late reply friends..
    below are the Explian plans for both databases:
                                                      BAD database
    PLAN_TABLE_OUTPUT
    SQL_ID  3bp3rynsds42r, child number 0
    SELECT CDU.USER_NAME, DECODE(:B2 , 'QUEUE','begin
    dbms_aqadm.grant_queue_privilege(''ALL'','''||:B1
    ||''','''||CDU.USER_NAME||''',true); end;', 'grant '||DECODE(:B2 ,
    'TABLE','SELECT,INSERT,UPDATE,DELETE'||DECODE(CDU.ADMIN,'Y',',REFERENCES
    ',''), 'VIEW',DECODE(:B3 ,NULL,'SELECT','SELECT,INSERT,UPDATE,DELETE'||D
    ECODE(CDU.ADMIN,'Y',',REFERENCES','')), 'TYPE','EXECUTE',
    'SEQUENCE','SELECT', 'EXECUTE')||' on
    '||PACK_UTILS.GET_SCHEMA_OWNER||'.'||:B1 ||' to
    "'||CDU.USER_NAME||'"'||DECODE(CDU.ADMIN,'Y',' WITH GRANT OPTION',''))
    PLAN_TABLE_OUTPUT
    AS GRANT_SOURCE, 'create or replace synonym
    "'||CDU.USER_NAME||'"."'||:B1 ||'" for
    "'||PACK_UTILS.GET_SCHEMA_OWNER||'"."'||:B1 ||'"' AS SYNONYM_SOURCE,
    NVL2( S.TABLE_NAME, 'Y', 'N' ) SYNONYM_EXISTS FROM CD_USERS CDU,
    ALL_SYNONYMS S, ALL_USERS U WHERE :B2 IN
    ('TYPE','TABLE','VIEW','FUNCTION','PROCEDURE','PACKAGE','SEQUENCE','QUEU
    E','SYNONYM') AND CDU.USER_TYPE = 'S' AND CDU.USER_NAME = U.USERNAME
    AND CDU.USER_NAME = S.OWNER AND :B1 = S.TABLE_NAME
    Plan hash value: 2404940060
    PLAN_TABLE_OUTPUT
    | Id  | Operation                                            | Name               | Rows  | Bytes |TempSpc| Cost (%CPU)| Time     |
    |   0 | SELECT STATEMENT                                     |                    |       |       |       |    94 (100)|          |
    |*  1 |  FILTER                                              |                    |       |       |       |            |          |
    |*  2 |   HASH JOIN                                          |                    |    81 |  6723 |       |    94   (0)| 00:00:01 |
    |   3 |    NESTED LOOPS OUTER                                |                    |    81 |  6480 |       |    91   (0)| 00:00:01 |
    |*  4 |     HASH JOIN                                        |                    |    81 |  3726 |       |    10   (0)| 00:00:01 |
    |*  5 |      HASH JOIN                                       |                    |   137 |  3699 |       |     7   (0)| 00:00:01 |
    |   6 |       TABLE ACCESS FULL                              | TS$                |     7 |    21 |       |     3   (0)| 00:00:01 |
    |*  7 |       TABLE ACCESS FULL                              | USER$              |   137 |  3288 |       |     4   (0)| 00:00:01 |
    PLAN_TABLE_OUTPUT
    |*  8 |      TABLE ACCESS FULL                               | CD_USERS           |    82 |  1558 |       |     3   (0)| 00:00:01 |
    |   9 |     VIEW                                             | ALL_SYNONYMS       |     1 |    34 |       |     1   (0)| 00:00:01 |
    |  10 |      SORT UNIQUE                                     |                    |       |       |       |            |          |
    |  11 |       UNION-ALL PARTITION                            |                    |       |       |       |            |          |
    |* 12 |        FILTER                                        |                    |       |       |       |            |          |
    |  13 |         NESTED LOOPS                                 |                    |     1 |   116 |       |    60   (0)| 00:00:01 |
    |  14 |          NESTED LOOPS                                |                    |     1 |    94 |       |    59   (0)| 00:00:01 |
    |  15 |           NESTED LOOPS                               |                    |    12 |   684 |       |    23   (0)| 00:00:01 |
    |  16 |            TABLE ACCESS BY INDEX ROWID               | USER$              |     1 |    19 |       |     1   (0)| 00:00:01 |
    |* 17 |             INDEX UNIQUE SCAN                        | I_USER1            |     1 |       |       |     0   (0)|          |
    |  18 |            TABLE ACCESS BY INDEX ROWID               | SYN$               |    12 |   456 |       |    22   (0)| 00:00:01 |
    PLAN_TABLE_OUTPUT
    |* 19 |             INDEX SKIP SCAN                          | I_SYN2             |    12 |       |       |    12   (0)| 00:00:01 |
    |* 20 |           TABLE ACCESS BY INDEX ROWID                | OBJ$               |     1 |    37 |       |     3   (0)| 00:00:01 |
    |* 21 |            INDEX RANGE SCAN                          | I_OBJ1             |     1 |       |       |     2   (0)| 00:00:01 |
    |* 22 |          INDEX RANGE SCAN                            | I_USER2            |     1 |    22 |       |     1   (0)| 00:00:01 |
    |* 23 |         FIXED TABLE FULL                             | X$KZSPR            |    17 |   119 |       |     0   (0)|          |
    |* 24 |         FILTER                                       |                    |       |       |       |            |          |
    |* 25 |          FILTER                                      |                    |       |       |       |            |          |
    |  26 |           NESTED LOOPS                               |                    |    27 |  2430 |       |     6   (0)| 00:00:01 |
    |  27 |            NESTED LOOPS                              |                    |     1 |    78 |       |     4   (0)| 00:00:01 |
    |  28 |             NESTED LOOPS                             |                    |     1 |    56 |       |     3   (0)| 00:00:01 |
    |  29 |              TABLE ACCESS BY INDEX ROWID             | USER$              |     1 |    19 |       |     1   (0)| 00:00:01 |
    PLAN_TABLE_OUTPUT
    |* 30 |               INDEX UNIQUE SCAN                      | I_USER1            |     1 |       |       |     0   (0)|          |
    |* 31 |              INDEX RANGE SCAN                        | I_OBJ5             |     1 |    37 |       |     2   (0)| 00:00:01 |
    |* 32 |             INDEX RANGE SCAN                         | I_USER2            |     1 |    22 |       |     1   (0)| 00:00:01 |
    |* 33 |            INDEX RANGE SCAN                          | I_OBJAUTH1         |    28 |   336 |       |     2   (0)| 00:00:01 |
    |* 34 |          FIXED TABLE FULL                            | X$KZSRO            |     1 |     3 |       |     0   (0)|          |
    |  35 |          NESTED LOOPS                                |                    |     1 |    30 |       |     3   (0)| 00:00:01 |
    |* 36 |           INDEX SKIP SCAN                            | I_USER2            |     1 |    20 |       |     1   (0)| 00:00:01 |
    |* 37 |           INDEX RANGE SCAN                           | I_OBJ4             |     1 |    10 |       |     2   (0)| 00:00:01 |
    |  38 |         NESTED LOOPS                                 |                    |     1 |    30 |       |     3   (0)| 00:00:01 |
    |* 39 |          INDEX SKIP SCAN                             | I_USER2            |     1 |    20 |       |     1   (0)| 00:00:01 |
    |* 40 |          INDEX RANGE SCAN                            | I_OBJ4             |     1 |    10 |       |     2   (0)| 00:00:01 |
    PLAN_TABLE_OUTPUT
    |* 41 |        FILTER                                        |                    |       |       |       |            |          |
    |  42 |         NESTED LOOPS                                 |                    |     1 |   129 |       |  4301   (2)| 00:00:39 |
    |  43 |          NESTED LOOPS                                |                    |     1 |   129 |       |  4301   (2)| 00:00:39 |
    |  44 |           NESTED LOOPS                               |                    |     1 |    91 |       |  4299   (2)| 00:00:39 |
    |  45 |            NESTED LOOPS                              |                    |     1 |    69 |       |  4298   (2)| 00:00:39 |
    |  46 |             NESTED LOOPS                             |                    |     2 |    64 |       |  4292   (2)| 00:00:39 |
    |  47 |              TABLE ACCESS BY INDEX ROWID             | USER$              |     1 |    19 |       |     1   (0)| 00:00:01 |
    |* 48 |               INDEX UNIQUE SCAN                      | I_USER1            |     1 |       |       |     0   (0)|          |
    |  49 |              VIEW                                    | _ALL_SYNONYMS_TREE |     2 |    26 |       |  4291   (2)| 00:00:39 |
    |* 50 |               CONNECT BY NO FILTERING WITH START-WITH|                    |       |       |       |            |          |
    |* 51 |                FILTER                                |                    |       |       |       |            |          |
    PLAN_TABLE_OUTPUT
    |* 52 |                 HASH JOIN                            |                    | 18370 |  2080K|       |  4291   (2)| 00:00:39 |
    |  53 |                  INDEX FULL SCAN                     | I_USER2            |   207 |  4554 |       |     1   (0)| 00:00:01 |
    |* 54 |                  HASH JOIN                           |                    | 18370 |  1686K|       |  4290   (2)| 00:00:39 |
    |  55 |                   TABLE ACCESS FULL                  | USER$              |   207 |  3933 |       |     4   (0)| 00:00:01 |
    |* 56 |                   HASH JOIN                          |                    |  2922K|   209M|    18M|  4269   (1)| 00:00:39 |
    |* 57 |                    INDEX FAST FULL SCAN              | I_OBJ2             |   387K|    13M|       |  1323   (1)| 00:00:12 |
    |  58 |                    TABLE ACCESS FULL                 | SYN$               |   400K|    14M|       |   566   (2)| 00:00:06 |
    |  59 |                 NESTED LOOPS                         |                    |     1 |    30 |       |     3   (0)| 00:00:01 |
    |* 60 |                  INDEX SKIP SCAN                     | I_USER2            |     1 |    20 |       |     1   (0)| 00:00:01 |
    |* 61 |                  INDEX RANGE SCAN                    | I_OBJ4             |     1 |    10 |       |     2   (0)| 00:00:01 |
    |* 62 |                FILTER                                |                    |       |       |       |            |          |
    PLAN_TABLE_OUTPUT
    |  63 |                 TABLE ACCESS BY INDEX ROWID          | SYN$               |     1 |    38 |       |     3   (0)| 00:00:01 |
    |* 64 |                  INDEX UNIQUE SCAN                   | I_SYN1             |     1 |       |       |     2   (0)| 00:00:01 |
    |* 65 |                 FILTER                               |                    |       |       |       |            |          |
    |* 66 |                  FILTER                              |                    |       |       |       |            |          |
    |  67 |                   NESTED LOOPS                       |                    |    27 |  2430 |       |     6   (0)| 00:00:01 |
    |  68 |                    NESTED LOOPS                      |                    |     1 |    78 |       |     4   (0)| 00:00:01 |
    |  69 |                     NESTED LOOPS                     |                    |     1 |    56 |       |     3   (0)| 00:00:01 |
    |  70 |                      TABLE ACCESS BY INDEX ROWID     | USER$              |     1 |    19 |       |     1   (0)| 00:00:01 |
    |* 71 |                       INDEX UNIQUE SCAN              | I_USER1            |     1 |       |       |     0   (0)|          |
    |* 72 |                      INDEX RANGE SCAN                | I_OBJ5             |     1 |    37 |       |     2   (0)| 00:00:01 |
    |* 73 |                     INDEX RANGE SCAN                 | I_USER2            |     1 |    22 |       |     1   (0)| 00:00:01 |
    PLAN_TABLE_OUTPUT
    |* 74 |                    INDEX RANGE SCAN                  | I_OBJAUTH1         |    28 |   336 |       |     2   (0)| 00:00:01 |
    |* 75 |                  FIXED TABLE FULL                    | X$KZSRO            |     1 |     3 |       |     0   (0)|          |
    |  76 |                  NESTED LOOPS                        |                    |     1 |    30 |       |     3   (0)| 00:00:01 |
    |* 77 |                   INDEX SKIP SCAN                    | I_USER2            |     1 |    20 |       |     1   (0)| 00:00:01 |
    |* 78 |                   INDEX RANGE SCAN                   | I_OBJ4             |     1 |    10 |       |     2   (0)| 00:00:01 |
    |* 79 |                 FIXED TABLE FULL                     | X$KZSPR            |    17 |   119 |       |     0   (0)|          |
    |* 80 |             TABLE ACCESS BY INDEX ROWID              | OBJ$               |     1 |    37 |       |     3   (0)| 00:00:01 |
    |* 81 |              INDEX RANGE SCAN                        | I_OBJ1             |     1 |       |       |     2   (0)| 00:00:01 |
    |* 82 |            INDEX RANGE SCAN                          | I_USER2            |     1 |    22 |       |     1   (0)| 00:00:01 |
    |* 83 |           INDEX UNIQUE SCAN                          | I_SYN1             |     1 |       |       |     1   (0)| 00:00:01 |
    |* 84 |          TABLE ACCESS BY INDEX ROWID                 | SYN$               |     1 |    38 |       |     2   (0)| 00:00:01 |
    PLAN_TABLE_OUTPUT
    |  85 |         NESTED LOOPS                                 |                    |     1 |    30 |       |     3   (0)| 00:00:01 |
    |* 86 |          INDEX SKIP SCAN                             | I_USER2            |     1 |    20 |       |     1   (0)| 00:00:01 |
    |* 87 |          INDEX RANGE SCAN                            | I_OBJ4             |     1 |    10 |       |     2   (0)| 00:00:01 |
    |  88 |    TABLE ACCESS FULL                                 | TS$                |     7 |    21 |       |     3   (0)| 00:00:01 |
    Predicate Information (identified by operation id):
       1 - filter((:B2='TYPE' OR :B2='TABLE' OR :B2='VIEW' OR :B2='FUNCTION' OR :B2='PROCEDURE' OR :B2='PACKAGE' OR
                  :B2='SEQUENCE' OR :B2='QUEUE' OR :B2='SYNONYM'))
    PLAN_TABLE_OUTPUT
       2 - access("U"."TEMPTS#"="TTS"."TS#")
       4 - access("CDU"."USER_NAME"="U"."NAME")
       5 - access("U"."DATATS#"="DTS"."TS#")
       7 - filter("U"."TYPE#"=1)
       8 - filter("CDU"."USER_TYPE"='S')
      12 - filter(((INTERNAL_FUNCTION("O"."SPARE3") OR ("S"."NODE" IS NULL AND  IS NOT NULL) OR  IS NOT NULL) AND
                  (("O"."TYPE#"<>4 AND "O"."TYPE#"<>5 AND "O"."TYPE#"<>7 AND "O"."TYPE#"<>8 AND "O"."TYPE#"<>9 AND "O"."TYPE#"<>10 AND
                  "O"."TYPE#"<>11 AND "O"."TYPE#"<>12 AND "O"."TYPE#"<>13 AND "O"."TYPE#"<>14 AND "O"."TYPE#"<>22 AND "O"."TYPE#"<>87 AND
                  "O"."TYPE#"<>88) OR BITAND("U"."SPARE1",16)=0 OR (SYS_CONTEXT('userenv','current_edition_name')='ORA$BASE' AND
                  "U"."TYPE#"<>2) OR ("U"."TYPE#"=2 AND "U"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))) OR  IS NOT
                  NULL)))
    PLAN_TABLE_OUTPUT
      17 - access("U"."NAME"="CDU"."USER_NAME")
      19 - access("S"."NAME"=:B1)
           filter("S"."NAME"=:B1)
      20 - filter("O"."SPARE3"="U"."USER#")
      21 - access("O"."OBJ#"="S"."OBJ#" AND "O"."TYPE#"=5)
           filter("O"."TYPE#"=5)
      22 - access("O"."OWNER#"="U"."USER#")
      23 - filter((((-"KZSPRPRV")=(-45) OR (-"KZSPRPRV")=(-47) OR (-"KZSPRPRV")=(-48) OR (-"KZSPRPRV")=(-49) OR
                  (-"KZSPRPRV")=(-50)) AND "INST_ID"=USERENV('INSTANCE')))
      24 - filter((("BA"."GRANTOR#"=USERENV('SCHEMAID') OR  IS NOT NULL) AND (("O"."TYPE#"<>4 AND "O"."TYPE#"<>5 AND
                  "O"."TYPE#"<>7 AND "O"."TYPE#"<>8 AND "O"."TYPE#"<>9 AND "O"."TYPE#"<>10 AND "O"."TYPE#"<>11 AND "O"."TYPE#"<>12 AND
    PLAN_TABLE_OUTPUT
                  "O"."TYPE#"<>13 AND "O"."TYPE#"<>14 AND "O"."TYPE#"<>22 AND "O"."TYPE#"<>87 AND "O"."TYPE#"<>88) OR
                  BITAND("U"."SPARE1",16)=0 OR (INTERNAL_FUNCTION("O"."TYPE#") AND
                  ((SYS_CONTEXT('userenv','current_edition_name')='ORA$BASE' AND "U"."TYPE#"<>2) OR ("U"."TYPE#"=2 AND
                  "U"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))) OR  IS NOT NULL)))))
      25 - filter(:B1 IS NULL)
      30 - access("BU"."NAME"=:B1)
      31 - access("BU"."USER#"="O"."SPARE3" AND "O"."NAME"=:B1)
      32 - access("O"."OWNER#"="U"."USER#")
      33 - access("BA"."OBJ#"="O"."OBJ#")
      34 - filter("KZSROROL"=:B1)
      36 - access("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id')))
    PLAN_TABLE_OUTPUT
           filter(("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))))
      37 - access("O2"."DATAOBJ#"=:B1 AND "O2"."TYPE#"=88 AND "O2"."OWNER#"="U2"."USER#")
      39 - access("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id')))
           filter(("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))))
      40 - access("O2"."DATAOBJ#"=:B1 AND "O2"."TYPE#"=88 AND "O2"."OWNER#"="U2"."USER#")
      41 - filter((("O"."TYPE#"<>4 AND "O"."TYPE#"<>5 AND "O"."TYPE#"<>7 AND "O"."TYPE#"<>8 AND "O"."TYPE#"<>9 AND
                  "O"."TYPE#"<>10 AND "O"."TYPE#"<>11 AND "O"."TYPE#"<>12 AND "O"."TYPE#"<>13 AND "O"."TYPE#"<>14 AND "O"."TYPE#"<>22 AND
                  "O"."TYPE#"<>87 AND "O"."TYPE#"<>88) OR BITAND("U"."SPARE1",16)=0 OR
                  (SYS_CONTEXT('userenv','current_edition_name')='ORA$BASE' AND "U"."TYPE#"<>2) OR ("U"."TYPE#"=2 AND
                  "U"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))) OR  IS NOT NULL))
      48 - access("U"."NAME"="CDU"."USER_NAME")
    PLAN_TABLE_OUTPUT
      50 - access("S"."BASE_SYN_ID"=PRIOR NULL)
           filter( IS NOT NULL)
      51 - filter((("O"."TYPE#"<>4 AND "O"."TYPE#"<>5 AND "O"."TYPE#"<>7 AND "O"."TYPE#"<>8 AND "O"."TYPE#"<>9 AND
                  "O"."TYPE#"<>10 AND "O"."TYPE#"<>11 AND "O"."TYPE#"<>12 AND "O"."TYPE#"<>13 AND "O"."TYPE#"<>14 AND "O"."TYPE#"<>22 AND
                  "O"."TYPE#"<>87 AND "O"."TYPE#"<>88) OR BITAND("U"."SPARE1",16)=0 OR
                  (SYS_CONTEXT('userenv','current_edition_name')='ORA$BASE' AND "U"."TYPE#"<>2) OR ("U"."TYPE#"=2 AND
                  "U"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))) OR  IS NOT NULL))
      52 - access("O"."OWNER#"="U"."USER#")
      54 - access("S"."OWNER"="BU"."NAME" AND "BU"."USER#"="O"."SPARE3")
      56 - access("S"."NAME"="O"."NAME")
      57 - filter("O"."TYPE#"=5)
    PLAN_TABLE_OUTPUT
      60 - access("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id')))
           filter(("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))))
      61 - access("O2"."DATAOBJ#"=:B1 AND "O2"."TYPE#"=88 AND "O2"."OWNER#"="U2"."USER#")
      62 - filter(( IS NOT NULL OR ("S"."NODE" IS NULL AND  IS NOT NULL)))
      64 - access("S"."OBJ#"=:B1)
      65 - filter((("BA"."GRANTOR#"=USERENV('SCHEMAID') OR  IS NOT NULL) AND (("O"."TYPE#"<>4 AND "O"."TYPE#"<>5 AND
                  "O"."TYPE#"<>7 AND "O"."TYPE#"<>8 AND "O"."TYPE#"<>9 AND "O"."TYPE#"<>10 AND "O"."TYPE#"<>11 AND "O"."TYPE#"<>12 AND
                  "O"."TYPE#"<>13 AND "O"."TYPE#"<>14 AND "O"."TYPE#"<>22 AND "O"."TYPE#"<>87 AND "O"."TYPE#"<>88) OR
                  BITAND("U"."SPARE1",16)=0 OR (INTERNAL_FUNCTION("O"."TYPE#") AND
                  ((SYS_CONTEXT('userenv','current_edition_name')='ORA$BASE' AND "U"."TYPE#"<>2) OR ("U"."TYPE#"=2 AND
                  "U"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))) OR  IS NOT NULL)))))
    PLAN_TABLE_OUTPUT
      66 - filter(:B1 IS NULL)
      71 - access("BU"."NAME"=:B1)
      72 - access("BU"."USER#"="O"."SPARE3" AND "O"."NAME"=:B1)
      73 - access("O"."OWNER#"="U"."USER#")
      74 - access("BA"."OBJ#"="O"."OBJ#")
      75 - filter("KZSROROL"=:B1)
      77 - access("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id')))
           filter(("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))))
      78 - access("O2"."DATAOBJ#"=:B1 AND "O2"."TYPE#"=88 AND "O2"."OWNER#"="U2"."USER#")
      79 - filter((((-"KZSPRPRV")=(-45) OR (-"KZSPRPRV")=(-47) OR (-"KZSPRPRV")=(-48) OR (-"KZSPRPRV")=(-49) OR
                  (-"KZSPRPRV")=(-50)) AND "INST_ID"=USERENV('INSTANCE')))
    PLAN_TABLE_OUTPUT
      80 - filter("O"."SPARE3"="U"."USER#")
      81 - access("O"."OBJ#"="ST"."SYN_ID" AND "O"."TYPE#"=5)
           filter("O"."TYPE#"=5)
      82 - access("O"."OWNER#"="U"."USER#")
      83 - access("S"."OBJ#"="ST"."SYN_ID")
           filter("O"."OBJ#"="S"."OBJ#")
      84 - filter("S"."NAME"=:B1)
      86 - access("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id')))
           filter(("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))))
      87 - access("O2"."DATAOBJ#"=:B1 AND "O2"."TYPE#"=88 AND "O2"."OWNER#"="U2"."USER#")
    209 rows selected
    ==============================================================================================
                                                                     Good one
    PLAN_TABLE_OUTPUT
    SQL_ID  3bp3rynsds42r, child number 0
    SELECT CDU.USER_NAME, DECODE(:B2 , 'QUEUE','begin
    dbms_aqadm.grant_queue_privilege(''ALL'','''||:B1
    ||''','''||CDU.USER_NAME||''',true); end;', 'grant '||DECODE(:B2 ,
    'TABLE','SELECT,INSERT,UPDATE,DELETE'||DECODE(CDU.ADMIN,'Y',',REFERENCES
    ',''), 'VIEW',DECODE(:B3 ,NULL,'SELECT','SELECT,INSERT,UPDATE,DELETE'||D
    ECODE(CDU.ADMIN,'Y',',REFERENCES','')), 'TYPE','EXECUTE',
    'SEQUENCE','SELECT', 'EXECUTE')||' on
    '||PACK_UTILS.GET_SCHEMA_OWNER||'.'||:B1 ||' to
    "'||CDU.USER_NAME||'"'||DECODE(CDU.ADMIN,'Y',' WITH GRANT OPTION',''))
    PLAN_TABLE_OUTPUT
    AS GRANT_SOURCE, 'create or replace synonym
    "'||CDU.USER_NAME||'"."'||:B1 ||'" for
    "'||PACK_UTILS.GET_SCHEMA_OWNER||'"."'||:B1 ||'"' AS SYNONYM_SOURCE,
    NVL2( S.TABLE_NAME, 'Y', 'N' ) SYNONYM_EXISTS FROM CD_USERS CDU,
    ALL_SYNONYMS S, ALL_USERS U WHERE :B2 IN
    ('TYPE','TABLE','VIEW','FUNCTION','PROCEDURE','PACKAGE','SEQUENCE','QUEU
    E','SYNONYM') AND CDU.USER_TYPE = 'S' AND CDU.USER_NAME = U.USERNAME
    AND CDU.USER_NAME = S.OWNER AND :B1 = S.TABLE_NAME
    Plan hash value: 2146531909
    PLAN_TABLE_OUTPUT
    | Id  | Operation                                          | Name               | Rows  | Bytes |TempSpc| Cost (%CPU)| Time     |
    |   0 | SELECT STATEMENT                                   |                    |       |       |       |    15 (100)|          |
    |*  1 |  FILTER                                            |                    |       |       |       |            |          |
    |   2 |   NESTED LOOPS OUTER                               |                    |     3 |   273 |       |    15   (0)| 00:00:01 |
    |   3 |    NESTED LOOPS                                    |                    |     3 |   153 |       |    12   (0)| 00:00:01 |
    |   4 |     NESTED LOOPS                                   |                    |     3 |   144 |       |     9   (0)| 00:00:01 |
    |   5 |      NESTED LOOPS                                  |                    |     3 |   135 |       |     6   (0)| 00:00:01 |
    |*  6 |       TABLE ACCESS FULL                            | USER$              |    55 |  1375 |       |     4   (0)| 00:00:01 |
    |*  7 |       TABLE ACCESS BY INDEX ROWID                  | CD_USERS           |     1 |    20 |       |     1   (0)| 00:00:01 |
    PLAN_TABLE_OUTPUT
    |*  8 |        INDEX UNIQUE SCAN                           | I1_CD_USERS        |     1 |       |       |     0   (0)|          |
    |   9 |      TABLE ACCESS CLUSTER                          | TS$                |     1 |     3 |       |     1   (0)| 00:00:01 |
    |* 10 |       INDEX UNIQUE SCAN                            | I_TS#              |     1 |       |       |     0   (0)|          |
    |  11 |     TABLE ACCESS CLUSTER                           | TS$                |     1 |     3 |       |     1   (0)| 00:00:01 |
    |* 12 |      INDEX UNIQUE SCAN                             | I_TS#              |     1 |       |       |     0   (0)|          |
    |  13 |    VIEW                                            | ALL_SYNONYMS       |     1 |    40 |       |     1   (0)| 00:00:01 |
    |  14 |     SORT UNIQUE                                    |                    |       |       |       |            |          |
    |  15 |      UNION-ALL PARTITION                           |                    |       |       |       |            |          |
    |* 16 |       FILTER                                       |                    |       |       |       |            |          |
    |  17 |        NESTED LOOPS                                |                    |     1 |   116 |       |    19   (0)| 00:00:01 |
    |  18 |         NESTED LOOPS                               |                    |     1 |    94 |       |    18   (0)| 00:00:01 |
    PLAN_TABLE_OUTPUT
    |  19 |          NESTED LOOPS                              |                    |     1 |    56 |       |    15   (0)| 00:00:01 |
    |  20 |           TABLE ACCESS BY INDEX ROWID              | USER$              |     1 |    20 |       |     1   (0)| 00:00:01 |
    |* 21 |            INDEX UNIQUE SCAN                       | I_USER1            |     1 |       |       |     0   (0)|          |
    |  22 |           TABLE ACCESS BY INDEX ROWID              | SYN$               |     1 |    36 |       |    14   (0)| 00:00:01 |
    |* 23 |            INDEX SKIP SCAN                         | I_SYN2             |     1 |       |       |    12   (0)| 00:00:01 |
    |* 24 |          TABLE ACCESS BY INDEX ROWID               | OBJ$               |     1 |    38 |       |     3   (0)| 00:00:01 |
    |* 25 |           INDEX RANGE SCAN                         | I_OBJ1             |     1 |       |       |     2   (0)| 00:00:01 |
    |* 26 |         INDEX RANGE SCAN                           | I_USER2            |     1 |    22 |       |     1   (0)| 00:00:01 |
    |* 27 |        FIXED TABLE FULL                            | X$KZSPR            |    17 |   119 |       |     0   (0)|          |
    |* 28 |        FILTER                                      |                    |       |       |       |            |          |
    |* 29 |         FILTER                                     |                    |       |       |       |            |          |
    PLAN_TABLE_OUTPUT
    |  30 |          NESTED LOOPS                              |                    |     3 |   279 |       |     5   (0)| 00:00:01 |
    |  31 |           NESTED LOOPS                             |                    |     1 |    80 |       |     4   (0)| 00:00:01 |
    |  32 |            NESTED LOOPS                            |                    |     1 |    58 |       |     3   (0)| 00:00:01 |
    |  33 |             TABLE ACCESS BY INDEX ROWID            | USER$              |     1 |    20 |       |     1   (0)| 00:00:01 |
    |* 34 |              INDEX UNIQUE SCAN                     | I_USER1            |     1 |       |       |     0   (0)|          |
    |* 35 |             INDEX RANGE SCAN                       | I_OBJ5             |     1 |    38 |       |     2   (0)| 00:00:01 |
    |* 36 |            INDEX RANGE SCAN                        | I_USER2            |     1 |    22 |       |     1   (0)| 00:00:01 |
    |* 37 |           INDEX RANGE SCAN                         | I_OBJAUTH1         |     3 |    39 |       |     1   (0)| 00:00:01 |
    |* 38 |         FIXED TABLE FULL                           | X$KZSRO            |     1 |     3 |       |     0   (0)|          |
    |  39 |         NESTED LOOPS                               |                    |     1 |    32 |       |     3   (0)| 00:00:01 |
    |* 40 |          INDEX SKIP SCAN                           | I_USER2            |     1 |    20 |       |     1   (0)| 00:00:01 |
    PLAN_TABLE_OUTPUT
    |* 41 |          INDEX RANGE SCAN                          | I_OBJ4             |     1 |    12 |       |     2   (0)| 00:00:01 |
    |  42 |        NESTED LOOPS                                |                    |     1 |    32 |       |     3   (0)| 00:00:01 |
    |* 43 |         INDEX SKIP SCAN                            | I_USER2            |     1 |    20 |       |     1   (0)| 00:00:01 |
    |* 44 |         INDEX RANGE SCAN                           | I_OBJ4             |     1 |    12 |       |     2   (0)| 00:00:01 |
    |* 45 |       FILTER                                       |                    |       |       |       |            |          |
    |  46 |        NESTED LOOPS                                |                    |     1 |   129 |       |  1065   (1)| 00:00:13 |
    |  47 |         NESTED LOOPS                               |                    |     1 |   107 |       |  1064   (1)| 00:00:13 |
    |  48 |          NESTED LOOPS                              |                    |     1 |    69 |       |  1061   (1)| 00:00:13 |
    |  49 |           NESTED LOOPS                             |                    |     2 |    66 |       |  1059   (1)| 00:00:13 |
    |  50 |            TABLE ACCESS BY INDEX ROWID             | USER$              |     1 |    20 |       |     1   (0)| 00:00:01 |
    |* 51 |             INDEX UNIQUE SCAN                      | I_USER1            |     1 |       |       |     0   (0)|          |
    PLAN_TABLE_OUTPUT
    |  52 |            VIEW                                    | _ALL_SYNONYMS_TREE |     2 |    26 |       |  1058   (1)| 00:00:13 |
    |* 53 |             CONNECT BY NO FILTERING WITH START-WITH|                    |       |       |       |            |          |
    |* 54 |              FILTER                                |                    |       |       |       |            |          |
    |* 55 |               HASH JOIN                            |                    |   705 | 81780 |       |  1058   (1)| 00:00:13 |
    |  56 |                INDEX FULL SCAN                     | I_USER2            |   124 |  2728 |       |     1   (0)| 00:00:01 |
    |* 57 |                HASH JOIN                           |                    |   705 | 66270 |       |  1057   (1)| 00:00:13 |
    |  58 |                 TABLE ACCESS FULL                  | USER$              |   124 |  2480 |       |     4   (0)| 00:00:01 |
    |* 59 |                 HASH JOIN                          |                    | 66964 |  4839K|  2232K|  1052   (1)| 00:00:13 |
    |  60 |                  TABLE ACCESS FULL                 | SYN$               | 47615 |  1673K|       |    78   (2)| 00:00:01 |
    |* 61 |                  INDEX FAST FULL SCAN              | I_OBJ5             | 47862 |  1776K|       |   752   (1)| 00:00:10 |
    |  62 |               NESTED LOOPS                         |                    |     1 |    32 |       |     3   (0)| 00:00:01 |
    PLAN_TABLE_OUTPUT
    |* 63 |                INDEX SKIP SCAN                     | I_USER2            |     1 |    20 |       |     1   (0)| 00:00:01 |
    |* 64 |                INDEX RANGE SCAN                    | I_OBJ4             |     1 |    12 |       |     2   (0)| 00:00:01 |
    |* 65 |              FILTER                                |                    |       |       |       |            |          |
    |  66 |               TABLE ACCESS BY INDEX ROWID          | SYN$               |     1 |    36 |       |     2   (0)| 00:00:01 |
    |* 67 |                INDEX UNIQUE SCAN                   | I_SYN1             |     1 |       |       |     1   (0)| 00:00:01 |
    |* 68 |               FILTER                               |                    |       |       |       |            |          |
    |* 69 |                FILTER                              |                    |       |       |       |            |          |
    |  70 |                 NESTED LOOPS                       |                    |     3 |   279 |       |     5   (0)| 00:00:01 |
    |  71 |                  NESTED LOOPS                      |                    |     1 |    80 |       |     4   (0)| 00:00:01 |
    |  72 |                   NESTED LOOPS                     |                    |     1 |    58 |       |     3   (0)| 00:00:01 |
    |  73 |                    TABLE ACCESS BY INDEX ROWID     | USER$              |     1 |    20 |       |     1   (0)| 00:00:01 |
    PLAN_TABLE_OUTPUT
    |* 74 |                     INDEX UNIQUE SCAN              | I_USER1            |     1 |       |       |     0   (0)|          |
    |* 75 |                    INDEX RANGE SCAN                | I_OBJ5             |     1 |    38 |       |     2   (0)| 00:00:01 |
    |* 76 |                   INDEX RANGE SCAN                 | I_USER2            |     1 |    22 |       |     1   (0)| 00:00:01 |
    |* 77 |                  INDEX RANGE SCAN                  | I_OBJAUTH1         |     3 |    39 |       |     1   (0)| 00:00:01 |
    |* 78 |                FIXED TABLE FULL                    | X$KZSRO            |     1 |     3 |       |     0   (0)|          |
    |  79 |                NESTED LOOPS                        |                    |     1 |    32 |       |     3   (0)| 00:00:01 |
    |* 80 |                 INDEX SKIP SCAN                    | I_USER2            |     1 |    20 |       |     1   (0)| 00:00:01 |
    |* 81 |                 INDEX RANGE SCAN                   | I_OBJ4             |     1 |    12 |       |     2   (0)| 00:00:01 |
    |* 82 |               FIXED TABLE FULL                     | X$KZSPR            |    17 |   119 |       |     0   (0)|          |
    |* 83 |           TABLE ACCESS BY INDEX ROWID              | SYN$               |     1 |    36 |       |     1   (0)| 00:00:01 |
    |* 84 |            INDEX UNIQUE SCAN                       | I_SYN1             |     1 |       |       |     0   (0)|          |
    PLAN_TABLE_OUTPUT
    |* 85 |          TABLE ACCESS BY INDEX ROWID               | OBJ$               |     1 |    38 |       |     3   (0)| 00:00:01 |
    |* 86 |           INDEX RANGE SCAN                         | I_OBJ1             |     1 |       |       |     2   (0)| 00:00:01 |
    |* 87 |         INDEX RANGE SCAN                           | I_USER2            |     1 |    22 |       |     1   (0)| 00:00:01 |
    |  88 |        NESTED LOOPS                                |                    |     1 |    32 |       |     3   (0)| 00:00:01 |
    |* 89 |         INDEX SKIP SCAN                            | I_USER2            |     1 |    20 |       |     1   (0)| 00:00:01 |
    |* 90 |         INDEX RANGE SCAN                           | I_OBJ4             |     1 |    12 |       |     2   (0)| 00:00:01 |
    Predicate Information (identified by operation id):
    PLAN_TABLE_OUTPUT
       1 - filter((:B2='TYPE' OR :B2='TABLE' OR :B2='VIEW' OR :B2='FUNCTION' OR :B2='PROCEDURE' OR :B2='PACKAGE' OR
                  :B2='SEQUENCE' OR :B2='QUEUE' OR :B2='SYNONYM'))
       6 - filter("U"."TYPE#"=1)
       7 - filter("CDU"."USER_TYPE"='S')
       8 - access("CDU"."USER_NAME"="U"."NAME")
      10 - access("U"."DATATS#"="DTS"."TS#")
      12 - access("U"."TEMPTS#"="TTS"."TS#")
      16 - filter(((INTERNAL_FUNCTION("O"."SPARE3") OR ("S"."NODE" IS NULL AND  IS NOT NULL) OR  IS NOT NULL) AND
                  (("O"."TYPE#"<>4 AND "O"."TYPE#"<>5 AND "O"."TYPE#"<>7 AND "O"."TYPE#"<>8 AND "O"."TYPE#"<>9 AND "O"."TYPE#"<>10 AND
                  "O"."TYPE#"<>11 AND "O"."TYPE#"<>12 AND "O"."TYPE#"<>13 AND "O"."TYPE#"<>14 AND "O"."TYPE#"<>22 AND "O"."TYPE#"<>87 AND
                  "O"."TYPE#"<>88) OR BITAND("U"."SPARE1",16)=0 OR (SYS_CONTEXT('userenv','current_edition_name')='ORA$BASE' AND
    PLAN_TABLE_OUTPUT
                  "U"."TYPE#"<>2) OR ("U"."TYPE#"=2 AND "U"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))) OR  IS NOT
                  NULL)))
      21 - access("U"."NAME"="CDU"."USER_NAME")
      23 - access("S"."NAME"=:B1)
           filter("S"."NAME"=:B1)
      24 - filter("O"."SPARE3"="U"."USER#")
      25 - access("O"."OBJ#"="S"."OBJ#" AND "O"."TYPE#"=5)
           filter("O"."TYPE#"=5)
      26 - access("O"."OWNER#"="U"."USER#")
      27 - filter((((-"KZSPRPRV")=(-45) OR (-"KZSPRPRV")=(-47) OR (-"KZSPRPRV")=(-48) OR (-"KZSPRPRV")=(-49) OR
                  (-"KZSPRPRV")=(-50)) AND "INST_ID"=USERENV('INSTANCE')))
    PLAN_TABLE_OUTPUT
      28 - filter((("BA"."GRANTOR#"=USERENV('SCHEMAID') OR  IS NOT NULL) AND (("O"."TYPE#"<>4 AND "O"."TYPE#"<>5 AND
                  "O"."TYPE#"<>7 AND "O"."TYPE#"<>8 AND "O"."TYPE#"<>9 AND "O"."TYPE#"<>10 AND "O"."TYPE#"<>11 AND "O"."TYPE#"<>12 AND
                  "O"."TYPE#"<>13 AND "O"."TYPE#"<>14 AND "O"."TYPE#"<>22 AND "O"."TYPE#"<>87 AND "O"."TYPE#"<>88) OR
                  BITAND("U"."SPARE1",16)=0 OR (INTERNAL_FUNCTION("O"."TYPE#") AND
                  ((SYS_CONTEXT('userenv','current_edition_name')='ORA$BASE' AND "U"."TYPE#"<>2) OR ("U"."TYPE#"=2 AND
                  "U"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))) OR  IS NOT NULL)))))
      29 - filter(:B1 IS NULL)
      34 - access("BU"."NAME"=:B1)
      35 - access("BU"."USER#"="O"."SPARE3" AND "O"."NAME"=:B1)
      36 - access("O"."OWNER#"="U"."USER#")
      37 - access("BA"."OBJ#"="O"."OBJ#")
    PLAN_TABLE_OUTPUT
      38 - filter("KZSROROL"=:B1)
      40 - access("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id')))
           filter(("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))))
      41 - access("O2"."DATAOBJ#"=:B1 AND "O2"."TYPE#"=88 AND "O2"."OWNER#"="U2"."USER#")
      43 - access("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id')))
           filter(("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))))
      44 - access("O2"."DATAOBJ#"=:B1 AND "O2"."TYPE#"=88 AND "O2"."OWNER#"="U2"."USER#")
      45 - filter((("O"."TYPE#"<>4 AND "O"."TYPE#"<>5 AND "O"."TYPE#"<>7 AND "O"."TYPE#"<>8 AND "O"."TYPE#"<>9 AND
                  "O"."TYPE#"<>10 AND "O"."TYPE#"<>11 AND "O"."TYPE#"<>12 AND "O"."TYPE#"<>13 AND "O"."TYPE#"<>14 AND "O"."TYPE#"<>22 AND
                  "O"."TYPE#"<>87 AND "O"."TYPE#"<>88) OR BITAND("U"."SPARE1",16)=0 OR
                  (SYS_CONTEXT('userenv','current_edition_name')='ORA$BASE' AND "U"."TYPE#"<>2) OR ("U"."TYPE#"=2 AND
    PLAN_TABLE_OUTPUT
                  "U"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))) OR  IS NOT NULL))
      51 - access("U"."NAME"="CDU"."USER_NAME")
      53 - access("S"."BASE_SYN_ID"=PRIOR NULL)
           filter( IS NOT NULL)
      54 - filter((("O"."TYPE#"<>4 AND "O"."TYPE#"<>5 AND "O"."TYPE#"<>7 AND "O"."TYPE#"<>8 AND "O"."TYPE#"<>9 AND
                  "O"."TYPE#"<>10 AND "O"."TYPE#"<>11 AND "O"."TYPE#"<>12 AND "O"."TYPE#"<>13 AND "O"."TYPE#"<>14 AND "O"."TYPE#"<>22 AND
                  "O"."TYPE#"<>87 AND "O"."TYPE#"<>88) OR BITAND("U"."SPARE1",16)=0 OR
                  (SYS_CONTEXT('userenv','current_edition_name')='ORA$BASE' AND "U"."TYPE#"<>2) OR ("U"."TYPE#"=2 AND
                  "U"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))) OR  IS NOT NULL))
      55 - access("O"."OWNER#"="U"."USER#")
      57 - access("S"."OWNER"="BU"."NAME" AND "BU"."USER#"="O"."SPARE3")
    PLAN_TABLE_OUTPUT
      59 - access("S"."NAME"="O"."NAME")
      61 - filter("O"."TYPE#"=5)
      63 - access("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id')))
           filter(("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))))
      64 - access("O2"."DATAOBJ#"=:B1 AND "O2"."TYPE#"=88 AND "O2"."OWNER#"="U2"."USER#")
      65 - filter(( IS NOT NULL OR ("S"."NODE" IS NULL AND  IS NOT NULL)))
      67 - access("S"."OBJ#"=:B1)
      68 - filter((("BA"."GRANTOR#"=USERENV('SCHEMAID') OR  IS NOT NULL) AND (("O"."TYPE#"<>4 AND "O"."TYPE#"<>5 AND
                  "O"."TYPE#"<>7 AND "O"."TYPE#"<>8 AND "O"."TYPE#"<>9 AND "O"."TYPE#"<>10 AND "O"."TYPE#"<>11 AND "O"."TYPE#"<>12 AND
                  "O"."TYPE#"<>13 AND "O"."TYPE#"<>14 AND "O"."TYPE#"<>22 AND "O"."TYPE#"<>87 AND "O"."TYPE#"<>88) OR
                  BITAND("U"."SPARE1",16)=0 OR (INTERNAL_FUNCTION("O"."TYPE#") AND
    PLAN_TABLE_OUTPUT
                  ((SYS_CONTEXT('userenv','current_edition_name')='ORA$BASE' AND "U"."TYPE#"<>2) OR ("U"."TYPE#"=2 AND
                  "U"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))) OR  IS NOT NULL)))))
      69 - filter(:B1 IS NULL)
      74 - access("BU"."NAME"=:B1)
      75 - access("BU"."USER#"="O"."SPARE3" AND "O"."NAME"=:B1)
      76 - access("O"."OWNER#"="U"."USER#")
      77 - access("BA"."OBJ#"="O"."OBJ#")
      78 - filter("KZSROROL"=:B1)
      80 - access("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id')))
           filter(("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))))
      81 - access("O2"."DATAOBJ#"=:B1 AND "O2"."TY

  • Oracle online in one node but not on another

    Hi all,
    I have Oracle 9i running in a 2 node windows 2003 cluster. When I try to move oracle from node 2 to node 1 it fails with ora-01017 invalid username or password and come online on node 2.
    I noticed in the event log that in node1 it try to put the database online with user sys and privilege none, but in node 2 the event is with user sys and privilege sysdba.
    Does anyone have any suggestion on what is the problem?
    Thanks,

    Let me try to explain better.
    I opened Windows Cluster Administration and moved the Oracle group to node 1. The group came offline and windows tried to bring it online on node 1 where the error happens and after 3 fails it returns online to node 2.
    I can't say it better because I'm still learning about this.
    I can't give you more information right now because I'm at home and can't access the server but tommorow i'll post the exact error and the software versions to see if it helps.
    Thanks for your time
    DELLS1 is node 1 and DELLS2 is node 2 of cluster
    Oracle is currently running in DELLS2 and this is what happen whe I try to move it to DELLS1
    Database Version
    Oracle9i Enterprise Edition Release 9.2.0.6.0 - Production
    PL/SQL Release 9.2.0.6.0 - Production
    CORE     9.2.0.6.0     Production
    TNS for 32-bit Windows: Version 9.2.0.6.0 - Production
    NLSRTL Version 9.2.0.6.0 - Production
    Event Log
    OracleMSCSServices - Oracle Database resource DLL is about to bring PSI online.
    Oracle.psi - Audit trail: ACTION : 'CONNECT' DATABASE USER: 'SYS' PRIVILEGE : NONE CLIENT USER: PSICLU CLIENT TERMINAL: DELLS1 STATUS: 1017 .
    OracleMSCSServices - ORA-01017: invalid username/password; logon denied
    Error bringing resource PSI online.
    Then after 3 attempts
    OracleMSCSServices - Oracle Database resource DLL is about to bring PSI online.
    Oracle.psi - Audit trail: ACTION : 'CONNECT' DATABASE USER: 'SYS' PRIVILEGE : SYSDBA CLIENT USER: PSICLU CLIENT TERMINAL: DELLS2 STATUS: 0 .
    Oracle.psi - Audit trail: ACTION : 'CONNECT' DATABASE USER: 'SYS' PRIVILEGE : SYSDBA CLIENT USER: PSICLU CLIENT TERMINAL: DELLS2 STATUS: 0 .
    Oracle.psi - Audit trail: ACTION : 'STARTUP' DATABASE USER: 'SYS' PRIVILEGE : SYSDBA CLIENT USER: PSICLU CLIENT TERMINAL: Not Available STATUS: 0 .
    Edited by: Reise on Mar 26, 2009 9:29 AM
    Added info on oracle versions and event log

  • ASM on one node crashes when we start the other two nodes ASM

    We completed database build in Aug 2010
    We complete PSU patching in Jan ending
    Feb 4th the database crashed
    We cannot start ASM on node1
    ASM starts good on node2 and node3 but node1 cannot join
    If ASM is down on node2, node3 then we can start ASM node1Reconfiguration started (old inc 0, new inc 6)
    ASM instance
    List of nodes:
    0 1 2
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    * allocate domain 1, invalid = TRUE
    * allocate domain 2, invalid = TRUE
    Mon Mar 01 16:53:00 2010
    Trace dumping is performing id=[cdmp_20100301165301]
    Mon Mar 01 16:53:55 2010
    ERROR: LMD0 (ospid: 274638) detects an idle connection to instance 2
    Mon Mar 01 16:54:44 2010
    Errors in file /oradb/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_860280.trc (incident=116865):
    ORA-29740: evicted by member 1, group incarnation 8
    Incident details in: /oradb/oracle/diag/asm/+asm/+ASM1/incident/incdir_116865/+ASM1_lmon_860280_i116865.trc
    Errors in file /oradb/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_860280.trc:
    ORA-29740: evicted by member 1, group incarnation 8
    LMON (ospid: 860280): terminating the instance due to error 29740
    Mon Mar 01 16:54:46 2010
    System state dump is made for local instance
    Errors in file /oradb/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_614488.trc (incident=116833):
    ORA-29740: evicted by member , group incarnation
    Incident details in: /oradb/oracle/diag/asm/+asm/+ASM1/incident/incdir_116833/+ASM1_diag_614488_i116833.trc
    Mon Mar 01 16:54:46 2010
    ORA-1092 : opitsk aborting process
    Errors in file /oradb/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_614488.trc:
    ORA-29740: evicted by member , group incarnation
    Trace dumping is performing id=[cdmp_20100301165446]
    Instance terminated by LMON, pid = 860280
    Another thing we found that when we start ASM on node1, the cluster interconnect hangs when we try to ping
    We did modify the cluster_interconnect parameter to try to start using public interface but the issued remained the same and we were not able to ping public interface
    The crs is fine
    $ crs_stat -t
    Name Type Target State Host
    ora....p1.inst application ONLINE OFFLINE
    ora....p2.inst application ONLINE ONLINE noden2
    ora....p3.inst application ONLINE ONLINE noden3
    ora....1p2.srv application ONLINE ONLINE noden2
    ora....1p3.srv application ONLINE ONLINE noden3
    ora.....net.cs application ONLINE ONLINE noden1
    ora.appl.db application ONLINE ONLINE noden1
    ora....SM1.asm application ONLINE OFFLINE
    ora....N1.lsnr application ONLINE ONLINE noden1
    ora....8n1.gsd application ONLINE ONLINE noden1
    ora....8n1.ons application ONLINE ONLINE noden1
    ora....8n1.vip application ONLINE ONLINE noden1
    ora....SM2.asm application ONLINE ONLINE noden2
    ora....N2.lsnr application ONLINE ONLINE noden2
    ora....8n2.gsd application ONLINE ONLINE noden2
    ora....8n2.ons application ONLINE ONLINE noden2
    ora....8n2.vip application ONLINE ONLINE noden2
    ora....SM3.asm application ONLINE ONLINE noden3
    ora....N3.lsnr application ONLINE ONLINE noden3
    ora....8n3.gsd application ONLINE ONLINE noden3
    ora....8n3.ons application ONLINE ONLINE noden3
    ora....8n3.vip application ONLINE ONLINE noden3
    Any inpts can help

    Env
    3-node RAC
    oracle version 11.1.0.7
    Latest PSU Jan applied
    OS is AIX version is 6100-02==========
    LMON trace files
    ==========
    Trace file /oradb/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_860280.trc
    Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
    With the Partitioning, Real Application Clusters, OLAP, Data Mining
    and Real Application Testing options
    ORACLE_HOME = /oradb/oracle/product/11.1/asm_1
    System name:     AIX
    Node name:     host-node1
    Release:     1
    Version:     6
    Machine:     00C39EA44C00
    Instance name: +ASM1
    Redo thread mounted by this instance: 0 <none>
    Oracle process number: 8
    Unix process pid: 860280, image: oracle@host-node1 (LMON)
    *** 2010-03-01 16:50:23.023
    *** SESSION ID:(218.1) 2010-03-01 16:50:23.023
    *** CLIENT ID:() 2010-03-01 16:50:23.023
    *** SERVICE NAME:() 2010-03-01 16:50:23.023
    *** MODULE NAME:() 2010-03-01 16:50:23.023
    *** ACTION NAME:() 2010-03-01 16:50:23.023
    GES resources 5596 pool 6
    GES enqueues 7959
    GES IPC: Receivers 2 Senders 2
    GES IPC: Buffers Receive 1000 Send (i:1150 b:482) Reserve 402
    GES IPC: Msg Size Regular 416 Batch 8192
    Batching factor: enqueue replay 201, ack 224
    Batching factor: cache replay 126 size per lock 64
    kjxggin: CGS tickets = 1000
    kjxgrdmpcpu: CPU Total 6 Core 3 Socket -1 OCPU 6
    kjxgrdmpcpu: High load threshold 21504
    *** 2010-03-01 16:50:23.362
    kjxgmrcfg: Reconfiguration started, type 1
    kjxgmcs: Setting state to 0 0.
    *** 2010-03-01 16:50:23.363
    Name Service frozen
    kjxgmcs: Setting state to 0 1.
    kjxgrdecidever: No old version members in the cluster
    kjxgrssvote: reconfig bitmap chksum 0x88477268 cnt 3 master 0 ret 0
    ksirValidateModuleInfo: action = 10 startup = 0
    Name Service Mode: multi (0x21)
    kjfcpiora: published my fusion master weight 5322
    kjfcpiora: publish my flogb 9
    kjfcpiora: publish my cluster_database_instances parameter=3
    kjxggpoll: change poll time to 50 ms
    kjxgrpropmsg: SSMEMI: inst 1 - no disk vote
    kjxgrpropmsg: SSMEMI: inst 1 - no disk vote
    kjxgrpropmsg: SSMEMI: inst 2 - no disk vote
    SSVOTE: Master indicates no Disk Voting
    kjxgmps: proposing substate 2
    kjxgmcs: Setting state to 6 2.
    kjfmuin: bitmap 0 1 2
    kjfmmhi: received msg from 0 (inc 6)
    kjfmmhi: received msg from 1 (inc 2)
    kjfmmhi: received msg from 2 (inc 4)
    Performed the unique instance identification check
    kjxgmps: proposing substate 3
    kjxgmcs: Setting state to 6 3.
    Name Service recovery started
    Deleted all dead-instance name entries
    kjxgmps: proposing substate 4
    kjxgmcs: Setting state to 6 4.
    Multicasted all local name entries for publish
    Replayed all pending requests
    kjxgmps: proposing substate 5
    kjxgmcs: Setting state to 6 5.
    Name Service normal
    Name Service recovery done
    *** 2010-03-01 16:50:23.889
    *** 2010-03-01 16:50:23.958
    kjxgmps: proposing substate 6
    kjxgmcs: Setting state to 6 6.
    kjxggpoll: change poll time to 600 ms
    2010-03-01 16:50:23.980620 :
    ********* kjfcrfg() called, BEGIN LMON RCFG *********
    kjfcrfg: DRM window size = 0->128 (min lognb = 9)
    2010-03-01 16:50:23.980811 :
    Reconfiguration started (old inc 0, new inc 6)
    ASM instance
    Send timeout: 300 secs
    Defer Queue timeout: 360 secs
    Synchronization timeout: 420 sec
    List of nodes:
    0 1 2
    *** 2010-03-01 16:50:24.023
    2010-03-01 16:50:24.034432 : Global Resource Directory frozen
    node 0
    release 11 1 0 7
    node 1
    release 11 1 0 7
    node 2
    release 11 1 0 7
    number of mastership buckets = 128
    2010-03-01 16:50:24.034959 :
    domain attach called for domid 0
    * kjbdomalc: domain 0 invalid = TRUE
    * kjbdomatt: first attach for domain 0
    asby init, 0/0/x1
    asby returns, 0/0/x1/false
    * Domain maps before reconfiguration:
    * DOMAIN 0 (valid 1): 0
    * End of domain mappings
    * Domain maps after recomputation:
    * DOMAIN 0 (valid 1): 0 1 2
    * End of domain mappings
    Dead inst
    Join inst 0 1 2
    Exist inst
    Active Sendback Threshold = 50 %
    Communication channels reestablished
    2010-03-01 16:50:24.152688 :
    received all domreplay (6.6)
    2010-03-01 16:50:24.152732 :
    sent master 1 (6.6)
    *** 2010-03-01 16:53:00.494
    kjfmReceiverHealthCB_Check: Reciever [0] is healthy.
    2010-03-01 16:52:56.921800 : Received comm error info from 2 (cnt 1)
    kjxgrvalid: valid - 0.1 : (6 6) from 2
    kjxgrrcfgchk: Initiating reconfig, reason=3
    kjxgrrcfgchk: COMM rcfg - Disk Vote Required
    2010-03-01 16:52:57.077877 : kjxgrnetchk: start 0x53001440, end 0x53019ae0
    2010-03-01 16:52:57.077906 : kjxgrnetchk: Sending comm check req to 1
    2010-03-01 16:52:57.078140 : kjxgrnetchk: Sending comm check req to 2
    kjxgrrcfgchk: prev pstate 5 mapsz 512
    kjxgrrcfgchk: new bmp: 0 1 2
    kjxgrrcfgchk: work bmp: 0 1 2
    kjxgrrcfgchk: rr bmp: 0 1 2
    *** 2010-03-01 16:53:00.792
    kjxgmrcfg: Reconfiguration started, type 3
    kjxgmcs: Setting state to 6 0.
    *** 2010-03-01 16:53:00.792
    Name Service frozen
    kjxgmcs: Setting state to 6 1.
    kjxgrdecidever: No old version members in the cluster
    kjxgrmsghndlr: Queue msg (0x110a21e50->0x110f09b90) type 7 for later
    *** 2010-03-01 16:54:43.233
    kjxgrssvote: reconfig bitmap chksum 0x88477268 cnt 3 master 2 ret 0
    kjxgrrcfgchk: disable CGS timeout
    kjxggpoll: change poll time to 50 ms
    * kjfcchknested: CGS rcfg detected in step 7.0.0
    SSVOTE: Master indicates Disk Voting required
    2010-03-01 16:54:37.535518 : kjxgrmsghndlr: evict req from 1 for 0, seq (8, 8) vers 2193970751
    2010-03-01 16:54:37.535587 : kjxgrdtrt: Evicted by 1, seq (8, 8)
    IMR state information
    Member 0, thread -1, state 0x2:c, flags 0x2c48
    RR seq commit 6 cur 8
    Propstate 3 prv 2 pending 0
    rcfg rsn 3, rcfg time 1392514113, mem ct 3
    master 2, master rcfg time 1392479783
    evicted memcnt 0, starttm 0 chkcnt 0
    system load 241 (normal)
    Member information:
    Member 0, incarn 6, version 0x82c5563f, thrd -1
    prev thrd -1, status 0x1203 (JR..), err 0x0000
    Member 1, incarn 6, version 0x82c1073b, thrd 2
    prev thrd -1, status 0x1007 (JRM.), err 0x0002
    Member 2, incarn 6, version 0x82c114ee, thrd 3
    prev thrd -1, status 0x0007 (JRM.), err 0x0000
    =====================================================
    Group name: +ASM
    Member id: 0
    Cached KGXGN event: 0
    Group State:
    State: 6 1
    Reconfig started start-tm 0x4b8c373c tmout period 0xffffffff state 0x2
    Reconfig INPG type 3 inc 6 rsn 0 data 0x0
    Reconfig COMP type 1 inc 6 rsn 0 data 0x0
    Commited Map: 0 1 2
    New Map: 0 1 2
    KGXGN Map: 0 1 2
    KGXGN Map2: 0 1 2
    Master node: 0
    Memcnt 3 Rcvcnt 0
    Substate Proposal: false
    Inc Proposal:
    incarn 0 memcnt 0 master 0
    proposal false matched false
    map:
    Master Inc State:
    incarn 0 memcnt 0 agrees 0 flag 0x1
    wmap:
    nmap:
    ubmap:
    Substate Handler Execution State
    substate 0 status done
    substate 1 status done
    substate 2 status done
    substate 3 status done
    substate 4 status done
    substate 5 status done
    substate 6 status done
    IMR hist: 20[0x0a00:0x53019b0e] 4[0x0007:0x53019b0e] 3[0x0006:0x53019b0e]
    IMR hist: 20[0x0902:0x53019b0e] 20[0x0702:0x53019b0b] 20[0x0702:0x53019b0a]
    IMR hist: 20[0x0702:0x53019b0a] 1[0x0006:0x53019b0a] 20[0x0702:0x53019aff]
    IMR hist: 10[0x0006:0x52fdbdb1] 20[0x0b00:0x52fdbdb1] 9[0x0006:0x52fdbdaf]
    IMR hist: 20[0x0a02:0x52fdbdaf] 20[0x0a01:0x52fdbce1] 20[0x0a00:0x52fdbc8a]
    IMR hist: 4[0x0005:0x52fdbc86] 3[0x0004:0x52fdbc4c] 20[0x0900:0x52fdbc4c]
    IMR hist: 20[0x0802:0x52fdbc08] 20[0x0801:0x52fdbc08] 20[0x0801:0x52fdbc08]
    IMR hist: 20[0x0602:0x52fdbc08] 20[0x0601:0x52fdbc08] 20[0x0601:0x52fdbc08]
    IMR hist: 20[0x0800:0x52fdbc08] 20[0x0700:0x52fdbc08] 20[0x0602:0x52fdbc07]
    IMR hist: 20[0x0800:0x52fdbc07] 20[0x0700:0x52fdbc07] 1[0x0000:0x52fdbbb8]
    IMR hist: 0[0x0000:0x00000000] 0[0x0000:0x00000000]
    KJM HIST LMD0:
    7:0 6:0 5:7:0 12:97697 7:0 6:0 5:7:0 12:97696 7:0 6:0
    5:7:0 12:97703 7:0 6:0 5:7:0 2:0 1:0 12:97713 7:0 6:0
    5:7:0 12:97766 7:0 6:0 5:7:0 12:97782 7:0 6:0 5:7:0 12:97778
    7:0 6:0 5:7:0 12:97799 7:0 6:0 5:7:0 12:97771 7:0 6:0
    5:7:0 12:97784 7:0 6:0 5:7:0 12:97805 7:0 6:0 5:7:0 12:97785
    7:0 6:0 5:7:0 12:97757 7:0 6:0 5:7:0 12:97770 7:0 6:0
    5:7:0 12:97784 7:0 6:0
    KJM HIST LMS0:
    7:0 6:0 5:7:0 10:0 12:97697 7:0 6:0 5:7:0 10:0 12:97696
    7:0 6:0 5:7:0 10:0 12:97703 7:0 6:0 5:7:0 10:0 12:97713
    7:0 6:0 5:7:0 10:0 2:0 12:97766 7:0 6:0 5:7:0 10:0
    12:97782 7:0 6:0 5:7:0 10:0 12:97778 7:0 6:0 5:7:0 10:0
    12:97799 7:0 6:0 5:7:0 10:0 12:97771 7:0 6:0 5:7:0 10:0
    12:97784 7:0 6:0 5:7:0 10:0 12:97805 7:0 6:0 5:7:0 10:0
    12:97785 7:0 6:0 5:7:0
    DUMP state for lmd0 (ospid 274638)
    DUMP IPC context for lmd0 (ospid 274638)
    Dumping process 9.274638 info:
    *** 2010-03-01 16:54:43.664
    Process diagnostic dump for oracle@host-node1 (LMD0), OS id=274638,
    pid: 9, proc_ser: 1, sid: 217, sess_ser: 1
    loadavg : 1.72 1.07 0.90
    swap info: free_mem = 28642.09M rsv = 16.00M
    alloc = 21.13M avail = 4096.00M swap_free = 4074.87M
    F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
    240001 A oracle 274638 1 0 60 20 12ca9f590 156060 16:50:23 - 0:00 asm_lmd0_+ASM1
    Short stack dump:
    <-ksedsts()+0254<-ksdxfstk()+0028<-ksdxcb()+05d8<-sspuser()+0074<-4750<-poll()+000c<-sskgxp_select()+00e4<-skgxpiwait()+08a4<-skgxpwait()+06fc<-ksxpwait()+081c<-ksliwat()+0a58<-kslwaitctx()+0150<-kslwait()+006c<-ksxprcvimd()+0368<-kjctr_rksxp()+013c<-kjctrcv()+0160<-kjcsrmg()+005c<-kjmdm()+2454<-ksbrdp()+075c<-opirip()+0444<-opidrv()+0414<-sou2o()+0090<-opimai_real()+0148<-main()+0090<-__start()+0070
    Process diagnostic dump actual duration=0.161000 sec
    (max dump time=30.000000 sec)
    *** 2010-03-01 16:54:43.825
    SO: 0x70000001ff913a0, type: 2, owner: 0x0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x70000001ff913a0, name=process, file=ksu.h LINE:10706 ID:, pg=0
    (process) Oracle pid:9, ser:1, calls cur/top: 0x70000001f733140/0x70000001f733140
    flags : (0x6) SYSTEM
    flags2: (0x100), flags3: (0x0)
    int error: 0, call error: 0, sess error: 0, txn error 0
    ksudlp FALSE at location: 0
    (post info) last post received: 0 0 83
    last post received-location: kji.h LINE:2369 ID:kjga: clear wait for lmon
    last process to post me: 70000001ff903b0 1 6
    last post sent: 0 0 25
    last post sent-location: ksa2.h LINE:282 ID:ksasnd
    last process posted by me: 70000001ff903b0 1 6
    (latch info) wait_event=68 bits=0
    Process Group: DEFAULT, pseudo proc: 0x70000001f4851d0
    O/S info: user: oracle, term: UNKNOWN, ospid: 274638
    OSD pid info: Unix process pid: 274638, image: oracle@host-node1 (LMD0)
    Dump of memory from 0x070000001FF70038 to 0x070000001FF70240
    70000001FF70030 00000000 00000000 [........]
    70000001FF70040 00000000 00000000 00000000 00000000 [................]
    Repeat 31 times
    SO: 0x70000001f6de4a0, type: 4, owner: 0x70000001ff913a0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x70000001ff913a0, name=session, file=ksu.h LINE:10719 ID:, pg=0
    (session) sid: 217 ser: 1 trans: 0x0, creator: 0x70000001ff913a0
    flags: (0x51) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
    flags2: (0x408) -/-
    DID: , short-term DID:
    txn branch: 0x0
    oct: 0, prv: 0, sql: 0x0, psql: 0x0, user: 0/SYS
    ksuxds FALSE at location: 0
    service name: SYS$BACKGROUND
    Current Wait Stack:
    0: waiting for 'ges remote message'
    waittime=40, loop=0, p3=44
    wait_id=2613 seq_num=2614 snap_id=1
    wait times: snap=0.018269 sec, exc=0.018269 sec, total=0.018269 sec
    wait times: max=0.080000 sec
    wait counts: calls=1 os=1
    in_wait=1 iflags=0x5a8
    Wait State:
    auto_close=0 flags=0x22 boundary=0x0/-1
    Session Wait History:
    0: waited for 'ges remote message'
    waittime=40, loop=0, p3=44
    wait_id=2612 seq_num=2613 snap_id=1
    wait times: snap=0.160172 sec, exc=0.160172 sec, total=0.160172 sec
    wait times: max=0.080000 sec
    wait counts: calls=1 os=1
    occurred after 0.000008 sec of elapsed time
    1: waited for 'ges remote message'
    waittime=40, loop=0, p3=44
    wait_id=2611 seq_num=2612 snap_id=1
    wait times: snap=0.096359 sec, exc=0.096359 sec, total=0.096359 sec
    wait times: max=0.080000 sec
    wait counts: calls=1 os=1
    occurred after 0.000008 sec of elapsed time
    2: waited for 'ges remote message'
    waittime=40, loop=0, p3=44
    wait_id=2610 seq_num=2611 snap_id=1
    wait times: snap=0.098065 sec, exc=0.098065 sec, total=0.098065 sec
    wait times: max=0.080000 sec
    wait counts: calls=1 os=1
    occurred after 0.000007 sec of elapsed time
    3: waited for 'ges remote message'
    waittime=40, loop=0, p3=44
    wait_id=2609 seq_num=2610 snap_id=1
    wait times: snap=0.097831 sec, exc=0.097831 sec, total=0.097831 sec
    wait times: max=0.080000 sec
    wait counts: calls=1 os=1
    occurred after 0.000014 sec of elapsed time
    4: waited for 'ges remote message'
    waittime=40, loop=0, p3=44
    wait_id=2608 seq_num=2609 snap_id=1
    wait times: snap=0.095876 sec, exc=0.095876 sec, total=0.095876 sec
    wait times: max=0.080000 sec
    wait counts: calls=1 os=1
    occurred after 0.000008 sec of elapsed time
    5: waited for 'ges remote message'
    waittime=40, loop=0, p3=44
    wait_id=2607 seq_num=2608 snap_id=1
    wait times: snap=0.098788 sec, exc=0.098788 sec, total=0.098788 sec
    wait times: max=0.080000 sec
    wait counts: calls=1 os=1
    occurred after 0.000006 sec of elapsed time
    6: waited for 'ges remote message'
    waittime=40, loop=0, p3=44
    wait_id=2606 seq_num=2607 snap_id=1
    wait times: snap=0.098854 sec, exc=0.098854 sec, total=0.098854 sec
    wait times: max=0.080000 sec
    wait counts: calls=1 os=1
    occurred after 0.000007 sec of elapsed time
    7: waited for 'ges remote message'
    waittime=40, loop=0, p3=44
    wait_id=2605 seq_num=2606 snap_id=1
    wait times: snap=0.098040 sec, exc=0.098040 sec, total=0.098040 sec
    wait times: max=0.080000 sec
    wait counts: calls=1 os=1
    occurred after 0.000008 sec of elapsed time
    8: waited for 'ges remote message'
    waittime=40, loop=0, p3=44
    wait_id=2604 seq_num=2605 snap_id=1
    wait times: snap=0.097322 sec, exc=0.097322 sec, total=0.097322 sec
    wait times: max=0.080000 sec
    wait counts: calls=1 os=1
    occurred after 0.000007 sec of elapsed time
    9: waited for 'ges remote message'
    waittime=40, loop=0, p3=44
    wait_id=2603 seq_num=2604 snap_id=1
    wait times: snap=0.097334 sec, exc=0.097334 sec, total=0.097334 sec
    wait times: max=0.080000 sec
    wait counts: calls=1 os=1
    occurred after 0.000008 sec of elapsed time
    Sampled Session History
    The sampled session history is constructed by sampling
    the target session every 1 second. The sampling process
    captures at each sample if the session is in a non-idle wait,
    an idle wait, or not in a wait. If the session is in a
    non-idle wait then one interval is shown for all the samples
    the session was in the same non-idle wait. If the
    session is in an idle wait or not in a wait for
    consecutive samples then one interval is shown for all
    the consecutive samples. Though we display these consecutive
    samples in a single interval the session may NOT be continuously
    idle or not in a wait (the sampling process does not know).
    The history is displayed in reverse chronological order.
    sample interval: 1 sec, max history 120 sec
    KSFD PGA DUMPS
    Number of completed I/O requests=0 flags=0
    END OF PROCESS STATE
    LMON IPC context:
    ksxpdmp: facility 0 (?) (0x1, 0x0) counts 0, 0
    ksxpdmp: Dumping the osd context
    SKGXP: SKGXPCTX: 0x1103bfb58 ctx
    SKGXP:
    SKGXP: WAIT HISTORY
    SKGXP: Time(msec)     Wait Type     Return Code
    SKGXP: ----------     ---------     ------------
    SKGXP: 0          NORMAL          SUCC
    SKGXP: 0          NORMAL          SUCC
    SKGXP: 0          NORMAL          SUCC
    SKGXP: 0          NORMAL          SUCC
    SKGXP: 0          NORMAL          TIMEDOUT
    SKGXP: 12          NORMAL          TIMEDOUT
    SKGXP: 0          NORMAL          TIMEDOUT
    SKGXP: 20          NORMAL          TIMEDOUT
    SKGXP: 0          NORMAL          TIMEDOUT
    SKGXP: 19          NORMAL          TIMEDOUT
    SKGXP: 0          NORMAL          TIMEDOUT
    SKGXP: 20          NORMAL          TIMEDOUT
    SKGXP: 0          NORMAL          TIMEDOUT
    SKGXP: 19          NORMAL          TIMEDOUT
    SKGXP: 0          NORMAL          TIMEDOUT
    SKGXP: 20          NORMAL          TIMEDOUT
    SKGXP: wait delta 0 sec (27 msec) ctx ts 0x3e377 last ts 0x3e381
    SKGXP: user cpu time since last wait 0 sec 0 ticks
    SKGXP: system cpu time since last wait 0 sec 0 ticks
    SKGXP: locked 1
    SKGXP: blocked 51
    SKGXP: timed wait receives 0
    SKGXP: admno 0x485303b1 admport:
    SKGXP: SSKGXPT 0x103c0a74 flags sockno 12 IP 192.168.253.49 UDP 49777
    SKGXP: context timestamp 0x3e377
    SKGXP: buffers queued on port 1105aa950
    SKGXP:
    SKGXP: Dumping Connection Handle Table
    SKGXP: sconno accono ertt state seq# RcvPid TotCreditsSKGXP: sent rtrans acks
    SKGXP: CNH Table Bucket: 10
    SKGXP: 0x339d0248 0x6dd6841c 64 4 32838 589900 8SKGXP: 75d 5d 32838d
    SKGXP: CNH Table Bucket: 11
    SKGXP: 0x339d0249 0x75ef4c98 32 4 32811 1007758 8SKGXP: 48d 12d 32811d
    SKGXP: CNH Table Bucket: 12
    SKGXP: 0x339d024a 0x75703ec2 16 4 32763 524518 8SKGXP: 0d 0d 0d
    SKGXP: CNH Table Bucket: 13
    SKGXP: 0x339d024b 0x41094259 16 4 32763 520260 8SKGXP: 0d 0d 0d
    SKGXP: CNH Table Bucket: 14
    SKGXP: 0x339d024c 0x7c1c696c 16 4 32763 585808 8SKGXP: 0d 0d 0d
    SKGXP: CNH Table Bucket: 15
    SKGXP: 0x339d024d 0x138c8c4a 16 4 32763 843952 8SKGXP: 0d 0d 0d
    SKGXP:
    SKGXP: Dumping Accept Handle Table
    SKGXP: ach accono sconno admno state SndPid seq# rcv rtrans acks credits
    SKGXP: ACH Table Bucket: 1472
    SKGXP: 0x111088010 0x48cb4387 0x3365b236 0x1fe7dc68 40 1007758 32812 49 0 26 8
    SKGXP: ACH Table Bucket: 1474
    SKGXP: 0x11108b730 0x48cb4389 0x1c69654a 0x7183ff4c 40 589900 32838 75 0 52 8
    Incident 116865 created, dump file: /oradb/oracle/diag/asm/+asm/+ASM1/incident/incdir_116865/+ASM1_lmon_860280_i116865.trc
    ORA-29740: evicted by member 1, group incarnation 8
    error 29740 detected in background process
    ORA-29740: evicted by member 1, group incarnation 8
    *** 2010-03-01 16:54:46.430
    LMON (ospid: 860280): terminating the instance due to error 29740
    ksuitm: waiting up to [5] seconds before killing DIAG
    ==========
    DIAG trace files
    =========
    Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
    With the Partitioning, Real Application Clusters, OLAP, Data Mining
    and Real Application Testing options
    ORACLE_HOME = /oradb/oracle/product/11.1/asm_1
    System name:     AIX
    Node name:     host-node1
    Release:     1
    Version:     6
    Machine:     00C39EA44C00
    Instance name: +ASM1
    Redo thread mounted by this instance: 0 <none>
    Oracle process number: 4
    Unix process pid: 614488, image: oracle@host-node1 (DIAG)
    *** 2010-03-01 16:50:22.947
    *** SESSION ID:(222.1) 2010-03-01 16:50:22.947
    *** CLIENT ID:() 2010-03-01 16:50:22.947
    *** SERVICE NAME:() 2010-03-01 16:50:22.947
    *** MODULE NAME:() 2010-03-01 16:50:22.947
    *** ACTION NAME:() 2010-03-01 16:50:22.947
    Node id: 0
    List of nodes: 0, 1, 2,
    *** 2010-03-01 16:50:22.948
    Reconfiguration starts [incarn=0]
    *** 2010-03-01 16:50:22.948
    I'm the master node
    Group reconfiguration cleanup
    *** 2010-03-01 16:50:23.602
    A rcfg proposal from node 2 is received
    *** 2010-03-01 16:50:23.602
    A rcfg proposal from node 1 is received
    *** 2010-03-01 16:50:23.602
    Reconfiguration completes [incarn=3]
    *** 2010-03-01 16:53:00.877
    A dump event msg is rcv'd
    REQUEST:trace dump in directory cdmp_20100301165301
    *** 2010-03-01 16:53:00.877
    Trace dumping is performing id=[cdmp_20100301165301]....
    *** 2010-03-01 16:53:01.041
    Trace dumping is done
    *** 2010-03-01 16:54:46.560
    Instance is terminating by process 860280 [ospid=oracle@host-node1 (LMON)]
    Performing diagnostic data dump for this instance
    Incident 116833 created, dump file: /oradb/oracle/diag/asm/+asm/+ASM1/incident/incdir_116833/+ASM1_diag_614488_i116833.trc
    ORA-29740: evicted by member , group incarnation
    Error 29740 encountered during system state dump
    *** 2010-03-01 16:54:49.280
    ----- Error Stack Dump -----
    ORA-29740: evicted by member , group incarnation
    *** 2010-03-01 16:54:49.281
    Trace dumping is performing id=[cdmp_20100301165446]....
    *** 2010-03-01 16:54:49.433
    Trace dumping is done

Maybe you are looking for

  • Is empty string and Null same?

    create table x (empid number); Table created. insert into x values (''); 1 row created. insert into x values (null); 1 row created. SQL> set null <<>> SQL> select * from x;      EMPID <<>> <<>>So i can safely change an Insert statement like insert in

  • IPhone 6 Plus screen rotation issue

    The home and lock screens on my wife's iPhone 6 plus rotate.  Can this be shut off?  And if I turn on the screen rotation lock, will it stop rotation of the lock and home screens as well as the regular use screens (i.e. - apps, Safari, etc...)?  Can

  • Another last problem, thanks

    hi, I have another question, here's goes, I will need to make a frame that inside there will be text field, one button, and label and when you click on the button what is written inside the textfield will go to the label. something like this: textfie

  • Screen Problem..Help?

    Hello, I have my iphone now for 6 months now and just yesterday I noticed a bright spot on the screen. I wanst looking for it, it is just very noticible. It is in the upper left and shows up on all screens/images. I cant provide a pic, because my cam

  • Wwctx_api.NO_SESSION_EXCEPTION

    Hi, I have the following problem (I'm very new to writing portlets, so maybe I'm just overlooking something stupid..?): I have created a new schema "portal_pckg" (via 'Create new schema' in Portal) in which we are planning to store plsql-packages to