SQLDeveloper, db links, and ora-12514

Question

There are several ducks to line up here, so please bear with me.
First, the lineup, the batting order …
Workstation WKS_DBA; Win 7 Pro 64bit; Oracle 10g 32bit client; Oracle 11g 64bit client (in that order in PATH); SQL Developer version 3.2.20.09, build MAIN-09.87
Workstation WKS_OLD; Win 7 Pro 32bit; Oracle 10g 32bit client; SQL Developer version 3.0.04, build MAIN-04.34
Workstation WKS-NEW; Win 7 Pro 64bit; 11g 32bit client; 11g 64bit client (in that order in PATH).
Database server SVRLCL; Oracle Linux 5.6 x86-64; Oracle database 10.2.0.4 64bit (let’s call it DBLCL); database link LK_DBRMT pointing to ..
Database server SVRRMT; Windows server 2003 R2, SP2, 32bit; Oracle database 10.2.0.4 (let’s call it DBRMT).
The situation: Connected to DBLCL, we have an UPDATE statement with a WHERE clause that includes a SELECT on a table @LK_DBRMT.
If the statement is executed by SQLDeveloper on WKS_DBA, it works.
If the statement is executed by SQLDeveloper on WKS_OLD, it works.
If the statement is executed by SQLDeveloper on WKS_NEW, it returns an ora-12514, pointing to the db link.
What’s equally strange is that from WKS_OLD and WKS_NEW, SQLDev is sending some CREATE DATABASE LINK statements prior to sending the UPDATE statement. When coming from WKS_NEW, that supposed db link creation includes SERVICE_NAME=null. Here is a sequence of packets received by database DBLCL from SQLDev, beginning with the last query that SQLDev issued on its own behalf, the select from dba.xml_schemas.  The only statement we asked sqldev to execute was the UPDATE.
(obviously, the trace has been edited in several ways for brevity and readability. Actualy packet contents include only key clear text to identify what was going on.)
15:02:26:948]     nttrd:      socket 24 had bytes read=112
15:02:26:948]     nsprecv:    |..select.1.from.dba_xml_schemas.|
15:02:26:949]     nttwr:      socket 24 had bytes written=159
15:02:26:949]     nspsend:    |........|
15:02:31:124]     nttrd:      socket 24 had bytes read=277
15:02:31:124]     nsprecv:      |....CREATE.DATABASE.LINK.LK_DBRMT..CONNECT.TO.kilroy.IDENTIFIED.BY.mypswd..USING.'(DESCRIPTION.=(ADDRESS_LIST.=(ADDRESS.=.(PROTOCOL.=.TCP)(HOST.=.172.nn.nnn.88)(PORT.=.1521)))(CONNECT_DATA.=.(SERVICE_NAME.=.null))).'..|
15:02:31:128]     nttwr:      socket 24 had bytes written=60
15:02:31:128]     nspsend:    |.<............................D.............................    |
15:02:31:129]     nttrd:      socket 24 had bytes read=269
15:02:31:129]     nsprecv:      |....CREATE.DATABASE.LINK.LK_DBRMT..CONNECT.TO.kilroy.IDENTIFIED.BY.mypswd..USING.'(DESCRIPTION.=(ADDRESS_LIST.=(ADDRESS.=.(PROTOCOL.=.TCP)(HOST.=.172.nn.nnn.88)(PORT.=.1521)))(CONNECT_DATA.=.(SERVICE_NAME.=.null))).'..|
15:02:31:129]     nttwr:      socket 24 had bytes written=11
15:02:31:129]     nspsend:    |...........     |
15:02:31:130]     nttwr:      socket 24 had bytes written=11
15:02:31:130]     nspsend:    |...........     |
15:02:31:130]     nttrd:      socket 24 had bytes read=11
15:02:31:130]     nsprecv:    |...........     |
15:02:31:130]     nttwr:      socket 24 had bytes written=84
15:02:31:130]     nspsend:    |...(ORA-02011:.duplicate.database.link.name.    |
15:02:31:131]     nttrd:      socket 24 had bytes read=269
15:02:31:131]     nsprecv:      |....CREATE.DATABASE.LINK.LK_DBRMT..CONNECT.TO.kilroy.IDENTIFIED.BY.mypswd..USING.'(DESCRIPTION.=(ADDRESS_LIST.=(ADDRESS.=.(PROTOCOL.=.TCP)(HOST.=.172.nn.nnn.88)(PORT.=.1521)))(CONNECT_DATA.=.(SERVICE_NAME.=.null))).'..|
15:02:31:132]     nttwr:      socket 24 had bytes written=11
15:02:31:132]     nspsend:    |...........     |
15:02:31:132]     nttwr:      socket 24 had bytes written=11
15:02:31:132]     nspsend:    |...........     |
15:02:31:132]     nttrd:      socket 24 had bytes read=11
15:02:31:132]     nsprecv:    |...........     |
15:02:31:132]     nttwr:      socket 24 had bytes written=84
15:02:31:132]     nspsend:    |...(ORA-02011:.duplicate.database.link.name.    |
15:02:31:134]     nttrd:      socket 24 had bytes read=848
15:02:31:134]     nsprecv:    |.....update.TRANS_DETAIL.RT.|
15:02:31:138]     nttwr:      socket 11 had bytes written=227
15:02:31:138]     nspsend:      |..(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=172.nn.nnn.88)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=null)(CID=(PROGRAM=oracle)(HOST=svrlcl.myorg.org)(USER=KIlroy))))     |
15:02:31:138]     nttrd:      socket 11 had bytes read=103
15:02:31:138]     nsprecv:      "|""..[(DESCRIPTION=(TMP=)(VSNNUM=169870336)(ERR=12514)(ERROR_STACK=(ERROR=(CODE=12514)(EMFI=4)))) |
15:02:31:139]     nttwr:      socket 24 had bytes written=11
15:02:31:139]     nspsend:    |...........     |
15:02:31:139]     nttwr:      socket 24 had bytes written=11
15:02:31:140]     nspsend:    |...........     |
15:02:31:140]     nttrd:      socket 24 had bytes read=11
15:02:31:140]     nsprecv:    |...........     |
15:02:31:140]     nttwr:      socket 24 had bytes written=136
15:02:31:140]     nspsend:    |....[ORA-12514:.TNS:listener.does.not.currently.know.of.service.requested.in.connect.descriptor.|
Here is the same sequence, using WKS_OLD
09:25:34:694]     nttrd:      socket 24 had bytes read=112
09:25:34:694]     nsprecv:    |..select.1.from.dba_xml_schemas.|
09:25:34:694]     nttwr:      socket 24 had bytes written=159
09:25:34:694]     nspsend:    |........|
09:25:39:883]     nttrd:      socket 24 had bytes read=279
09:25:39:883]     nsprecv:      |....CREATE.DATABASE.LINK.LK_DBRMT..CONNECT.TO.kilroy.IDENTIFIED.BY.mypswd..USING.'(DESCRIPTION.=(ADDRESS_LIST.=(ADDRESS.=.(PROTOCOL.=.TCP)(HOST.=.172.nn.nnn.88)(PORT.=.1521)))(CONNECT_DATA.=.(SERVICE_NAME.=.DBRMT))).'|
09:25:39:979]     nttwr:      socket 24 had bytes written=60
09:25:39:979]     nspsend:    |.<......|
09:25:39:980]     nttrd:      socket 24 had bytes read=271
09:25:39:980]     nsprecv:      |....CREATE.DATABASE.LINK.LK_DBRMT..CONNECT.TO.kilroy.IDENTIFIED.BY.mypswd..USING.'(DESCRIPTION.=(ADDRESS_LIST.=(ADDRESS.=.(PROTOCOL.=.TCP)(HOST.=.172.nn.nnn.88)(PORT.=.1521)))(CONNECT_DATA.=.(SERVICE_NAME.=.DBRMT))).'|
09:25:39:981]     nttwr:      socket 24 had bytes written=11
09:25:39:981]     nspsend:    |...........     |
09:25:39:981]     nttwr:      socket 24 had bytes written=11
09:25:39:981]     nspsend:    |...........     |
09:25:39:981]     nttrd:      socket 24 had bytes read=11
09:25:39:981]     nsprecv:    |...........     |
09:25:39:982]     nttwr:      socket 24 had bytes written=84
09:25:39:982]     nspsend:    |...(ORA-02011:.duplicate.database.link.name.    |
09:25:39:983]     nttrd:      socket 24 had bytes read=271
09:25:39:983]     nsprecv:      |....CREATE.DATABASE.LINK.LK_DBRMT..CONNECT.TO.kilroy.IDENTIFIED.BY.mypswd..USING.'(DESCRIPTION.=(ADDRESS_LIST.=(ADDRESS.=.(PROTOCOL.=.TCP)(HOST.=.172.nn.nnn.88)(PORT.=.1521)))(CONNECT_DATA.=.(SERVICE_NAME.=.DBRMT))).'|
09:25:39:984]     nttwr:      socket 24 had bytes written=11
09:25:39:984]     nspsend:    |........... |
09:25:39:985]     nttwr:      socket 24 had bytes written=11
09:25:39:985]     nspsend:    |...........     |
09:25:39:985]     nttrd:      socket 24 had bytes read=11
09:25:39:985]     nsprecv:    |...........     |
09:25:39:985]     nttwr:      socket 24 had bytes written=84
09:25:39:986]     nspsend:    |...(ORA-02011:.duplicate.database.link.name.    |
09:25:39:988]     nttrd:      socket 24 had bytes read=864
09:25:39:988]     nsprecv:    |.....update.ROY_TRANS_DETAIL.RT.|
09:25:40:015]     nttwr:      socket 12 had bytes written=229
09:25:40:015]     nspsend:      |..(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=172.nn.nnn.88)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=DBRMT)(CID=(PROGRAM=oracle)(HOST=svrlcl.myorg.org)(USER=KIlroy))))   |
09:25:40:020]     nttrd:      socket 12 had bytes read=8
09:25:40:020]     nsprecv:    |........|
09:25:40:020]     nttwr:      socket 12 had bytes written=229
09:25:40:020]     nspsend:      |..(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=172.nn.nnn.88)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=DBRMT)(CID=(PROGRAM=oracle)(HOST=svrlcl.myorg.org)(USER=KIlroy))))   |
09:25:40:021]     nttrd:      socket 12 had bytes read=32
09:25:40:021]     nsprecv:    |........|
09:25:40:021]     nttwr:      socket 12 had bytes written=38
09:25:40:021]     nspsend:    |..x86_64/Linux.2.4.xx.  |
09:25:40:021]     nttrd:      socket 12 had bytes read=179
09:25:40:021]     nsprecv:    |.....IBMPC/WIN_NT-8.1.0.|
And the same sequence when executing from WKS_DBA
[10-JUL-2013      14:34:24:546]     nttrd:      socket 24 had bytes read=111
[10-JUL-2013      14:34:24:546]     nsprecv:    |.select.1.from.dba_xml_schemas.w|
[10-JUL-2013      14:34:24:547]     nttwr:      socket 24 had bytes written=159
[10-JUL-2013      14:34:24:547]     nspsend:    |........|
[10-JUL-2013      14:34:45:262]     nttrd:      socket 24 had bytes read=856
[10-JUL-2013      14:34:45:262]     nsprecv:    |.....update.ROY_TRANS_DETAIL.RT.|
[10-JUL-2013      14:34:45:267]     nttwr:      socket 11 had bytes written=226
[10-JUL-2013      14:34:45:267]     nspsend:      |..(DESCRIPTION=(address=(protocol=tcp)(host=svrrmt)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=DBRMT)(CID=(PROGRAM=oracle)(HOST=svrlcl.myorg.org)(USER=estevens))))      |
[10-JUL-2013      14:34:45:272]     nttrd:      socket 11 had bytes read=8
[10-JUL-2013      14:34:45:272]     nsprecv:   
[10-JUL-2013      14:34:45:272]     nsprecv:    |........|
[10-JUL-2013      14:34:45:272]     nttwr:      socket 11 had bytes written=226
[10-JUL-2013      14:34:45:272]     nspsend:      |..(DESCRIPTION=(address=(protocol=tcp)(host=svrrmt)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=DBRMT)(CID=(PROGRAM=oracle)(HOST=svrlcl.myorg.org)(USER=estevens))))      |
[10-JUL-2013      14:34:45:273]     nttrd:      socket 11 had bytes read=32
[10-JUL-2013      14:34:45:273]     nsprecv:    |........|
[10-JUL-2013      14:34:45:273]     nttwr:      socket 11 had bytes written=38
[10-JUL-2013      14:34:45:273]     nspsend:    |..x86_64/Linux.2.4.xx.  |
[10-JUL-2013      14:34:45:273]     nttrd:      socket 11 had bytes read=179
[10-JUL-2013      14:34:45:273]     nsprecv:    |.....IBMPC/WIN_NT-8.1.0.|
Several questions suggest themselves, not the least of which are
1- Why is SQLDev even issuing those CREATE DATABASE LINK statements in the first place?
2- Where is it getting the information with which to do that?
3- Why is the same version of SQLDev behaving differently on WKS_DBA and WKS_NEW ?

Answer 1

rp0428 wrote:
>
On WKS_NEW, the UPDATE returns an ora-12514 on the reference to the db_link.  Of course the db_link is at the database so at first blush the client should have not bearing.
>
I don't know why sql developer should be sending CREATE DATABASE LINK ddl at all so the dev team should look into this. Gary's attempt to get a response from Vadim didnt' work.
The only 'common' thing I see is that on WKS_NEW you have the 32 bit Oracle client ahead of the 64 bit client; the others don't have that.
Ok, on Friday, I had un-restricted access to the problem workstation (developer was out of office).  I completely removed all oracle clients, then started re-installing and testing. I removed and reinstalled only the Oracle clients, not SQL Dev.  All removals were 'ripping it out by the roots' - deleting the installation directories (c:\app\oracle  etc), the inventory directories (c:\Program Files\oracle and c:\Program Files (x86)\oracle) and the registry entires (hklm\software\oracle and hklm\software\Wow6432Node\oracle).
First I installed just the 32-bit. SQL Dev choked and said 'cannot load 32-bit libraries'
Removed the 32 and installed the 64 bit.  Ran the tests and got the same results as before .. SELECT works, SELECT in the UPDATE fails
Removed 64, installed 32 then 64. Ran the tests and got the same results as before .. SELECT works, SELECT in the UPDATE fails
Reversed the PATH entries, putting 32-bit first. Ran the tests and got the same results as before .. SELECT works, SELECT in the UPDATE fails
Which begs the question: what type of connection are you using: OCI or thin? If OCI try using the THIN driver to see if you get the same issue and log entries.
If that selection is made by setting the connection properties as "basic" (thin?) or "tns" (OCI?) then yes, tried it both ways, no change.
As to other things to try I would suggest more tests to try to isolate the offending part a little more.
Test various aspects of the DB_LINK itself to see which use cases work and which do not. Many of these can just be done from WKS_NEW
1. Try a plain SELECT * from myTable@dblink - tests if the DB link itself even works in the FROM clause with NO local table reference.
2. Try a similar SELECT * and add a second remote table in the WHERE clause - that is ALL tables and conditions are from the remote server.
3. Try a SELECT from a local table that joins the remote table and db link in the FROM clause rather than the WHERE clause
I think the original pretty well covers the bases.  Here's the actual full update statement (with certain names made generic for public posting)
update LOCAL_TRANS_DETAIL RT
   set rt.sent_to_remoteapp_date = sysdate
where rt.transactionid in
(select rt.transactionid
   FROM LOCAL_TRANS_DETAIL RT
   JOIN LOCALSCHEMA.RMT_MASTER_ITEMS RMI ON RT.NEW_ITEM_NUM = RMI.NEW_ITEM_NUM
   JOIN RMTSCHEMA_A.PAC_PRODUCT_NUMBER@RMTDB PPN ON RT.NEW_ITEM_NUM = PPN.PRODUCT_NUMBER
   JOIN RMTSCHEMA_A.PAC_PRODUCT_NUMBER@RMTDB PPN2 ON PPN.PRODUCT_ID = PPN2.PRODUCT_ID AND PPN2.PRIMARY_SW = 'Y'
   JOIN RMTSCHEMA_B.PRD_IMPRESSION@RMTDB PI ON PI.ORDER_NO = PPN2.PRODUCT_NUMBER
WHERE RT.SENT_TO_remoteapp_DATE is NULL and
      rt.order_id = 'W4011693' and
      rt.item_ship_date < '01-AUG-2013')
4. Try removing the 32 bit Oracle client form the WKS_NEW machine to test if it was getting in the way.
Done. No difference, see previous comment.