Gateway for Sybase - core dump [npixfc()+243] [SIGSEGV]

Question

Hi All,
We are using a couple of gateways to various Sybase servers in our environment, which generally work very fine.
However, occasionally we find the following lines in the alert.log on one or the other node of the cluster.
The DB platform is currently 11.2.0.1 in RAC on Linux x86-64 RHEL5, but similar problems happened
before on 11.1.0.7 (the entries in the logs were not as descriptive as on 11gR2).
Thu Feb 11 07:48:33 2010
HS: Lost RPC connection to remote Agent...
HS: ... Agent SID = (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=xxx043vip)(PORT=1521))(CONNECT_DATA=(SID=PM04PMDB))), NCR status = -2147385341
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x7FFF20B670B8] [PC:0x59BAADD, npixfc()+243] [flags: 0x0, count: 1]
Errors in file /oracle/app/diag/rdbms/xxx1/xxx11/trace/xxx11_ora_32050.trc (incident=16497):
ORA-07445: exception encountered: core dump [npixfc()+243] [SIGSEGV] [ADDR:0x7FFF20B670B8] [PC:0x59BAADD] [Address not mapped to object] []
ORA-28511: lost RPC connection to heterogeneous remote agent using SID=ORA-28511: lost RPC connection to heterogeneous remote agent using SID=(DESCRIPTION=(ADDRESS=(PR
OTOCOL=TCP)(HOST=xxx043vip)(PORT=1521))(CONNECT_DATA=(SID=PM04PMDB)))
ORA-02063: preceding line from PM04PMDB
Incident details in: /oracle/app/diag/rdbms/xxx1/xxx11/incident/incdir_16497/xxx11_ora_32050_i16497.trc
The .trc file starts with:
Dump file /oracle/app/diag/rdbms/xxx1/xxx11/incident/incdir_16497/xxx11_ora_32050_i16497.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORACLE_HOME = /oracle/app/product/11.2.0/dbhome_1
System name: Linux
Node name: xxx043
Release: 2.6.18-128.el5
Version: #1 SMP Wed Dec 17 11:41:38 EST 2008
Machine: x86_64
Instance name: xxx11
Redo thread mounted by this instance: 1
Oracle process number: 62
Unix process pid: 32050, image: oracle@xxx043
*** 2010-02-11 07:48:33.186
*** SESSION ID:(695.7521) 2010-02-11 07:48:33.186
*** CLIENT ID:() 2010-02-11 07:48:33.186
*** SERVICE NAME:(xxx1) 2010-02-11 07:48:33.186
*** MODULE NAME:(JDBC Thin Client) 2010-02-11 07:48:33.186
*** ACTION NAME:() 2010-02-11 07:48:33.186
Dump continued from file: /oracle/app/diag/rdbms/xxx1/xxx11/trace/xxx11_ora_32050.trc
ORA-07445: exception encountered: core dump [npixfc()+243] [SIGSEGV] [ADDR:0x7FFF20B670B8] [PC:0x59BAADD] [Address not mapped to object] []
ORA-28511: lost RPC connection to heterogeneous remote agent using SID=ORA-28511: lost RPC connection to heterogeneo
========= Dump for incident 16497 (ORA 7445 [npixfc()+243]) ========
----- Beginning of Customized Incident Dump(s) -----
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x7FFF20B670B8] [PC:0x59BAADD, npixfc()+243] [flags: 0x0, count: 1]
Registers:
%rax: 0x000000000a99d750 %rbx: 0x0000000000000001 %rcx: 0x00007fff20b6dee0
%rdx: 0x0000000000000000 %rdi: 0x00007fff20b6d710 %rsi: 0x00002aed89f745a0
%rsp: 0x00007fff20b670c0 %rbp: 0x00007fff20b6dce0 %r8: 0x0000000000000000
%r9: 0x0000000000000000 %r10: 0x0000000000000000 %r11: 0x0000000000000001
%r12: 0x0000000000000010 %r13: 0x00002aed89fa7560 %r14: 0x00002aed89f73c18
%r15: 0x0000000000000001 %rip: 0x00000000059baadd %efl: 0x0000000000010202
(0x59baadd) call 0x9d24a0(0x59baae2) mov %eax,-0x14(%rbp)
(0x59baae5) test %eax,%eax
(0x59baae7) jnz 0x59bfd26
(0x59baaed) mov -0x120(%rbp),%rsi
Is it something to worry about? Any hints about the reasons?
Thanks,
P.

Answer 1

Thanks for your reply. I expected the advice to utilize the support for that :)
The results of my investigation:
Listener log - normal activity. New connections to the DB and to different gateways (including the one that caused the problem) every couple of seconds, everything seemed to be working fine.
Alert log - no other entries +/- 1 hour around the incident, except log switching.
I checked the application logs and managed to find out the session that caused the problem. It is an ETL process that runs the same extraction query on Sybase machine every minute. The query does not change and it was executed successfully hundreds of times before. Other similar processes were running fine, executing similar queries on all Sybase machines, including the one in the incident. So the incident seems to be random, with no correlation to other activities.
The flow of events in the problematic session iis as follows:
1. opens Gateway connection and sets session parameters on Sybase by executing:
dbms_hs_passthrough.execute_immediate@PM04PMDB('set lock wait 60 ') -> that went OK
2. starts data extraction by running:
insert into ... select ... from ...@PM04PMDB -> that operation hang, no further logs from this point
3. commits
4. closes DB link by executing:
alter session close database link PM04PMDB
The SQL that failed was executed at 7:23, and the incident entry in the alert log is dated 7:33.
Is it possible that the 10 seconds delay is caused by HS_FDS_CONNECT_PROPERTIES="timeout='10'" ?
The gateway config:
HS_FDS_CONNECT_INFO=xxx
HS_FDS_TRANSACTION_MODEL=READ_ONLY
HS_FDS_TRACE_LEVEL=OFF
HS_OPEN_CURSORS=10
HS_FDS_CONNECT_PROPERTIES="timeout='10'"
HS_FDS_RECOVERY_ACCOUNT=xxx
HS_FDS_RECOVERY_PWD=xxx
Listener config:
(SID_DESC=
(SID_NAME=PM04PMDB)
(ORACLE_HOME=/oracle/app/product/11.2.0/dbhome_1)
(ENVS=LD_LIBRARY_PATH=/oracle/app/product/11.2.0/dbhome_1/dg4sybs/driver/lib:/oracle/app/product/11.2.0/dbhome_1/lib)
(PROGRAM=dg4sybs)
)