JVM SIGSEGV error running Java 1.5.0_01 on linux

Question

Three times last week we had the JVM come down hard with a SISSEGV error running our web application. Each crash was on a different machine. The offending thread was always in libnet.so shared library handling either a getHostByAddr() or getLocalHostName() call. I included a portion of one of the JVM dumps below. In the other two cases we are in either an Oracle JDBC driver or a SQL Server JDBC driver (tds).
We are not able to issolate and reproduce this problem at will. This is happening in our QA environment and we are planning to go to production soon. Any ideas or information would be most helpful.
We've just decided to upgrade to Java 1.5.0_04 in the hope it might help this problem. Too early to tell if this is making a difference or not.
The JVM is running Tomcat 5.5.9.
OS:Red Hat Enterprise Linux AS release 3 (Taroon Update 2)
vm_info: Java HotSpot(TM) Server VM (1.5.0_01-b08) for linux-x86, built on Dec 6 2004 19:33:50 by java_re with gcc 3.2.1-7a (J2SE release)
Thanks for your attention.
--Brad
first part of jvm dump follows:
# An unexpected error has been detected by HotSpot Virtual Machine:
# SIGSEGV (0xb) at pc=0x007737b9, pid=16179, tid=122411952
# Java VM: Java HotSpot(TM) Server VM (1.5.0_01-b08 mixed mode)
# Problematic frame:
# C [libc.so.6+0x797b9] memchr+0x29
--------------- T H R E A D ---------------
Current thread (0xb192d0f0): JavaThread "http-80-Processor25" daemon [_thread_in_native, id=16341]
siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0xb0bfe0c2
Registers:
EAX=0xb0bfe0c2, EBX=0x0082ea98, ECX=0x00000a0a, EDX=0x0a0a0a0a
ESP=0x074bcae8, EBP=0x074bcb14, ESI=0x00000036, EDI=0x000003e7
EIP=0x007737b9, CR2=0xb0bfe0c2, EFLAGS=0x00010202
Top of Stack: (sp=0x074bcae8)
0x074bcae8: 000003e7 00000036 0075ae2b b0bfe0c2
0x074bcaf8: 0000000a 00000036 b0bff038 074bcc48
0x074bcb08: 0082ea98 b2f87428 074bcc48 074bcb38
0x074bcb18: 0075adb3 b2f87428 074bcc48 000003e7
0x074bcb28: 0000000a 00000001 00000000 0082ea98
0x074bcb38: 074bcb64 00763ebb b2f87428 074bcc48
0x074bcb48: 000003e7 0000000a 00000001 00000000
0x074bcb58: 0011c3a4 074bcc48 000003e8 074bcbb0
Instructions: (pc=0x007737b9)
0x007737a9: 89 d1 c1 e2 10 66 89 ca a8 03 0f 84 cd 00 00 00
0x007737b9: 38 10 0f 84 67 01 00 00 40 4e 0f 84 3f 01 00 00
Stack: [0x0747d000,0x074be000), sp=0x074bcae8, free space=254k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.6+0x797b9] memchr+0x29
C [libc.so.6+0x60db3] IOgetline+0x43
C [libc.so.6+0x69ebb] fgets_unlocked+0x5b
C [libnss_files.so.2+0x4b14] nssfiles_gethostbyaddr_r+0x184
C [libc.so.6+0xef7ed] gethostbyaddr_r+0xdd
C [libnet.so+0x5321] Java_java_net_Inet4AddressImpl_getHostByAddr+0x10d
j java.net.Inet4AddressImpl.getHostByAddr([B)Ljava/lang/String;+0
j java.net.InetAddress$1.getHostByAddr([B)Ljava/lang/String;+4
j java.net.InetAddress.getHostFromNameService(Ljava/net/InetAddress;Z)Ljava/lang/String;+7
j java.net.InetAddress.getHostName(Z)Ljava/lang/String;+10
j java.net.InetAddress.getHostName()Ljava/lang/String;+2
j org.apache.coyote.http11.Http11Processor.parseHost(Lorg/apache/tomcat/util/buf/MessageBytes;)V+38
j org.apache.coyote.http11.Http11Processor.prepareRequest()V+794

Answer 1

So, we started getting this problem again in production (on 1.5.0_06, even).
Following up on Dan Paum's report, I dug around a little deeper:
~/java/old/java/net$ javap Inet4AddressImpl
Compiled from "Inet4AddressImpl.java"
class java.net.Inet4AddressImpl extends java.lang.Object implements java.net.InetAddressImpl{
java.net.Inet4AddressImpl();
public native java.lang.String getLocalHostName() throws java.net.UnknownHostException;
public native byte[][] lookupAllHostAddr(java.lang.String) throws java.net.UnknownHostException;
public native java.lang.String getHostByAddr(byte[]) throws java.net.UnknownHostException;
public synchronized java.net.InetAddress anyLocalAddress();
public synchronized java.net.InetAddress loopbackAddress();
public boolean isReachable(java.net.InetAddress, int, java.net.NetworkInterface, int) throws java.io.IOException;
Both anyLocalAddress() and loopbackAddress() are synchronized, but not getLocalHostName(). The native code for getLocalHostName() calls gethostbyaddr_r in lieu of synchronization. gethostbyaddr_r is a non-standard, non-portable call that, judging from a brief trawl through Google, has historically been problematic for multi-platform code.
I'm guessing this bug is another such problem.
I extracted Inet4AddressImpl.java from src.zip in the JDK, changed the declaration, compiled, and "updated" the modified version into lib/rt.jar:
~/java/new/java/net$ javap Inet4AddressImpl
Compiled from "Inet4AddressImpl.java"
class java.net.Inet4AddressImpl extends java.lang.Object implements java.net.InetAddressImpl{
java.net.Inet4AddressImpl();
public synchronized native java.lang.String getLocalHostName() throws java.net.UnknownHostException;
public native byte[][] lookupAllHostAddr(java.lang.String) throws java.net.UnknownHostException;
public native java.lang.String getHostByAddr(byte[]) throws java.net.UnknownHostException;
public synchronized java.net.InetAddress anyLocalAddress();
public synchronized java.net.InetAddress loopbackAddress();
public boolean isReachable(java.net.InetAddress, int, java.net.NetworkInterface, int) throws java.io.IOException;
So far, so good. I will stress test this over the next few days, and if it works, I'll open a bug in the bug database to get this fixed.