Strange RMI behavior and data

Question

Hello:
I have a very difficult and strange problem. I am hoping somebody can spot what is going on or maybe tell me I am way off.
I have a client-server application that uses RMI for network communication. The server is a database and the clients insert, update and delete data from it. When a client starts up, it does Naming.lookup, and upon a success, will register some callback methods so that it gets notified of any changes to the database.
Everything works, but the performance is somewhat sketchy. For example, I have one test where one client sends over 1,000 new items to the database. After about 10 seconds of great performance it suddenly stops, almost like it lost its connection. Then, after about 10-30 seconds later, it starts right back up and runs for another 10 seconds. I am not getting any exceptions thrown.
At first I thought this was a network configuration issue, but I am able to resolve names and IP address backwards and forwards across all the clients and the server.
I installed ethereal, a network sniffer, to try to find out what was going wrong. I noticed a strange pattern, between my clients and server. Every 10 seconds there is strange exchange between each client and the server that last about .1 of a second.
The client always sends the same 41 bytes of data and the server sends the same 23 bytes back, except for the second-to-last byte. That one is always incremented.
The exchange is continuously happening even when none of the clients are doing anything, which leads me to believe that it is just a normal part of RMI and isn�t doing anything. However, I have also noticed that my slowdowns ALWAYS occur when this happens. Below is a snippet of the data exchange between one of my clients and the server.
Has anybody ever seen this, or does anybody here have an idea of what is going wrong?
Client:
00000000 52
Server:
00000000 53
Client:
00000001 50 ac ed 00 05 77 22 00 00 00 00 00 00 00 02 01
00000011 82 0d da 00 00 01 0c ea 55 13 9f 80 00 ff ff ff
00000021 ff c6 90 44 fb 34 76 b1 8d
Server:
00000001 51 ac ed 00 05 77 10 01 01 82 0d da 00 00 01 0c
00000011 ea 55 13 9f 85 17 01
Client:
0000002A 52
Server:
00000018 53
Client:
0000002B 50 ac ed 00 05 77 22 00 00 00 00 00 00 00 03 01
0000003B 82 0d da 00 00 01 0c ea 55 13 9f 80 00 ff ff ff
0000004B ff c6 90 44 fb 34 76 b1 8d
Server:
00000019 51 ac ed 00 05 77 10 01 01 82 0d da 00 00 01 0c
00000029 ea 55 13 9f 85 18 01

Answer 1

Thanks for everybody�s help so far.
I have done some more testing and I found some usefull information in the transport.tcp log on the server side.
First, some background: I am now running with a single client attached to the database. My client is at 192.168.0.63 and my server at 192.168.0.21.
When my program starts up, my client opens up an RMI connection with the database to get the current data and to send database changes one. The client port is 3110 and the server port is 58252.
After I make the first change, the server makes a RMI connection back to my client to sends it datbase update. The server port is 58547 and the client is 3087.
I am sending the same two updates to the database every 5 seconds or so. After about 30 - 60, the database takes 12 seconds to contact the client where it normally take 0.005 seconds.
Just after the previous successfull update I saw this message
06:33 sun.rmi.transport.TCPChannel$Reaper run
FINER: RMI ConnectionExpiration-[192.168.0.63:3087]: wake up
Then at 6:36 I attemped to re-save another database object, I got my message that the server was about to attempt to fire off another update to the clients. Immediatly after that I got this log message
06:36 run.rmi.transport.tcp.TCPChannel new connection
FINE: Thread 3 - reuse connection
6 seconds afterwards I get this
6:44 sun.rmi.transport..tcp.TCPConnectionHandler run
FINE: RMI TCP Connection(3)-192.168.0.63 accepted from 192.168.0.63:3201
6: 44 sun.rmi.transport..tcp.TCPConnectionHandler run
FINE: RMI TCP Connection(3)-192.168.0.63(58252) suggesting 192.168.0.63:3201
6: 44 sun.rmi.transport..tcp.TCPConnectionHandler run
FINE: RMI TCP Connection(3)-192.168.0.63(58252) using 192.168.0.63:3087
The update to the client finishes at 6:48
In the packet sniffer I saw the same 12 second delay between the ACK and the PSH,ACK packets. In that delay, I saw that an RMI connection was initiated between the client on port 3201 and the server on port 585252. I am assuming that that connection is what was showing up at 6:44.
I guess that the connection between the database and the listener on the client is expiring for some reason. What I don't understand is why and who terminated the connection, the client or the server?
I was able to remove the program that was hiding my transport.tcp log messages, but I didn't see any of the connection re-establishment on the client side. Maybe I just didn't look hard enough