RAC node process using 25% physical memory

Question

We have a QA server is non-RAC, and production is two-node RAC. We have a migration app that does an INSERT from SELECT over 2 instances. All of the machines have been in successful day-to-day use for several months...our only troublespot seems to be the migration app.
Today we started the app on the QA server and watched the oracle processes using top. They ran normally and finished without any problems.
The same app started on either of the RAC nodes produced process memory errors and died.
As the app ran, there was a process reading the data from the source instance and a process writing to the target instance. We confirmed this by querying the session data. It doesn't matter which of the nodes runs which target process...the result is the same.
The reading process(session) on the source instance seems to run normally. The write process on the target instance, however, begins slowly accumulating memory in about 16M chunks and holds on to them.
We saw this in the RES and in the MEM columns of top. The target process never released any memory, but slowly grabbed it until its
RES was 4GB and the %MEM was about 30%. The app then died with process memory error. This is reproducible over several runs.
( Per Metalink Note 567506.1, the recommended value for Linux 64-bit is 4294967295 ..we have that set. )
There are other oracle processes and instances running on both nodes which do not seem to be affected. The total number of processes on each machine is around 750..much lower than the nprocs ulimit of 63K.
These process are both oracle sessions spawned by the app.
I haven't seen any info on the web or Metalink that matches these symptoms, so I thought I'd try the experts.
Why would the write session continuously use up physical memory, but only on RAC nodes?
We are running RHEL5 on Dell Poweredge 2950 w 16K Physical mem. Version of 10g is R2.0.4.

Answer 1

user12017889 wrote:
The write process on the target instance, however, begins slowly accumulating memory in about 16M chunks and holds on to them.Exactly what process is this? An Oracle server process? Dedicated or shared server?
We saw this in the RES and in the MEM columns of top. The target process never released any memory, but slowly grabbed it until its
RES was 4GB and the %MEM was about 30%. How does the writer process work? Does it use PL/SQL? Does it use bulk processing? How does it call the reader process? Or does the reader process call it? Is this over a database link.
The app then died with process memory error. This is reproducible over several runs.If this is an Oracle server process, then there should be an entry in the alert log of the instance that recorded the crash and includes the name of the trace file generated by the crash.