In OSB , xquery issue with large volume data

Question

Hi ,
I am facing one problem in xquery transformation in OSB.
There is one xquery transformation where I am comparing all the records and if there are similar records i am clubbing them under same first node.
Here i am reading the input file from the ftp process. This is perfectly working for the small size input data. When there is large input data then also its working , but its taking huge amount of time and the file is moving to error directory and i see the duplicate records created for the same input data. I am not seeing anything in the error log or normal log related to this file.
How to check what is exactly causing the issue here,  why it is moving to error directory and why i am getting duplicate data for large input( approx 1GB).
My Xquery is something like below.
<InputParameters>
                for $choice in $inputParameters1/choice              
                 let $withSamePrimaryID := ($inputParameters1/choice[PRIMARYID eq $choice/PRIMARYID])                
                 let $withSamePrimaryID8 := ($inputParameters1/choice[FIRSTNAME eq $choice/FIRSTNAME])
                 return
                  <choice>
                 if(data($withSamePrimaryID[1]/ClaimID) = data($withSamePrimaryID8[1]/ClaimID)) then
                 let $claimID:= $withSamePrimaryID[1]/ClaimID
                 return
                 <ClaimID>{$claimID}</ClaimID>                
                 else
                 <ClaimID>{ data($choice/ClaimID) }</ClaimID>

Answer 1

HI ,
I understand your use case is
a) read the file ( from ftp location.. txt file hopefully)
b) process the file ( your x query .. although will not get into details)
c) what to do with the file ( send it backend system via Business Service?)
Also noted the files with large size take long time to be processed . This depends on the memory/heap assigned to your JVM.
Can say that is expected behaviour.
the other point of file being moved to error dir etc - this could be the error handler doing the job ( if you one)
if no error handlers - look at the timeout and error condition scenarios on your service.
HTH