Large Attachment Feeding Problem

Question

Hi,
This is related to feeding large size documents to SES. By large size I mean documents having size of the order of 70 MB to 100 MB.
When I submit a document of such a large size to SES after setting the contents of the document using submitForProcessing(DocumentContainer target) I get an OutOfMemory Error from SES for this particular method call.
Is it a limitation of SES or am I not doing everything right?
Need comments.
Regards,
Shakti

Answer 1

Hi Rishi,
Thanks for your quick and precise reply.
The bigger concern here of the two is large number of readers being submitted to SES.
You are quite right here in stating that while crawling n number of docs, similar objects will be created for all the n docs on SES side. We have done our best to optimize the code for memory scale up.
I have two scenarios for my custom crawler:
1. I was able to successfully crawl 100k docs when number of users (readers) for each doc was very few (to be exact it was only 3).
2. When I performed the crawl with my custom crawler for same number of docs but number of users for each doc raised to 1k (1000) I got an Out of Memory error at the point in the code where I submit the users list to SES using addPrincipal( , , ) method defined for the interface DocumentAcl. The interesting observation was, in this scenario I was able to crawl only 48 docs and then Out of Memory Error.
Code Snippet:
//objDocAcl is an instance of DocumentAcl
//acl_attrib is the format of the principal entry and its value here is samAccountName for //MS Active Directory which is configured as id plugin here
for (int count = 0; count < readersList.length; count++) {
          reader = readersList[count];
          try {
               if (reader != null && acl_attrib != null) {
// This is the line of code where I get OOM Error
                    objDocAcl.addPrincipal(reader, acl_attrib,DocumentAcl.USER);
          } catch (ProcessingException exception) {
                    logger.info(iaSpecificProps.getString("Exception")
+exception.getMessage());
          reader = null;
Need your inputs.
Regards,
Shakti