One large message or multipels small ones?

Hi:
I'm working in an application using weblogic 8.1.
We receive in a JMS queue a large message containing thouthands of transactions to be processed.
What should be more eficient? processing the whole transactions in the message (with one MDB) or split the large message in smaller ones and process "in paralel" with multiples MDB and somehow join all the results to return as the whole process result (hiding the client from this split/join) ?
Keeping all the execution in the same transaction would be desireable, but we could create multiple independant transactions to process each small message, but the result must be consistent in what transactions were processed successfully and what weren't.
If splitting is the option: Is there any tip on doing this?
thanks in advance.
Guillermo.

Hi Guillermo,
I would recommend reading the book
Enterprise Integration Patterns by Gregor Hohpe
http://www.eaipatterns.com/
This book describes the split and join process very good.
If the messaging overhead is little compared to total job time it would be a good idea to split
(if the tasks also run well in paralel, get quick out of the DB if you run against one DB).
Are you doing transactions against a DB?
Are you using several computers to distribute the work or a multiprocessor computer?
You can also test performance and try to split to 1, 5 or 10 transactions/msg.
When you split, keep track of total number of transations per job (= N),
and when number of successful transactions + number of failed transactions = N you are finished.
You could send the result per msg to a success or failure channel.
Regards,
Magnus Strand

Similar Messages

JMS Sender SAP JMS Provide large message problem

Hi,
we have configured a JMS sender channel to pick up messages from a queue hosted by our SAP JMS provider. Unfortunately a message of about 6 MB size isn't picked up. Smaller messages are picked up.
Has anybody experienced a problem like this before?
Kind regards,
Heiko

Hey,
I guess you are using XML messages:
We had some problems with large XML messages (e.g more the 10MB),
This is usally cause by memory problems.
There are some work-around, like increasing the memory usage of the application server.
The first thing you do is try to understand where the message stacked (in the ABAP/J2EE).
If this is a memory problem, changing the memory configuration can improve this,
but be aware that there are hardware limitation (32bit application server can use with one process only 2GB), therefore messages over 100MB probably will not pass trough XI.
If you have huge files (e.g more than 100MB)
You must develop a program that would split the large message into several small messages. the program can not be written in the XI, and should be written before the adapter. (you might install and use in this case the conversion agent)
If you are using a message with csv format (not XML),
than it is possible to configure the adapter to split every X lines
(no program need to be written)

How do I divide a large catalogue into two smaller one on the same computer?

How can I divide a large catalogue into two smaller ones on the same computer? Can I just create a new catalogue and move files and folders from the old one to the new one? I am using PSE 12 in Windows 7.

A quick update....
I copied the folder in ~Library/Mail/V2/Mailboxes that contains all of my local mailboxes over to the same location in the new account. When I go into mail, the entire file structure is there, however it does not let me view (or search) any of the messages. The messages can be accessed through the Finder, though.
I tried to "Rebuild" the local mailboxes, but it didn't seem to do anything. Any advice would be appreciated.
JEG

In Mail, one mailbox for Recovered Message (AOL) keeps showing 1 very large message that I cannot delete. How can I get rid of this recurring problem, please?

In Mail on iMac, successfully running OS X Lion, one mailbox on My Mac for "Recovered Messages (from AOL)" keeps showing 1 very large message (more than 20 Mb) that I just cannot seem to delete. Each time I go into my In Box, the "loading" symbol spins and the message appears in the "Recovered Messages" mailbox. How can I get rid of this recurrent file, please?
At the same time, I'm not receviving any new mails in my In Box, although, if I look at the same account on my MacBook Pro, I can indeed see the incoming mails (but on that machine I do not have the "recovery" problem).
The help of a clear-thinking Apple fan would be greatly appreciated.
Many thanks.
From Ian in Paris, France

Ian
I worked it out.
Unhide your hidden files ( I used a widget from http://www.apple.com/downloads/dashboard/developer/hiddenfiles.html)
Go to your HD.
Go to Users.
Go to your House (home)
there should be a hidden Library folder there (it will be transparent)
Go to Mail in this folder
The next folder ( for me ) is V2
Click on that and the next one will be a whole list of your mail servers, and one folder called Mailboxes
Click on that and there should be a folder called recovered messages (server) . mbox
Click on that there a random numbered/lettered folder -> data
In that data folder is a list of random numbered folders (i.e a folder called 2, one called 9 etc) and in EACH of these, another numbered folder, and then a folder called messages.
In the messages folder delete all of the ebmx (I think that's what they were from memory, sorry I forgot as I already deleted my trash after my golden moment).
This was GOLDEN for me. Reason being, when I went to delete my "recovered file" in mail, it would give me an error message " cannot delete 2500 files". I knew it was only 1 file so this was weird. Why 2500 files? Because if you click on the ebmx files like I did, hey presto, it turned out that they were ALL THE SAME MESSAGE = 2500 times. In each of those folders in the random numbers, in their related message folder.
Now remember - DONT delete the folder, make sure you have gone to the message folder, found all those pesky ebmx files and deleted THOSE, not the folder.
It worked for me. No restarting or anything. And recovered file. GONE.
Started receiving and syncing mail again. Woohoo.
Best wishes.

Parallel processing for one large message

I have some troubles from messaging performance perspective.
Sender:ABAP Proxy
Receiver:File Adapter
I'd like use parallel processing for one large message.
And the file for receiver is needed to be one file.
Could you let me know how to set them ?
Best regards,
Koji Nagai

Hi
Can you elaborate your requirement more?
How are you trying to achieve parallel processing in XI.
Since you mentioned that the source is Proxy, there should be some trigger mechanism say selection screen, you restrict the values here and use append strategy in File and can execute the same.
REgards
Krish

JDBC Adapter: J2EE server crashes while sending large messages

We want to use the following scenario to transfer data from a MS SQL Server to SAP BW via XI:
JDBC Sender Adapter XI SAP ABAP Proxy.
All works fine with a small amount of data. But if the select statement delivers too many record sets and the size of the transformed XML payload is greater then 50 MB the J2EE server crashes. A complete restart is necessary. It seems to be am memory problem.
Here are the entries from our log files:
dev_server0
[Thr 6151] Mon Jul 24 12:46:57 2006
[Thr 6151] JLaunchIExitJava: exit hook is called (rc=666)
[Thr 6151] **********************************************************************
ERROR => The Java VM terminated with a non-zero exit code.
Please see SAP Note 940893 , section 'J2EE Engine exit codes'
for additional information and trouble shooting.
[Thr 6151] SigISetIgnoreAction : SIG_IGN for signal 17
[Thr 6151] JLaunchCloseProgram: good bye (exitcode=666)
std_server0.out
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
Is this a general problem of the XI or a specific one of our configuration? Is it possible to transfer such large messages via XI? If not, is there a workaround for such scenarios?
(Memory heap size of the J2EE server is 1024 MB.)

> Hi Gil,
>
> i had nearly the same problems some times in praxis
> and the mapping was the reason. Just change your
> interface determination temporary, delete the mapping
> and test again to find out if the mapping is the
> reason.
>
> Regards,
> Udo
I have changed my interface determination so that no message mapping is used. The J2EE server still crashes.
> Hi Gil,
> This does sounds like a memory problem especially
> when it comes to 50M message with a minimum XI sys
> requierments...
> To be sure you can check on the RWB for the
> componnent monitoring at the JDBC adapters and look
> for your adapter
> look at the status of the adapter and the trace
> there...
Hi Nimrod
In case of such an error I have no entries in channel monitor. So I can't see anything there. I have also no entries in message monitor of the RWB in this case. So I don't get any information with standard XI tools.
> My reccomendation to you is to set the poll intervall
> to a shorter period,this way you'll make sure you get
> less records...I hope you have remembered to add a
> status/flag column on the table to be set after
> selection so no duplicate records will be taken on
> the second pools.
>
The problem is that the source of my data is not a simple SQL statement but a stored procedure. So I don't know exactly how many records will be delivered. A update command is not possible.

How to hanlde breaked large messages in sender JMS adapter

Hi,
I have been asked like 'how can we handle breaked large messages in sender JMS adapter?'and lets say I am getting some messages as it is and some are breaked into small segements for the mesaages that are large....
do we need to use module,if so is there any standard module that we can use to hanlde this type of scenarios?
thank you in advance.
babu

http://biemond.blogspot.com/2009/10/jms-request-reply-interaction-pattern.html
if you check the blog of edwin, see his comments at the bottom
there he gives a suggestion how to add the selector properties to filter on
and this one is maybe helpfull
http://blogs.oracle.com/adapters/2010/05/configuring_request-reply_in_jmsadapter.html

Best practices for handling large messages in JCAPS 5.1.3?

Hi all,
We have ran into problems while processing larges messages in JCAPS 5.1.3. Or, they are not that large really. Only 10-20 MB.
Our setup looks like this:
We retrieve flat file messages with from an FTP server. They are put onto a JMS queue and are then converted to and from different XML formats in several steps using a couple of jcds with JMS queues between them.
It seems that we can handle one message at a time but as soon as we get two of these messages simultaneously the logicalhost freezes and crashes in one of the conversion steps without any error message reported in the logicalhost log. We can't relate the crashes to a specific jcd and it seems that the memory consumption increases A LOT for the logicalhost-process while handling the messages. After restart of the server the message that are in the queues are usually converted ok. Sometimes we have however seen that some message seems to disappear. Scary stuff!
I have heard of two possible solutions to handle large messages in JCAPS so far; Splitting them into smaller chunks or streaming them. These solutions are however not an option in our setup.
We have manipulated the JVM memory settings without any improvements and we have discussed the issue with Sun's support but they have not been able to help us yet.
My questions:
* Any ideas how to handle large messages most efficiently?
* Any ideas why the crashes occur without error messages in the logs or nothing?
* Any ideas why messages sometimes disappear?
* Any other suggestions?
Thanks
/Alex

* Any ideas how to handle large messages most efficiently? --
Strictly If you want to send entire file content in JMS message then i don't have answer for this question.
Generally we use following process
After reading the file from FTP location, we just archive in local directory and send a JMS message to queue
which contains file name and file location. Most of places we never send file content in JMS message.
* Any ideas why the crashes occur without error messages in the logs or nothing?
Whenever JMSIQ manager memory size is more lgocialhosts stop processing. I will not say it is down. They
stop processing or processing might take lot of time
* Any ideas why messages sometimes disappear?
Unless persistent is enabled i believe there are high chances of loosing a message when logicalhosts
goes down. This is not the case always but we have faced similar issue when IQ manager was flooded with lot
of messages.
* Any other suggestions
If file size is more then better to stream the file to local directory from FTP location and send only the file
location in JMS message.
Hope it would help.

Response delay in Soap Sender while processing large messages

Hi Experts,
              In one of the scenario our sender adapter is SOAP. After receiving the message its sending back the Response as HTTP 200 OK for success messges it is working fine when the message is small. When ever i am getting a large message that time its taking little bit processing time so my sender system not getting the HTTP 200 OK with in the time. (Suppose with in 3 seconds). In this case source system is sendin again the same message thinking that message is not success.
My Sender system they dont want to increase the waiting time from 3 second to .. Seconds
I cont use AAE becasue we are at 7.0.
I have to use Adapter Engine. I will not using direct communicatin with Integration Server.
     Can you please let me know how can i resolve this issue..
Thanks & Regards,
Puru

>>In this case source system is sendin again the same message thinking that message is not success.My Sender system they dont want to increase the waiting time from 3 second to .. Seconds
Looks like you are doing asynchronous SOAP scenario.   You could adjust the message reprocessing time little bit later. I meant Pi does automatic reprocess messages after sometime if the mesg gets in to error state.
<IRRELEVANT LINK>
Edited by: Prateek Raj Srivastava on Mar 12, 2011 6:24 PM

Large messages cannot be saved out from tracking

Hi,
Using BizTalk 2013 Enterprise, we cannot save out larger messages from tracking. In one example we have an XML-message of 6 MB, saving it renders a zero byte file. Smaller messages can be saved.
Any ideas what this could be? Is it a known problem? Is there a hot fix?
Steps to reproduce:
Tracked Message Events -> Search -> Event Type Receive, Unparsed Interchange -> Open, Save Message (>6 MB)
Thanks!

Found a work-around.. Right click.. Save message.
Glad to hear that you have solved this issue by yourself, and it is very appreciated to share your solution to us.
Best regards
Angie
We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click
HERE to participate the survey.

Large Message picking Wrong Queue.....???

Hi,
     We are processing a large message of size around 50 MB and its going to queue XBTO9. But i want it to be processed through XBTM.
Can someone please reply of how can it be done.
Thanks
Sourav

Its in KB...
Parameter EO_MSG_SIZE_LIMIT
Meaning
   The parameter EO_MSG_SIZE_LIMIT enables you to process messages of a
   particular size in series. This applies for messages with the quality of
   service Exactly Once (EO). If the message is larger than the parameter
   value, the message is processed in an extra queue.
   The prefix for the outbound queue of the Integration Server is XBTM, for
   all others it is XBTL.
   This parameter is designed to improve performance since there is not
   usually enough memory to process multiple large messages simultaneously.
   Serial processing ensures that the main memory only has to process one
   large message at a time, while smaller messages and processes continue
   to be processed uninterrupted.
Possible Values
   0 - 2,097,151 (KB)
   Note: 0 means deactivated (no messages are processed in an extra queue);
   the maximum value 2,097,151 corresponds to the value of roughly 2 GB.
Nilesh

1 large lun or many smaller luns

Hi,
I'm running Oracle 10g/11g. I'm NOT using ASM (that isn't an option right now). My storage is IBM DS3500 with IBM SVC in front of it.
My question is, is it better to have 1 large lun or many smaller luns for the database (assuming its the same number of spindles in both cases)?
Are there any limitations with queue depth..etc. I need to worry about with the 1 large lun?
Any info would be greatly appreciated.
Thanks!

Hi,
You opened this thread on ASM forum and you are not using ASM Filesystem (???????)....what gets dificult to answer your questions.
Well...
First you need to consult the manual/whitepapers/technotes of filesystem that you will use to check what are the recommendations for the database using this filesystem.
eg. using JFS2 on AIX you can enable CIO...
Another point:
Create large luns can be useful and can be not. All depends on the characteristics of your env.
e.g: I believe is not good placing 2 databases with different characteristcs of access/troughput in same filesystem. One database can cause performance issue on others database if theys share same Lun.
I particularly dislike Large Luns to an environment that will store several database .... I usually use Large Luns for large databases, and yet without sharing the area with other databases.
My thoughts {message:id=9676881} although it is valid for ASM.
I recommend you read it:
http://docs.oracle.com/cd/E11882_01/server.112/e16638/iodesign.htm#PFGRF015
Regards,
Levi Pereira

Issue while processing large message using schema exposed as WCFservice

I am using biztalk 2010 and am facing one issue.
I have a schema exposed as WCF service. Used the ecf pubulishing wizard for the same.
When we post a large message using this WCF service it throws http 400 error.
We have isolated the problem to webserver.
Is there a way to update the MEXbindings to set the request message size.
If anyone has faced similar problem before request your help

Hi,
Following settings in the web.config file of the WCF service may help.
<bindings>
    <basicHttpBinding>
        <binding
name="basicHttp" allowCookies="true"
maxReceivedMessageSize="10000000"
maxBufferSize="10000000"
maxBufferPoolSize="10000000">
            <readerQuotas
maxDepth="32"
maxArrayLength="100000000"
maxStringContentLength="100000000"/>
        </binding>
    </basicHttpBinding>
</bindings>
HTH
Sumit
Sumit Verma - MCTS BizTalk 2006/2010 - Please indicate "Mark as Answer" or "Mark as Helpful" if this post has answered the question

ABAP Mapping for Large Messages

Hi Folks,
We are exploring different options for dealing with the fact that XI will choke on very large messages/files. One of the options that we are considering is a third party tool that bypasses XI. However, we've just learned that it may be possible to solve the large message problem by using ABAP Mapping, because supposedly, by doing so, one would bypass the large message being converted to XML as it comes into XI. The scenario involves messages coming into XI on their way to SAP R/3. Does the ABAP Mapping option appear to be viable to you experienced folks out there?
Thanks
Nic

Hey
tunneling as per the term is used mainly for IDOC but we use the term bypass to implement the same concept but with other interfaces.
see if u have simple 1-1 mapping then u can do a bypass scenario in which u are not doing anything in IR,u simple do the configurations in ID.
you can not drastically increase the performance just by choosing some specific mapping.mappin is not made for this .
for best performance you can design a bypass JDBC to IDOC scenario(but then you wont be able to do any message mapping.
have a look at the following for bypass scenario
/people/william.li/blog/2006/09/08/how-to-send-any-data-even-binary-through-xi-without-using-the-integration-repository
if u want to do message mapping then design a JDBC to Proxy scenario.
proxy is mainly used to enhance performance and would be the best bet for u i guess
Thanx
Aamir suhail
Message was edited by:
Aamir Suhail

Large Message Payload - Queue selection

Hi,
I have one message with payload size around 105MB when I am trying to send through XI this message gets failed in the queue (inbound) of integration engine saying some memory problem to process the message. I have seen the queue it is picking up some XBT09_000 but I want to use the queue XBTL* as this queue is for large messages. The whole thing is that when my message is around 100MB I want it route large message processing queue i.e XBTL*. Is there anyway to do this? Any help on this greatly appreciated.
Thanks.,
Daniel.LA

There is a way to defer large messages from the rest and process them using a job in a specific queue.
Please check this configuration :
<a href="http://help.sap.com/saphelp_nw2004s/helpdata/en/14/80243b4a66ae0ce10000000a11402f/frameset.htm">http://help.sap.com/saphelp_nw2004s/helpdata/en/14/80243b4a66ae0ce10000000a11402f/frameset.htm</a><a href="http://help.sap.com/saphelp_nw2004s/helpdata/en/14/8024">Message Selection Filter</a>
in this path:
Under Runtime - > Integration engin ->
Prioritized Message Processing
Queues for Prioritized Message
Good luck
Nimrod

One large message or multipels small ones?

Similar Messages

Maybe you are looking for