Problem enqueueing large message

I am enqueueing/dequeueing xml messages using the Java api. They are being sent as RAW data. When I try to enqueue a large message (219kb), I get:
"oracle.AQ.AQOralceSQLException: Data size bigger than max size for this type: 219546"
Does anyone know why, and how I can avoid this?
ps My C++ colleagues dont get this problem - v embarrassing!

RAW datatypes only can store up to 32 KB. You need to use a CLOB. Oracle shows how you can do this within books on line.

Similar Messages

Problem processing large message using dbadapter.

I have a process which is initiated by dbadapter fetch from table.
Its working fine when the records are less. But when the number of records
are more than 6000(more than 4MB) I am getting errors as below.
The process goes to off state after these errors.
Any body have any suggestions on how to process large messages ?
<2006-08-02 11:55:25,172> <ERROR> <default.collaxa.cube> <BaseCubeSessionBean::logError> Error while invoking bean "cube delivery": Delivery callback message serialization failed.
An attempt to serialize the delivery callback messages for conversation "LocalGUID:75f32d7727f922f9:1712b3a:10ccf9e4cf4:-7f6d", message "75f32d7727f922f9:1712b3a:10ccf9e4cf4:-7f6c" to binary format has failed. The exception reported is:
<2006-08-02 11:55:36,473> <ERROR> <default.collaxa.cube> <BaseCubeSessionBean::logError> Error while invoking bean "delivery": Delivery callback message serialization failed.
An attempt to serialize the delivery callback messages for conversation "LocalGUID:75f32d7727f922f9:1712b3a:10ccf9e4cf4:-7f6d", message "75f32d7727f922f9:1712b3a:10ccf9e4cf4:-7f6c" to binary format has failed. The exception reported is:
<2006-08-02 11:55:42,689> <ERROR> <default.collaxa.cube.activation> <AdapterFramework::Inbound> [OracleDB_ptt::receive(HccIauHdrCollection)] - JCA Activation Agent was unable to perform delivery of inbound message to BPEL Process 'bpel://localhost/default/IAUProcess~1.0/' due to: Delivery callback message serialization failed.
An attempt to serialize the delivery callback messages for conversation "LocalGUID:75f32d7727f922f9:1712b3a:10ccf9e4cf4:-7f6d", message "75f32d7727f922f9:1712b3a:10ccf9e4cf4:-7f6c" to binary format has failed. The exception reported is:
<2006-08-02 11:56:22,573> <ERROR> <default.collaxa.cube.activation> <AdapterFramework::Inbound>
com.oracle.bpel.client.ServerException: Delivery callback message serialization failed.
An attempt to serialize the delivery callback messages for conversation "LocalGUID:75f32d7727f922f9:1712b3a:10ccf9e4cf4:-7f6d", message "75f32d7727f922f9:1712b3a:10ccf9e4cf4:-7f6c" to binary format has failed. The exception reported is:
     at com.collaxa.cube.engine.delivery.DeliveryHandler.initialPostAnyType(DeliveryHandler.java:327)
     at com.collaxa.cube.engine.delivery.DeliveryHandler.initialPost(DeliveryHandler.java:218)
     at com.collaxa.cube.engine.delivery.DeliveryHandler.post(DeliveryHandler.java:82)
     at com.collaxa.cube.ejb.impl.DeliveryBean.post(DeliveryBean.java:181)
     at IDeliveryBean_StatelessSessionBeanWrapper22.post(IDeliveryBean_StatelessSessionBeanWrapper22.java:1052)
     at com.oracle.bpel.client.delivery.DeliveryService.post(DeliveryService.java:161)
     at oracle.tip.adapter.fw.AdapterFrameworkListenerBase$DeliveryServiceMonitor.send(AdapterFrameworkListenerBase.java:2358)
     at oracle.tip.adapter.fw.AdapterFrameworkListenerBase.executeDeliveryServiceSend(AdapterFrameworkListenerBase.java:487)
     at oracle.tip.adapter.fw.AdapterFrameworkListenerBase.deliveryServiceSend(AdapterFrameworkListenerBase.java:545)
     at oracle.tip.adapter.fw.jca.AdapterFrameworkListenerImpl.performSingleActivation(AdapterFrameworkListenerImpl.java:746)
     at oracle.tip.adapter.fw.jca.AdapterFrameworkListenerImpl.onMessage(AdapterFrameworkListenerImpl.java:614)
     at oracle.tip.adapter.fw.jca.messageinflow.MessageEndpointImpl.onMessage(MessageEndpointImpl.java:121)
     at oracle.tip.adapter.db.InboundWork.onMessageImpl(InboundWork.java:370)
     at oracle.tip.adapter.db.InboundWork.onMessage(InboundWork.java:332)
     at oracle.tip.adapter.db.InboundWork.transactionalUnit(InboundWork.java:301)
     at oracle.tip.adapter.db.InboundWork.runOnce(InboundWork.java:255)
     at oracle.tip.adapter.db.InboundWork.run(InboundWork.java:189)
     at oracle.tip.adapter.fw.jca.work.WorkerJob.go(WorkerJob.java:51)
     at oracle.tip.adapter.fw.common.ThreadPool.run(ThreadPool.java:267)
     at java.lang.Thread.run(Thread.java:534)
<2006-08-02 11:57:52,341> <ERROR> <default.collaxa.cube.ws> <Database Adapter::Outbound> <oracle.tip.adapter.db.InboundWork runOnce> Non retriable exception during polling of the database ORABPEL-11624 DBActivationSpec Polling Exception.
Query name: [OracleDB], Descriptor name: [IAUProcess.HccIauHdr]. Polling the database for events failed on this iteration.
If the cause is something like a database being down successful polling will resume once conditions change. Caused by javax.resource.ResourceException: ORABPEL-12509 Unable to post inbound message to BPEL business process.
The JCA Activation Agent of the Adapter Framework was unsuccessful in delivering an inbound message from the endpoint [OracleDB_ptt::receive(HccIauHdrCollection)] - due to the following reason: com.oracle.bpel.client.ServerException: Delivery callback message serialization failed.
An attempt to serialize the delivery callback messages for conversation "LocalGUID:75f32d7727f922f9:1712b3a:10ccf9e4cf4:-7f6d", message "75f32d7727f922f9:1712b3a:10ccf9e4cf4:-7f6c" to binary format has failed. The exception reported is:
Please examine the log file for any reasons. Make sure the inbound XML messages sent by the Resource Adapter comply to the XML schema definition of the corresponding inbound WSDL message element.
     at oracle.tip.adapter.fw.jca.AdapterFrameworkListenerImpl.onMessage(AdapterFrameworkListenerImpl.java:684)
     at oracle.tip.adapter.fw.jca.messageinflow.MessageEndpointImpl.onMessage(MessageEndpointImpl.java:121)
     at oracle.tip.adapter.db.InboundWork.onMessageImpl(InboundWork.java:370)
     at oracle.tip.adapter.db.InboundWork.onMessage(InboundWork.java:332)
     at oracle.tip.adapter.db.InboundWork.transactionalUnit(InboundWork.java:301)
     at oracle.tip.adapter.db.InboundWork.runOnce(InboundWork.java:255)
     at oracle.tip.adapter.db.InboundWork.run(InboundWork.java:189)
     at oracle.tip.adapter.fw.jca.work.WorkerJob.go(WorkerJob.java:51)
     at oracle.tip.adapter.fw.common.ThreadPool.run(ThreadPool.java:267)
     at java.lang.Thread.run(Thread.java:534)
Caused by: ORABPEL-12509
Unable to post inbound message to BPEL business process.
The JCA Activation Agent of the Adapter Framework was unsuccessful in delivering an inbound message from the endpoint [OracleDB_ptt::receive(HccIauHdrCollection)] - due to the following reason: com.oracle.bpel.client.ServerException: Delivery callback message serialization failed.
An attempt to serialize the delivery callback messages for conversation "LocalGUID:75f32d7727f922f9:1712b3a:10ccf9e4cf4:-7f6d", message "75f32d7727f922f9:1712b3a:10ccf9e4cf4:-7f6c" to binary format has failed. The exception reported is:
Please examine the log file for any reasons. Make sure the inbound XML messages sent by the Resource Adapter comply to the XML schema definition of the corresponding inbound WSDL message element.
     at oracle.tip.adapter.fw.jca.AdapterFrameworkListenerImpl.onMessage(AdapterFrameworkListenerImpl.java:628)
     ... 9 more
Caused by: com.oracle.bpel.client.ServerException: Delivery callback message serialization failed.
An attempt to serialize the delivery callback messages for conversation "LocalGUID:75f32d7727f922f9:1712b3a:10ccf9e4cf4:-7f6d", message "75f32d7727f922f9:1712b3a:10ccf9e4cf4:-7f6c" to binary format has failed. The exception reported is:
     at com.collaxa.cube.engine.delivery.DeliveryHandler.initialPostAnyType(DeliveryHandler.java:327)
     at com.collaxa.cube.engine.delivery.DeliveryHandler.initialPost(DeliveryHandler.java:218)
     at com.collaxa.cube.engine.delivery.DeliveryHandler.post(DeliveryHandler.java:82)
     at com.collaxa.cube.ejb.impl.DeliveryBean.post(DeliveryBean.java:181)
     at IDeliveryBean_StatelessSessionBeanWrapper22.post(IDeliveryBean_StatelessSessionBeanWrapper22.java:1052)
     at com.oracle.bpel.client.delivery.DeliveryService.post(DeliveryService.java:161)
     at oracle.tip.adapter.fw.AdapterFrameworkListenerBase$DeliveryServiceMonitor.send(AdapterFrameworkListenerBase.java:2358)
     at oracle.tip.adapter.fw.AdapterFrameworkListenerBase.executeDeliveryServiceSend(AdapterFrameworkListenerBase.java:487)
     at oracle.tip.adapter.fw.AdapterFrameworkListenerBase.deliveryServiceSend(AdapterFrameworkListenerBase.java:545)
     at oracle.tip.adapter.fw.jca.AdapterFrameworkListenerImpl.performSingleActivation(AdapterFrameworkListenerImpl.java:746)
     at oracle.tip.adapter.fw.jca.AdapterFrameworkListenerImpl.onMessage(AdapterFrameworkListenerImpl.java:614)
     ... 9 more
.

To process 6000 messages in one shot is not the best practice in BPEL. For that you have to choose concepts like datawarehouse or so.
But you might want to process it in batch mode. So think of using batch option in DB adapter and try to define MaxRaiseSize and MaxTransactionSize for your DB adapter. Further explanation is here
http://download-west.oracle.com/docs/cd/B14099_19/integrate.1012/b25307/adptr_db.htm#CHDHAIHA

In Mail, one mailbox for Recovered Message (AOL) keeps showing 1 very large message that I cannot delete. How can I get rid of this recurring problem, please?

In Mail on iMac, successfully running OS X Lion, one mailbox on My Mac for "Recovered Messages (from AOL)" keeps showing 1 very large message (more than 20 Mb) that I just cannot seem to delete. Each time I go into my In Box, the "loading" symbol spins and the message appears in the "Recovered Messages" mailbox. How can I get rid of this recurrent file, please?
At the same time, I'm not receviving any new mails in my In Box, although, if I look at the same account on my MacBook Pro, I can indeed see the incoming mails (but on that machine I do not have the "recovery" problem).
The help of a clear-thinking Apple fan would be greatly appreciated.
Many thanks.
From Ian in Paris, France

Ian
I worked it out.
Unhide your hidden files ( I used a widget from http://www.apple.com/downloads/dashboard/developer/hiddenfiles.html)
Go to your HD.
Go to Users.
Go to your House (home)
there should be a hidden Library folder there (it will be transparent)
Go to Mail in this folder
The next folder ( for me ) is V2
Click on that and the next one will be a whole list of your mail servers, and one folder called Mailboxes
Click on that and there should be a folder called recovered messages (server) . mbox
Click on that there a random numbered/lettered folder -> data
In that data folder is a list of random numbered folders (i.e a folder called 2, one called 9 etc) and in EACH of these, another numbered folder, and then a folder called messages.
In the messages folder delete all of the ebmx (I think that's what they were from memory, sorry I forgot as I already deleted my trash after my golden moment).
This was GOLDEN for me. Reason being, when I went to delete my "recovered file" in mail, it would give me an error message " cannot delete 2500 files". I knew it was only 1 file so this was weird. Why 2500 files? Because if you click on the ebmx files like I did, hey presto, it turned out that they were ALL THE SAME MESSAGE = 2500 times. In each of those folders in the random numbers, in their related message folder.
Now remember - DONT delete the folder, make sure you have gone to the message folder, found all those pesky ebmx files and deleted THOSE, not the folder.
It worked for me. No restarting or anything. And recovered file. GONE.
Started receiving and syncing mail again. Woohoo.
Best wishes.

JMS Sender SAP JMS Provide large message problem

Hi,
we have configured a JMS sender channel to pick up messages from a queue hosted by our SAP JMS provider. Unfortunately a message of about 6 MB size isn't picked up. Smaller messages are picked up.
Has anybody experienced a problem like this before?
Kind regards,
Heiko

Hey,
I guess you are using XML messages:
We had some problems with large XML messages (e.g more the 10MB),
This is usally cause by memory problems.
There are some work-around, like increasing the memory usage of the application server.
The first thing you do is try to understand where the message stacked (in the ABAP/J2EE).
If this is a memory problem, changing the memory configuration can improve this,
but be aware that there are hardware limitation (32bit application server can use with one process only 2GB), therefore messages over 100MB probably will not pass trough XI.
If you have huge files (e.g more than 100MB)
You must develop a program that would split the large message into several small messages. the program can not be written in the XI, and should be written before the adapter. (you might install and use in this case the conversion agent)
If you are using a message with csv format (not XML),
than it is possible to configure the adapter to split every X lines
(no program need to be written)

SOA 10g JMS adapter does not pick up large messages in Oracle AQ

I am using SOA 10g 10.1.3.4 MLR #9. When I enqueue a large JMS message ( of type SYS.AQ$_JMS_TEXT_MESSAGE ) into the AQ from SOA 11g it goes into the TEXT_LOB portion of the USER_DATA column in AQ. In addition to this GROUPID and GROUPSEQ do not seem to get set. When that happens, my JMS adapterfrom SOA 10g does not seem to dequeue the message. But if I enqueue the same message using sql and set the GROUPID and GROUPSEQ my JMS adapter from 10g picks up the clob message.
How do we make the 10g adapter always pick up the large message in TEXT_LOB? I don't have this problem when the message is small because it goes into TEXT_VC field instead of TEXT_LOB field.

We are having this problem and have not figured it out yet. We need help. An oracle SR is opened but no progress from Oracle yet. I need to get some traction on this so anyone's help is appreciated. Our suspicion is that the AQ client in the wls 10.3.1 is incorrectly choosing to put the xml payload in TEXT_LOB even though the character size is less than 4000.
Here is a summary of our observations....
Today we did some testing with our old 6.3.1 OSM environment and our new 7.0.2 environment. We tested each against our QA1 environment and then our QA2 environment. Scenrio 1 is our problem. Scenario 2,3,and 4 all behave similar to each other. Why does Scenario 1 behave differently. It is causing AIA a problem when it dequeues the message.
Scenario 1:
We pointed our 7.0.2 OSM/Weblogic 10.3.1/Out of the Box AQ (which is part of our QA1 environment) to send messages to the AIA AQ table in our QA1 environment. The AIA database uses ALT32UTF8 character set. We don't know what character set the 7.0.2 OSM/Weblogic/AQ uses. Wehn we sent messages to the AIA QA1 table we saw that everything under TEXT_LEN 1000 went to TEXT_VC, Over 1000 went to TEXT_LOB. This is causes AIA to throw an exception if the message is over 1000 when it dequeus the message, because it thinks it shoudl look in the TEXT_VC which is null for AIA.
Scenario 2:
We pointed our 7.0.2 OSM/Weblogic 10.3.1/Out of the box AQ (which is part of our QA1 environment) to send messages to the AIA AQ table in our QA2 enviornment. The AIA database uses UTF8 character set. We don't know what character set the 7.0.2 OSM/Weblogic/AQ uses. When we sent messages to the AIA QA2 AQ table we saw that everything under TEXT_LEN 4000 went to TEXT_VC. Over 4000 went to TEXT_LOB. This is fine for us.
Scenario 3:
We pointed our 6.3.1 OSM/Weblogic 9.2.1/Custom AQ Solution (which is part of our QA2 environment) to send messages to the AIA AQ table in our QA2 environment. The AIA database uses UTF8 character set. We don't know what character set the 6.3.1 OSM/Weblogic/AQ uses. When we sent messages to the AIA QA2 AQ table we saw that everything under TEXT_LEN 4000 went to TEXT_VC. Over 4000 went to TEXT_LOB. This is fine for us.
Scenario 4:
We pointed our 6.3.1 OSM/Weblogic 9.2.1/Custom AQ Solution (which is part of our DEVINT1 environment) to send messages to the AIA AQ table in our QA1 enviornment. The AIA database uses ALT32UTF8 character set. We don't know what character set the 6.3.1 OSM/Weblogic/AQ uses. When we sent messages to the AIA QA1 AQ table we saw that everything under TEXT_LEN 4000 went to TEXT_VC. Over 4000 went to TEXT_LOB. This is fine for us.

JDBC Adapter: J2EE server crashes while sending large messages

We want to use the following scenario to transfer data from a MS SQL Server to SAP BW via XI:
JDBC Sender Adapter XI SAP ABAP Proxy.
All works fine with a small amount of data. But if the select statement delivers too many record sets and the size of the transformed XML payload is greater then 50 MB the J2EE server crashes. A complete restart is necessary. It seems to be am memory problem.
Here are the entries from our log files:
dev_server0
[Thr 6151] Mon Jul 24 12:46:57 2006
[Thr 6151] JLaunchIExitJava: exit hook is called (rc=666)
[Thr 6151] **********************************************************************
ERROR => The Java VM terminated with a non-zero exit code.
Please see SAP Note 940893 , section 'J2EE Engine exit codes'
for additional information and trouble shooting.
[Thr 6151] SigISetIgnoreAction : SIG_IGN for signal 17
[Thr 6151] JLaunchCloseProgram: good bye (exitcode=666)
std_server0.out
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
Is this a general problem of the XI or a specific one of our configuration? Is it possible to transfer such large messages via XI? If not, is there a workaround for such scenarios?
(Memory heap size of the J2EE server is 1024 MB.)

> Hi Gil,
>
> i had nearly the same problems some times in praxis
> and the mapping was the reason. Just change your
> interface determination temporary, delete the mapping
> and test again to find out if the mapping is the
> reason.
>
> Regards,
> Udo
I have changed my interface determination so that no message mapping is used. The J2EE server still crashes.
> Hi Gil,
> This does sounds like a memory problem especially
> when it comes to 50M message with a minimum XI sys
> requierments...
> To be sure you can check on the RWB for the
> componnent monitoring at the JDBC adapters and look
> for your adapter
> look at the status of the adapter and the trace
> there...
Hi Nimrod
In case of such an error I have no entries in channel monitor. So I can't see anything there. I have also no entries in message monitor of the RWB in this case. So I don't get any information with standard XI tools.
> My reccomendation to you is to set the poll intervall
> to a shorter period,this way you'll make sure you get
> less records...I hope you have remembered to add a
> status/flag column on the table to be set after
> selection so no duplicate records will be taken on
> the second pools.
>
The problem is that the source of my data is not a simple SQL statement but a stored procedure. So I don't know exactly how many records will be delivered. A update command is not possible.

Issue while processing large message using schema exposed as WCFservice

I am using biztalk 2010 and am facing one issue.
I have a schema exposed as WCF service. Used the ecf pubulishing wizard for the same.
When we post a large message using this WCF service it throws http 400 error.
We have isolated the problem to webserver.
Is there a way to update the MEXbindings to set the request message size.
If anyone has faced similar problem before request your help

Hi,
Following settings in the web.config file of the WCF service may help.
<bindings>
    <basicHttpBinding>
        <binding
name="basicHttp" allowCookies="true"
maxReceivedMessageSize="10000000"
maxBufferSize="10000000"
maxBufferPoolSize="10000000">
            <readerQuotas
maxDepth="32"
maxArrayLength="100000000"
maxStringContentLength="100000000"/>
        </binding>
    </basicHttpBinding>
</bindings>
HTH
Sumit
Sumit Verma - MCTS BizTalk 2006/2010 - Please indicate "Mark as Answer" or "Mark as Helpful" if this post has answered the question

Change transaction using BDC-problem for large number of lines on screen

Hi All,
I am developing BAPI (using BDC) which creates quality notification in SAP which is entered via front end web application. Structure p_qmsm contain 3 lines of task in notification. The code is as given below. To avoid problem of large no of lines on screen,code lines starting with * is used. This actually for pagedown after entering every 2 lines and creates new line. so that 2 lines always push up on screen and there will not be problem for creating large no of lines on screen
perform bdc_dynpro using 'SAPLIQS0' '7200'.
perform bdc_field using 'BDC_OKCODE' '=10\TAB11'.
LOOP AT p_qmsm INTO wa_qmsm.
*IF wa_qmsm_cntr > 2.
       wa_qmsm_cntr = 2.
       perform bdc_dynpro using 'SAPLIQS0'                   '7204'.
       perform bdc_field   using 'BDC_OKCODE'                 '=PEND'.
ENDIF.
perform bdc_dynpro using 'SAPLIQS0' '7204'.
perform bdc_field using 'BDC_OKCODE' '/00'.
CONCATENATE 'VIQMSM-QSMNUM(' wa_qmsm_cntr ')' INTO wm_qmsm_qsmnum.
CONCATENATE 'VIQMSM-MNGRP(' wa_qmsm_cntr ')' INTO wm_qmsm_mngrp.
CONCATENATE 'VIQMSM-MNCOD(' wa_qmsm_cntr ')' INTO wm_qmsm_mncod.
CONCATENATE 'VIQMSM-MATXT(' wa_qmsm_cntr ')' INTO wm_qmsm_matxt.
perform bdc_field using wm_qmsm_qsmnum wa_qmsm-qsmnum.
perform bdc_field using wm_qmsm_mngrp wa_qmsm-mngrp.
perform bdc_field using wm_qmsm_mncod wa_qmsm-mncod.
perform bdc_field using wm_qmsm_matxt wa_qmsm-matxt.
wa_qmsm_cntr = wa_qmsm_cntr + 01.
ENDLOOP.
CALL TRANSACTION 'IQS2' USING wt_bdc
MODE 'N' UPDATE 'A' MESSAGES INTO P_MESSTAB.
The same code is used in modify mode also. web application is sending all 3 lines in modify mode even if single line is modified. It is already decided to send all rows back from web application to SAP in same sequence. It is working fine if i comment 5 lines which is starting with *. But in modify mode, how can i ensure that correct row is modified? and how can i achieve problem of large no of lines on screen?Please suggest?

Hi yogesh,
how can i ensure that correct row is modified?
1. For this we need to know two things ;
   a) the database table in which the entries are already stored
   b) the sequence in which they are displayed in the transaction.
2. So before changing any line, we need to compare (the primary key values / important values)
   a) as per the database table and as per the incoming data from web application (using bapi)
b) if the match is ok, it means that particular row was not modified, else modified.
how can i achieve problem of large no of lines on screen?
1. For this I am not sure about the transaction and its screen. Manytimes for appending row on the screen,
there is a PLUS + button on the grid toolbar. So for every entry, (inspite of some empty/filled rows already visible on the screen), we should use the + button, and this new row always appears on the top i.e. row number 1.
hope this helps.
regards,
amit m.

ABAP Mapping for Large Messages

Hi Folks,
We are exploring different options for dealing with the fact that XI will choke on very large messages/files. One of the options that we are considering is a third party tool that bypasses XI. However, we've just learned that it may be possible to solve the large message problem by using ABAP Mapping, because supposedly, by doing so, one would bypass the large message being converted to XML as it comes into XI. The scenario involves messages coming into XI on their way to SAP R/3. Does the ABAP Mapping option appear to be viable to you experienced folks out there?
Thanks
Nic

Hey
tunneling as per the term is used mainly for IDOC but we use the term bypass to implement the same concept but with other interfaces.
see if u have simple 1-1 mapping then u can do a bypass scenario in which u are not doing anything in IR,u simple do the configurations in ID.
you can not drastically increase the performance just by choosing some specific mapping.mappin is not made for this .
for best performance you can design a bypass JDBC to IDOC scenario(but then you wont be able to do any message mapping.
have a look at the following for bypass scenario
/people/william.li/blog/2006/09/08/how-to-send-any-data-even-binary-through-xi-without-using-the-integration-repository
if u want to do message mapping then design a JDBC to Proxy scenario.
proxy is mainly used to enhance performance and would be the best bet for u i guess
Thanx
Aamir suhail
Message was edited by:
Aamir Suhail

Best practices for handling large messages in JCAPS 5.1.3?

Hi all,
We have ran into problems while processing larges messages in JCAPS 5.1.3. Or, they are not that large really. Only 10-20 MB.
Our setup looks like this:
We retrieve flat file messages with from an FTP server. They are put onto a JMS queue and are then converted to and from different XML formats in several steps using a couple of jcds with JMS queues between them.
It seems that we can handle one message at a time but as soon as we get two of these messages simultaneously the logicalhost freezes and crashes in one of the conversion steps without any error message reported in the logicalhost log. We can't relate the crashes to a specific jcd and it seems that the memory consumption increases A LOT for the logicalhost-process while handling the messages. After restart of the server the message that are in the queues are usually converted ok. Sometimes we have however seen that some message seems to disappear. Scary stuff!
I have heard of two possible solutions to handle large messages in JCAPS so far; Splitting them into smaller chunks or streaming them. These solutions are however not an option in our setup.
We have manipulated the JVM memory settings without any improvements and we have discussed the issue with Sun's support but they have not been able to help us yet.
My questions:
* Any ideas how to handle large messages most efficiently?
* Any ideas why the crashes occur without error messages in the logs or nothing?
* Any ideas why messages sometimes disappear?
* Any other suggestions?
Thanks
/Alex

* Any ideas how to handle large messages most efficiently? --
Strictly If you want to send entire file content in JMS message then i don't have answer for this question.
Generally we use following process
After reading the file from FTP location, we just archive in local directory and send a JMS message to queue
which contains file name and file location. Most of places we never send file content in JMS message.
* Any ideas why the crashes occur without error messages in the logs or nothing?
Whenever JMSIQ manager memory size is more lgocialhosts stop processing. I will not say it is down. They
stop processing or processing might take lot of time
* Any ideas why messages sometimes disappear?
Unless persistent is enabled i believe there are high chances of loosing a message when logicalhosts
goes down. This is not the case always but we have faced similar issue when IQ manager was flooded with lot
of messages.
* Any other suggestions
If file size is more then better to stream the file to local directory from FTP location and send only the file
location in JMS message.
Hope it would help.

Handling large messages with MQ JMS sender adapter

Hi.
Im having trouble handling large messages with a MQ JMS sender adapter.
The messages are around 35-40MB.
Are there any settings I can ajust to make the communication channel work?
Error message is:
A channel error occurred. The detailed error (if any) : JMS error:MQJMS2002: failed to get message from MQ queue, Linked error:MQJE001: Completion Code 2, Reason 2010, Error Code:MQJMS2002
The communication channel works fine with small messages!
Im on SAP PI 7.11, MQ Driver is version 6.
Best Regards...
Peter

The problem solved itself, when the MQ server crashed and restarted.
I did find a note that might could have been useful:
Note 1258335 - Tuning the JMS service for large messages or many consumers
A relevant post as well: http://forums.sdn.sap.com/thread.jspa?threadID=1550399

Large Message Payload - Queue selection

Hi,
I have one message with payload size around 105MB when I am trying to send through XI this message gets failed in the queue (inbound) of integration engine saying some memory problem to process the message. I have seen the queue it is picking up some XBT09_000 but I want to use the queue XBTL* as this queue is for large messages. The whole thing is that when my message is around 100MB I want it route large message processing queue i.e XBTL*. Is there anyway to do this? Any help on this greatly appreciated.
Thanks.,
Daniel.LA

There is a way to defer large messages from the rest and process them using a job in a specific queue.
Please check this configuration :
<a href="http://help.sap.com/saphelp_nw2004s/helpdata/en/14/80243b4a66ae0ce10000000a11402f/frameset.htm">http://help.sap.com/saphelp_nw2004s/helpdata/en/14/80243b4a66ae0ce10000000a11402f/frameset.htm</a><a href="http://help.sap.com/saphelp_nw2004s/helpdata/en/14/8024">Message Selection Filter</a>
in this path:
Under Runtime - > Integration engin ->
Prioritized Message Processing
Queues for Prioritized Message
Good luck
Nimrod

Problem enqueuing with SYS.AQ$_JMS_TEXT_MESSAGE payload

We are trying to enqueue a message into a SYS.AQ$_JMS_TEXT_MESSAGE AQ from java using the standard AQjmsQueueSender. The message arrives onto the queue but the USER_DATA is always 'oracle.sql.STRUCT@5a41ec' instead of the actual message. When we enqueue from PL/SQL the USER_DATA contains the actual message. Any help would be greatly appreciated. Thanks.
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bi
PL/SQL Release 10.2.0.4.0 - Production
"CORE     10.2.0.4.0     Production"
TNS for Linux: Version 10.2.0.4.0 - Production
NLSRTL Version 10.2.0.4.0 - Production
QUEUE DETAILS
OWNER:     EIMGR
QUEUE_TABLE:     AQ_VENDORS_IN_TABLE
TYPE: OBJECT
OBJECT_TYPE:     SYS.AQ$_JMS_TEXT_MESSAGE
SORT_ORDER:     ENQUEUE_TIME
RECIPIENTS:     SINGLE
MESSAGE_GROUPING:     NONE
COMPATIBLE: 8.1.3
PRIMARY_INSTANCE: 0
SECONDARY_INSTANCE:     0
OWNER_INSTANCE:     1
USER_COMMENT: EI_QUEUE
SECURE:     NO
CREATED:     09-09-03
LAST_DDL_TIME: 09-09-03
-------------------------------------------------------------------------------------

We've discovered with PL/SQL that setting any headers on messages causes the same problem that we're seeing from JAVA. There is no option in JAVA to send messages without headers (it's part of the JMS spec). This can be seen with the following PL/SQL script (use null instead of HEADER in the message constructor to see it work):
DECLARE
Enqueue_options DBMS_AQ.enqueue_options_t;
Message_properties DBMS_AQ.message_properties_t;
Message_handle RAW(16);
User_prop_array SYS.AQ$_JMS_USERPROPARRAY;
Agent SYS.AQ$_AGENT;
Header SYS.AQ$_JMS_HEADER;
Message SYS.AQ$_JMS_TEXT_MESSAGE;
Message_text VARCHAR2(500);
BEGIN
Agent := SYS.AQ$_AGENT('',NULL,0);
User_prop_array := SYS.AQ$_JMS_USERPROPARRAY();
Header := SYS.AQ$_JMS_HEADER( Agent, '', 'dave', '', '', '', User_prop_array);
Message_text := 'Message from PL/SQL created at '|| TO_CHAR(SYSDATE, 'mm/dd/yyyy hh24:mi:ss');
Message := SYS.AQ$_JMS_TEXT_MESSAGE(Header, LENGTH(Message_text), Message_text, NULL);
DBMS_AQ.ENQUEUE(queue_name => 'AQ_VENDORS_IN',
Enqueue_options => enqueue_options,
Message_properties => message_properties,
Payload => message,
Msgid => message_handle);
COMMIT;
END;

Oracle 11g AQ : problem enqueue user-defined type with varchar2 attribute

Hello.
I have a problem enqueuing a user-defined type to the queue on Oracle 10g.
I'm using jdbc driver (ojdbc5.jar, version 11.1.0.6.0) as they provide oracle.jdbc.aq package.
The type is following:
CREATE OR REPLACE TYPE FIXED_T5 AS OBJECT
(id integer,
label varchar2(100),
code integer,
today date
)I have created a java class for this type with jpub utility supplied with oracle 11g client package:
jpub -user=scott/tger -url=jdbc:oracle:thin:@host:sid-sql=FIXED_T5:ru.invito.FixedType -compile=falseIt generated FixedType.java and FixedTypeRef.java files. Don't understand why i need the latter (FixedTypeRef).
Then in test app:
package ru.invito;
import java.io.IOException;
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.sql.SQLException;
import java.sql.Timestamp;
import java.util.Date;
import java.util.Properties;
import java.util.UUID;
import junit.framework.TestCase;
import oracle.jdbc.aq.AQAgent;
import oracle.jdbc.aq.AQEnqueueOptions;
import oracle.jdbc.aq.AQFactory;
import oracle.jdbc.aq.AQMessage;
import oracle.jdbc.aq.AQMessageProperties;
import oracle.jdbc.aq.AQEnqueueOptions.DeliveryMode;
import oracle.jdbc.aq.AQEnqueueOptions.VisibilityOption;
import oracle.jdbc.driver.OracleConnection;
import oracle.jdbc.driver.OracleDriver;
import oracle.sql.Datum;
import oracle.sql.STRUCT;
import oracle.sql.StructDescriptor;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
public class AqTest extends TestCase {
     protected Log logger = LogFactory.getLog(getClass());
     public void testEnqueue() throws Exception {
          OracleDriver dr = new OracleDriver();
          Properties prop = new Properties();
          prop.setProperty("user", Config.USERNAME);
          prop.setProperty("password", Config.PASSWORD);
          OracleConnection connection = (OracleConnection) dr.connect(Config.JDBC_URL, prop);
          assertNotNull(connection);
          connection.setAutoCommit(false);
          enqueueMessage(connection, "INVITO.FIXED_T5Q", null);
          connection.commit();
     private void enqueueMessage(OracleConnection conn, String queueName, AQAgent[] recipients) throws SQLException,
               IOException {
          logger.debug("----------- Enqueue start ------------");
          AQMessageProperties props = makeProps(queueName);
          AQMessage mesg = AQFactory.createAQMessage(props);
          String msqText = String.format("Hello, %s!", queueName);
          FixedType data = createData(36, msqText);
          Datum d = data.toDatum(conn);
          STRUCT s = (STRUCT) d;
          debugStruct("s", s);
          String toIdStr = byteBufferToHexString(s.getDescriptor().getOracleTypeADT().getTOID(), 20);
          logger.debug("s.toIdStr: " + toIdStr);
          StructDescriptor sd = StructDescriptor.createDescriptor("INVITO.FIXED_T5", conn);
          logger.debug("sd.toXMLString(): " + sd.toXMLString());
          mesg.setPayload(s);
          AQEnqueueOptions opt = makeEnqueueOptions();
          logger.debug("sending............");
          // execute the actual enqueue operation:
          conn.enqueue(queueName, opt, mesg);
          debugMessageId(mesg);
          logger.debug("----------- Enqueue done ------------");
     private void debugMessageId(AQMessage mesg) throws SQLException {
          byte[] mesgId = mesg.getMessageId();
          if (mesgId == null) {
               throw new IllegalStateException("message id is NULL");
          String mesgIdStr = byteBufferToHexString(mesgId, 20);
          logger.debug("Message ID from enqueue call: " + mesgIdStr);
      * @return
      * @throws SQLException
     private FixedType createData(int ID, String label) throws SQLException {
          FixedType data = new FixedType();
          data._struct.setNChar(1);// initializes the flag for 'label' field
          data.setId(ID);
          data.setLabel(label);
          data.setCode(1);
          Date today = new Date();
          data.setToday(new Timestamp(today.getTime()));
          return data;
      * @param string
      * @param s
      * @throws SQLException
     private void debugStruct(String string, STRUCT s) throws SQLException {
          logger.debug(s + ".debugString(): " + s.debugString());
          logger.debug(s + "s.dump(): " + s.dump());
      * @return
      * @throws SQLException
     private AQAgent makeAgent() throws SQLException {
          AQAgent ag = AQFactory.createAQAgent();
          ag.setName("AQ_TEST");
          String agentAddress = null;
          try {
               agentAddress = InetAddress.getLocalHost().getHostAddress();
          catch (UnknownHostException e) {
               logger.error("cannot resolve localhost ip address. will not set it as AQ Agent address");
               agentAddress = "NA";
          ag.setAddress(agentAddress);
          return ag;
     private AQMessageProperties makeProps(String queueName) throws SQLException {
          final String EXCEPTION_Q_TEMPLATE = "AQ$_%sT_E";
          final int DEFAULT_DELAY = 0;
          final int DEFAULT_EXPIRATION = -1;
          final int DEFAULT_PRIORITY = 0;
          AQMessageProperties propeties = AQFactory.createAQMessageProperties();
          propeties.setCorrelation(UUID.randomUUID().toString());
          propeties.setDelay(DEFAULT_DELAY);
          propeties.setExceptionQueue(String.format(EXCEPTION_Q_TEMPLATE, queueName));
          propeties.setExpiration(DEFAULT_EXPIRATION);
          propeties.setPriority(DEFAULT_PRIORITY);
          // propeties.setRecipientList(null);//should not set
          propeties.setSender(makeAgent());
          return propeties;
      * @return
      * @throws SQLException
     private AQEnqueueOptions makeEnqueueOptions() throws SQLException {
          AQEnqueueOptions opt = new AQEnqueueOptions();
          opt.setRetrieveMessageId(true);
          // these are the default settings (if none specified)
          opt.setDeliveryMode(DeliveryMode.PERSISTENT);
          opt.setTransformation(null);
          opt.setVisibility(VisibilityOption.ON_COMMIT);
          return opt;
      * Form the AQ reference
      * @param buffer
      * @param maxNbOfBytes
      * @return
     private static final String byteBufferToHexString(byte[] buffer, int maxNbOfBytes) {
          if (buffer == null)
               return null;
          int offset = 0;
          StringBuffer sb = new StringBuffer();
          while (offset < buffer.length && offset < maxNbOfBytes) {
               String hexrep = Integer.toHexString((int) buffer[offset] & 0xFF);
               if (hexrep.length() == 1)
                    hexrep = "0" + hexrep;
               sb.append(hexrep);
               offset++;
          String ret = sb.toString();
          return ret;
}The output is following:
[main] 2008-07-03 19:09:49,863 DEBUG [ru.invito.AqTest] - ----------- Enqueue start ------------
[main] 2008-07-03 19:09:50,348 DEBUG [ru.invito.AqTest] - [email protected](): name = INVITO.FIXED_T5 length = 4 attribute[0] = 36 attribute[1] = Hell
o, INVITO.FIXED_T5Q! attribute[2] = 1 attribute[3] = 2008-07-03 19:09:49.0
[main] 2008-07-03 19:09:50,363 DEBUG [ru.invito.AqTest] - [email protected](): name = INVITO.FIXED_T5
length = 4
ID = 36
LABEL = Hello, INVITO.FIXED_T5Q!
CODE = 1
TODAY = 2008-07-03 19:09:49.0
[main] 2008-07-03 19:09:50,363 DEBUG [ru.invito.AqTest] - s.toIdStr: 507ccce5b6e9f572e040007f01007203
[main] 2008-07-03 19:09:50,363 DEBUG [ru.invito.AqTest] - sd.toXMLString(): <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<StructDescriptor sqlName="INVITO.FIXED_T5" >
<OracleTypeADT sqlName="INVITO.FIXED_T5" typecode="0" tds_version="1"
           is_embedded="false" is_top_level="true" is_upt="false" finalType="true" subtype="false">
    <attributes>
      <attribute name="ID" type="INTEGER" >
        <OracleType typecode="2" />
      </attribute>
      <attribute name="LABEL" type="VARCHAR2" >
        <OracleType typecode="12" />
      </attribute>
      <attribute name="CODE" type="INTEGER" >
        <OracleType typecode="2" />
      </attribute>
      <attribute name="TODAY" type="DATE" >
        <OracleType typecode="0" />
      </attribute>
    </attributes>
</OracleTypeADT>
</StructDescriptor>
[main] 2008-07-03 19:09:50,379 DEBUG [ru.invito.AqTest] - sending............
[main] 2008-07-03 19:09:50,395 DEBUG [ru.invito.AqTest] - Message ID from enqueue call: 511ff143bd4fa536e040007f01003192
[main] 2008-07-03 19:09:50,395 DEBUG [ru.invito.AqTest] - ----------- Enqueue done ------------But when dequeueing the 'label' attribute is lost:
DECLARE
dequeue_options     DBMS_AQ.dequeue_options_t;
message_properties DBMS_AQ.message_properties_t;
message_handle      RAW(16);
message             fixed_t5;
BEGIN
dequeue_options.navigation := DBMS_AQ.FIRST_MESSAGE;
DBMS_AQ.DEQUEUE(
     queue_name          =>     'fixed_t5q',
     dequeue_options     =>     dequeue_options,
     message_properties =>     message_properties,
     payload             =>     message,
     msgid               =>     message_handle);
DBMS_OUTPUT.PUT_LINE('ID   : '||message.id);
DBMS_OUTPUT.PUT_LINE('Label: '||message.label);
DBMS_OUTPUT.PUT_LINE('Code : '||message.code);
DBMS_OUTPUT.PUT_LINE('Today: '||message.today);
COMMIT;
END;
ID   : 36
Label:
Code : 1
Today: 03.07.08
Could anyone tell me what is wrong with the setup/code?
Why 'label' not saved in queue, though i saw it is not empty in STRUCT?

Thank you for the reply!
I have enqueued:
[main] 2008-07-04 15:30:30,639 DEBUG [ru.invito.UserDefinedTypeAqTest$1] - [email protected](): name = INVITO.FIXED_T5
length = 4
ID = 41
LABEL = Hello, INVITO.FIXED_T5Q!
CODE = 1
TODAY = 2008-07-04 15:30:30.0and in table (select * from FIXED_T5QT) the 'label' is blank:
Q_NAME     FIXED_T5Q
MSGID     51310809B5EA3728E040007F01000C79
CORRID     b8f38fd3-4fa6-4e0f-85d1-2440d02d655e
PRIORITY     0
STATE     0
DELAY
EXPIRATION
TIME_MANAGER_INFO
LOCAL_ORDER_NO     0
CHAIN_NO     0
CSCN     0
DSCN     0
ENQ_TIME     04.07.2008 15:28:44
ENQ_UID     INVITO
ENQ_TID                       4012
DEQ_TIME
DEQ_UID
DEQ_TID
RETRY_COUNT     0
EXCEPTION_QSCHEMA     AQ$_INVITO
EXCEPTION_QUEUE     FIXED_T5QT_E
STEP_NO     0
RECIPIENT_KEY     0
DEQUEUE_MSGID
SENDER_NAME     AQ_TEST
SENDER_ADDRESS     10.1.1.137
SENDER_PROTOCOL
USER_DATA.ID     41
USER_DATA.LABEL
USER_DATA.CODE     1
USER_DATA.TODAY     04.07.2008 15:30:30I must point to a strange thing: when the FixedType instance is created (via new operator) and then the setLabel("....") called as:
FixedType data = new FixedType();
// hack: proper initialization for 'label' field
data._struct.setNChar(1);
data.setId(ID);
data.setLabel(label);
data.setCode(1);
Date today = new Date();
data.setToday(new Timestamp(today.getTime()));
Datum d = data.toDatum(connection);
STRUCT s = (STRUCT) d;
logger.debug(s + ".debugString(): " + s.debugString());
logger.debug(s + ".dump(): " + s.dump());and if i comment line (data._struct.setNChar(1);) the debug messages for created STRUCT also shows empty value for label.
But if i explicitly call data._struct.setNChar(1) then debug contains the label i defined in call to the setLabel method.

Large messages cannot be saved out from tracking

Hi,
Using BizTalk 2013 Enterprise, we cannot save out larger messages from tracking. In one example we have an XML-message of 6 MB, saving it renders a zero byte file. Smaller messages can be saved.
Any ideas what this could be? Is it a known problem? Is there a hot fix?
Steps to reproduce:
Tracked Message Events -> Search -> Event Type Receive, Unparsed Interchange -> Open, Save Message (>6 MB)
Thanks!

Found a work-around.. Right click.. Save message.
Glad to hear that you have solved this issue by yourself, and it is very appreciated to share your solution to us.
Best regards
Angie
We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click
HERE to participate the survey.

Problem enqueueing large message

Similar Messages

Maybe you are looking for