Concurrent (async, queue) Message processing - design issue

Hi,
The question pertains to the following scenario:
1. A single input gateway (queue) for messages.
2. Messages arriving from different systems. High incidence (>30 within a
min) during specific periods
3. Message should be processed in near real-time as and when it arrives.
4. A non-polling (async) client.
I'd thought of two diff approaches for it:
1> Have a lightweight single-threaded message consumer -
init connection etc and register a listener (both msg and excep).
start connection . wait indefinitely
Listener onMessage creates new thread for working and returns.
Worker thread - parses xml in message and updates database
2>Have a single main-thread with sub-threads
Each sub-thread has own session and consumer and listener instance
processing of messages is done in sub-thread itself.
Which approach would be better suited ? Is there any other approach (
(which of the above goes easy on resources, which is faster)
Thanks in advance.
Anshu.

It depends on how concurrent you want to be.
If you want to process the messages in order in a single thread; use 1 connection, session, consumer, thread.
However one of the main reasons for using queues is they provide load balancing across processes or threads. If you want some parallelisation, use 1 connection but many sessions & consumers (1 consumer per session) which will each have its own thread.
For ultimate performance & pooling of connections, sessions, threads as well as the MessageListener POJOs together with exception handling & transaction management use something like Message Driven POJOs...
http://jencks.org/Message+Driven+POJOs
Though if you are talking about 30-60 messages a minute, unless it takes you 1 second to process 1 message, those are very low throughput rates, so probably 1 single thread could do it :)
James
http://logicblaze.com/

Similar Messages

  • Message processing in the queue for long time

    Hello All,
    We are executing a file to Idoc scenario. The mapping is very complex.
    The file to be processed is of size 125kb size. Which generates around 450 Idocs.
    The file is picked by the adapter engine in hardly a minute.
    But I see the message taking 6 hours to process in the Queue.
    (SMQ2)
    Is there a way I can improve and speed up the Queue processing.

    Hi,
    >message taking 6 hours to process in the Queue.
    It should not happen,
    Check whether all the Queue are Registered.
    /people/sap.india5/blog/2006/01/03/xi-asynchronous-message-processing-understanding-xi-queues-part-i
    https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/59e837d3-0201-0010-c096-dc1869733413
    Regards
    Agasthuri Doss

  • Inbound queue stopped processing messages

    Inbound queues not processing the messages in XI server. it throwing the syserror.
    please advise.

    Hi Asha,
    Check your RFC destination which is connecting to your external system . ask your basis team to Provide necessary roles for the user ....
    > 837595 and assign the required authentication to the user being used in the RFC ADAPTER
    Also Please go through the notes#1487160 and #1393039.
    Regards,
    Naveen

  • Stateful message processing for async scenario

    Dear all,
    my async scenarios:
    1. RFC -> PI -> File
    2. File  -> PI -> RFC
    Following facts:
    - The second message is a response to the first one. Both message are separted and async.
    - Scenario 2 can be send some hours after Scenario1.
    - A response should only be send if  a specified field in the message of first scenario is filled. 
      For example: Message in scenario has a field <ResponseNeeded>YES</ResponseNeeded>
      This field does only exist in message of scenario 1.
    - Both message have an ID field (e.g. <ID>4711</ID>) which can be used to identified the
      corresponding message.
    Questions:
    - Is it possible to use stateful message processing for this scenario? Or do I need BMP + Correlation?
    - A service interface with pattern "stateful" can only be used synchronously. So is it
      possible to use this stateful for async message processing at least?
    I would assume that I will need a BPM that sends the messsage and keep alive until a
    response will be sends with same ID. After this I would check message of scenario1 and
    send (or even not) the message of scenario2 to receiver depending on <ResponseNeeded>.
    Thanks in advanced.
    Chris

    Hi Chris,
    Although BPM is possible here, I would not recommend you to use one here for the reason,
    >>Scenario 2 can be send some hours after Scenario1.
    This means you are keeping a process alive for so long which is not a good thing to do.
    You could end up with a lot of open/live process instances.
    Having said that, if you still wish to go ahead with BPM, the steps would be
    1. Receive async (RFC req) - start process, correlation based on ID.
    2. Switch step (Evaluate ResponseNeeded)
    branch 1(True): a.  Send asynch step - File
                               b. Receive async - File correlation - ID.
                               c. Send async - RFC call.
    branch 2(false) a. Send async -File
    3. Stop process.
    Regards
    Jai

  • SAP Blog XI Asynchronous Message Processing: Understanding XI Queues

    Hi,
    I was not able to access this blog anymore.
    It was at this link before : XI Asynchronous Message Processing: Understanding XI Queues -Part I
    Can someone please send me a copy if you've saved before? Thanks.
    Please email to yuanyuanlin2001  at yahoo.com

    hi,
    the link is working and you can still access that - I've just tried
    Regards,
    Michal Krawczyk

  • Set Message Processing Queue based on input payload

    Hi all..
    Can i set the message processing queue from graphical mapping?
    Receiver system in my scenario is ECC.Whuile sending messages to ECC, i need to send the message related to a particular ORDER Number in a particular queue.
    for EX: if ORDER Number  from my input payload is 457 i want to set the queue as ABC457.
    Please let me know how to get it done from mapping or if there is any other approach.
    Thanks
    Manohar

    Hi Ram,
    What do you mean by Queue here?? we can check condition in mapping level only based on ORDER Number we can send message to receiver.use enhance receiver determination.
    Regards,
    Raj

  • Design Issue with using MDB

    Lack of timer services with EJB 2.0 dictated our design to implement MDB, since we really needed to use a timeout mechanism for the calling client. Client puts requests into the queue and after processing, the response object messages are put into the response queue owned by the client i.e client configured queue on the JMS server. This model works for the case where the client is a stateless session bean. Now the issue is that we having another client - web application this time, with a servlet pumping requests to the request queue . How do I retain the same model but send queued messages back to the web application? since a response queue cannot be owned by the app. or can it? is there a work around for this scenario..anyone faced a similar situation?

    May be I was not clear enough in my first explanation, Let me try again to explain my scenario:
    My Expectation from Multi Provider is :
    IObjectA
    1AAA
    (From InfoObject)
    Union
    IObjectA     IObjectB     IObjectC
    1AAA     2BBB     3CCC
    (From Cube)
    The record in the multiprovider should be :
    IObjectA     IObjectB     IObjectC
    1AAA     2BBB     3CCC
    Because, this is what the Union says .. and the Definition of the multiprovider also says the same thing :
    http://help.sap.com/saphelp_bw30b/helpdata/EN/ad/6b023b6069d22ee10000000a11402f/frameset.htm
    Do you still think this is how the behaviour of the multiprovider.. if that is the case what would be the purpose of having an infoobject in the multiprovider.
    Thank you very much in advance for your responses.
    Best Regards.,
    Praveen.

  • Synchronous Send step showing errors in process designer

    Hi All,
    I have few questions in using a synchronous send step in
    BPM.
    Can I use a synchronous send step without opening Sync/async bridge.
    When is the sync/async bridge opened?
    Scenario:
    File async --> XI (receive async) --> sync send step to J2ee application (HTTP post) --> async response send to File
    Sync Abstract Interface:
    Input:
      Message type (same as outbound)
    Response:
    Message type (same as Http response)
    When I use the synchronous step, I see errors in the process designer saying
    Message to be sent and synchronous interface are not of the same type.  This happens for both request and response message.
    Please help me.
    Regards,
    Meher

    I think the Problem is with the INTERFACE Determination.
    You need a BPM for the above scenario, but not a Sync/Asynch Brindge
    In BPM
    Step 1 : Receive from File 1
    Step 2 : Do Transformation from File1 format to J2EE Request Format
    Step 3 :  Send a Synchronous send Step to J2EE
    Step 4 :  Transform the HTTP response to your File2 Format Response
    Step 5 :   Send the Response to File2
    Interface Determination
    File1 to j2ee Request
    J2ee Request/Response To  J2EE Request/Response
    J2ee Response to file 2
    Please check your Interface determination very closely.

  • Error receiving AS2 message from partner: B2B-50037:  B2B inbound message processing error

    B2B/SOA 11.1.1.6.0
    We are setting a new trading partner and when we started document transmissions we are getting errors on the inbound messages: B2B-50037:  B2B inbound message processing error.
    The attachment shows the relevant lines from the soa log and diagnostic log files.  Here is the error detail that shows:
    [URI: /b2b/httpreceiver] Error -:  B2B-50037:  B2B inbound message processing error[[
    Error -:  B2B-50037:  B2B inbound message processing error
            at oracle.tip.b2b.engine.Engine.processIncomingMessageImpl(Engine.java:3143)
            at oracle.tip.b2b.engine.Engine.processIncomingMessage(Engine.java:1650)
            at oracle.tip.b2b.transport.InterfaceListener.onMessageLocal(InterfaceListener.java:403)
            at oracle.tip.b2b.transport.InterfaceListener.onMessage(InterfaceListener.java:214)
            at oracle.tip.b2b.transport.basic.TransportServlet.doPost(TransportServlet.java:754)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
            at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
            at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
            at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
            at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
            at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
            at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
            at java.security.AccessController.doPrivileged(Native Method)
            at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
            at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:442)
            at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
            at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:171)
            at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
            at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
            at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:139)
            at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
            at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730)
            at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
            at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
            at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
            at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273)
            at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
            at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490)
            at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
            at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
    From the b2b_wire_message we get these Protocol_Transport_BINDINGS:
    ChannelName=TransportServlet
    Reverse-Via=LIN-ISA1
    AS2-To=accobra.....
    Date=Fri, 26 Sep 2014 05:46:17 +0000
    AS2-Version=1.2
    AS2-From=K.......
    Disposition-Notification-Options=signed-receipt-protocol=optional, pkcs7-signature; signed-receipt-micalg=optional, sha1
    Disposition-Notification-To=http://<ip&port>/as2in
    Message-ID=<[email protected]>
    MSG_RECEIVED_TIME=Fri Sep 26 00:46:17 CDT 2014
    ECID-Context=1.0050z5j^buc6yGn6wnZf6G0002f60007bt;kXjE1ZDLIPGIj2QCYV8QoKSSmLRO_PQT_IS
    Content-Type=application/pkcs7-mime; smime-type=enveloped-data; name=smime.p7m
    Proxy-Client-IP=172.17.25.101
    MIME-Version=1.0
    User-Agent=e-integration AS2 Server V 6.1.8
    X-Forwarded-For=172.17.25.101
    Content-Length=3602
    Host=nalinsoa05.abd.ad.acco.com
    x-weblogic-cluster-hash=QoZzGUzdcjBD5fGIE8Uos5abiHI
    EDIINT-Features=multiple-attachments, CEM
    Connection=Keep-Alive
    X-WebLogic-KeepAliveSecs=30
    X-WebLogic-Request-ClusterInfo=true
    The message creates a wire message, business message, and an application message.
    What doesn't happen is no MDN gets sent back to the partner.  It should be a synchronous MDN.
    We have double checked the certificates on both ends and they are OK.
    The document and Agreement get recognized OK:
    Refer To Message
    Refer To Message
    Sender Type
    AS2 Identifier
    Sender Value
    K. . .
    Receiver Type
    AS2 Identifier
    Receiver Value
    accobr. . .
    Sender
    K. . . l
    Receiver
    ACCO . . .
    Agreement Id
    K. . .l_EDI_X12_4010_856_856Def_Inbound
    Agreement
    K. . .l_EDI_X12_4010_856_856Def_Inbound
    Document Type
    856
    Document Protocol
    EDI_X12
    Document Version
    4010
    Message Type
    REQ
    Direction
    INBOUND
    State
    MSG_ERROR
    Acknowledgement Mode
    SYNC
    Response Mode
    ASYNC
    Send Time Stamp
    09/26/2014 12:46:17 AM
    Receive Time Stamp
    09/26/2014 12:46:17 AM
    The error is vague enough to provide little or no help in locating the root cause of the issue.
    Any assistance in providing information on how to get this working would be greatly appreciated.
    We do have dozens of other AS2 partners working in this instance just foe reference.  We are just having issues with this new partner setup.
    Thank you.
    Regards,
    Ken

    Ken,
    I am observing the Ack Mode is set as "SYNC" . This is selected by default. This option is available in the channel configuration section
    If the usecase is not SYNC, please change as ASYNC and test. It should work.

  • Messages are in to be delivered state and slows down the message processing

    Hello,
    Messages are in to be delivered state and slows down the message processing.
    this happenens in case of DB2 on some machine where the issue about high load on DB2  is caused by standard programming in SAP XI that executes a statement, where it uses the condition 'where 1<0' prio to the original  SQL statement configured in the channel configuration,
    Can you please help me out in solving this issue?
    Thanks,
    Soorya

    Status: TO_BE_DELIVERED
    Which means that the message was successfully delivered from Integration Server point of view and it states that the messages is initially handed over to the Messaging System.
    TO_BE_DELIVERED occurs while the message is put into the Messaging System receive queue.
    Solution:
    This is done via the Messaging System receive servlet:
    http://<Host>:<PORT>/MessagingSystem/receive/<CONNECTION>
    /<PROTOCOL>
    Only if this was executed successfully the Messaging System returns HTTP 200 to the Integration Server and the Status TO_BE_DELIVERED moves to DELIVERING
    1. Try logging into Sap GUI with two users: XIAPPLUSER & XIAFUSER
    to see if they are blocked
    2. We can check the messages:
    AdapterFramework
    com.sap.aii.adapterframework.serviceuser.language
    com.sap.aii.adapterframework.serviceuser.name = XIAFUSER
    com.sap.aii.adapterframework.serviceuser.pwd
    ApplicationSystem
    com.sap.aii.applicationsystem.serviceuser.language
    com.sap.aii.applicationsystem.serviceuser.name = XIAPPLUSER
    com.sap.aii.applicationsystem.serviceuser.pwd
    in the exchange profile to make sure the right passwords

  • Message Processing through Advanced Adapter Engine

    Hi All
    I am trying to design a high volume asynchronous interface through AAE .  The scenario is File to ABAP server proxy .
    I have few questions about AAE processing with Sender File and Receiver Soap adapter.
    1) Suppose if I have 50,000 transaction records in one file Will all go to one queue for processing or many queue processing for processing . if it is  many, is it configurable?
    2) Should the single file having 50,000 transaction be split to many messages  and processed in parallel .
    I am trying to find out how AAE helps to achieve performance .
    Thanks in Advance.

    hi,
    >>>1) Suppose if I have 50,000 transaction records in one file Will all go to one queue for processing or many queue processing for processing . if it is many, is it configurable?
    if you're pushing via AAE (ICO object) only then you can have 50k messages in one file easily
    (shound't be more then a few minutes to process in PI) if they are the standard size
    Regards,
    Michal Krawczyk

  • B2B-50037: B2B inbound message processing error

    We are trying to implement B2B with Customer EDI X12 850 Inbound with SOA
    Application server Version     11.1.1.4.0 (weblogic)
    Hitting the following error in the logs
    MDS-00013: no metadata found for metadata object "/soa/b2b/EDI_X12/4010/850/New/8504010Viterra.ecs"
    Was successful in overcoming all the usual obstacles of finding the TP, Document Type, protocol etc...However hitting the above error. 8504010Viterra.ecs is a custom spec builder document we created and "8504010Viterra.ecs" was browsed from the windows PC into the Document Type B2B set up. I am willing to use a seeded ecs file but do not know how to find one for X12 850 4010 version....?
    Went through one SR that asks you upload a group ECS file but that did not help in our case....Following was the SR
    "B2B-50037 B2B INBOUND MESSAGE PROCESSING" Error with EDIX12 And DEF 830 VER 200\0 [ID 1357136.1]
    Please help
    Now at the agreement level, if we set the translate to No and Validate to No then B2B completes normally but we do want B2B to translate the file so that we can get it in XML form for us to transform and process it in BPEL
    Any help or hint is really appreciated!

    Anuj
    THANK YOU SO MUCH!!! I cannot thank you enough for this
    I was playing around so much with the document type, the protocol version etc. that i must have inadvertently messed up the repository. I took the export of the data as you suggested, purged the design repository and reimported the zip. I also had to restart the server since the ftp was not kicking in...& wholla! the message got translated into XML and went into the queue
    Thanks again
    Gopal Iyer
    Edited by: ghiyer on Nov 10, 2011 10:07 AM

  • Analysis Process Designer and Inventory Management

    In the How to guide for Inventory Management it mentions that you can you can intialise stock in the Snap shot Cube using the Analysis Process Designer (APD).  Has anyone done this and if so can you explain how or outline some steps? Thanks

    Hallo,
    the APD does`nt be a useful tool for modeling a performant 
    data flow. A lot of SAP BW user think so.
    The performance problems are given by the ddic intern table, using to uploading the extracted data into the apd used wa_table and structures.
    To have the best performance on your scenario, please use the following scenario as possible.
    Step 1.
    upload the extracted data from the psa into a data layer
    there you can reduce and harmonize the data by using a transactional ods
    Step 2.
    build a infoset that joins the data
    Step 3.
    build one query to reducing data and make two copies of ist
    Step 4.
    build a useful data mining model
    Step 5.
    upload the results of the data mining model into a transactional ods
    Step 6.
    link the uploaded data into a infoset ore write back into a standard ods
    Step 7.
    query the data
    If you use this scenario you have a lot of benefits. Better performance, better quality of persistent data and actual and traininged data.
    The recommend next step (if you want a alerting) is to build a reporting agent report - if you have usefull processes in  the query.
    There are a workshop for Data mining and APD, named BW380.
    I hope I helped you.
    Otherwise give me a message.
    [email protected]

  • Getting Err  : An error occurred in the message processing program.

    Hi, Im wrking with Smartforms.
    I have assigned output type LD00 ,with driver prgrm name and Smf name.
    after that executing with VL03N , its giving error .
    Output could not be issued
    Message no. VL086
    Diagnosis
    An error occurred in the message processing program.
    Procedure
    Contact your system administrator.
    ...Actually i have orginal prgrm (zsmf_Orginal) there i had to do some modifications , so after copying that same form into another form(zsmf_Duplicate), i made some changes there, both form are executing in T code SMARTFORMS, but while testing with VL03N , its showing above mentioned err.
    Plz have eye on it.
    <removed by moderator>
    Regards
    vivek.
    Edited by: vivekd358 on Jul 10, 2010 7:35 AM
    Edited by: Thomas Zloch on Jul 12, 2010 11:35 AM

    closed

  • Message processing failed, FTP Receiver Adapter error...

    Hello all,
    We have a Idoc to File(FTP) scenario using PI.
    When PI try to send the file out to the FTP site, we get the following message in the communication channel monitoring and the file never reach the FTP site:
    Message processing failed. Cause: com.sap.aii.af.ra.ms.api.RecoverableException: Error when getting an FTP connection from connection pool: com.sap.aii.af.service.util.concurrent.ResourcePoolException: Unable to create new pooled resource: ConnectException: Connection timed out: connect
    When we look at the detail display, we can see that the connection as been establish with the FTP site but the adapter is unable to deliver the file...
    Any Idea why?
    Thanks in advance.

    Hi ,
    There are two things that you can do
    1- check the connection of FTP from command prompt. If it is acceebile from command prompt then check for authorization that wether you have access to post the file at FTP or not (Full access READ , WRITE and EXCECUTE)
    2- in your adpater change the connection mode from Per file transfer to "Permanent".
    Please feel free to reply on this thread if you are not able to.
    Thanks

Maybe you are looking for