Behaviour of tpforward when ipc queue is full

Hi,
Can anyone explain the action taken by a server when the ipc queue of the service/server
it is trying to tpforward to is full? What appears to happen is the typical blocking
behaviour of a tpacall without the TPNOBLOCK flag set. Is there anyway of controlling
this behaviour? What we would like to achieve is the equivalent of a TPNOBLOCK
flag being set, with the server giving a failure message back to the caller immediately
(rather than waiting for the blocking timeout) so that it is freed up to continue
processing. Any and all suggestions welcome!
Thanks & regards,
Malcolm.

Hi Todd,
Thank you for following this thread! I've attached to TMUSREVT with strace to monitor msg* calls and figured out that TMUSREVT communicates with BBL periodically. Then I have reproduced TMUSREVT server hang in my lab environment. I've set a small value of kernel.msgmnb(kernel.msgmnb = 10512), ran a test client with many subscriptions and simulated high load on the server. When a hang occured TMUSREVT's queue looked this:
------ Message Queues --------
key msqid owner perms used-bytes messages
0x00000000 532545581 pbtrunk 666 10352 12
(virtually full)
And then in the ULOG I found the following records:
184431.host!BBL.14147.3086141120.0: LIBTUX_CAT:1285: WARN: tpreturn message send blocked, will try file transfer
184431.host!TMSYSEVT.14189.3086939840.0: WARN: .SysMachineMsgq: simple1 message queue blocking
184501.host!BBL.14147.3086141120.0: LIBTUX_CAT:1286: ERROR: tpreturn could not send reply TPETIME - timeout occured
Does this make sense for you?

Similar Messages

  • WiSM error message : Message: CPU Receive Multicast Queue is full on Controller

    Hi,
    I heve several WiSM modules, the version is 7.0.220.0. Each of WiSM receive the following error message : CPU Receive Multicast Queue is full on Controller. I searched the bug toolkit, some old version of WLC hit this bug, but not affect our version.. Please help to mention me how to debug this error.
    Best Regards,

    Multicast Queue Depth
    The multicast queue depth is 512 packets on all controller platforms. However, the following message might appear on 2106 controllers: “Rx Multicast Queue is full on Controller.” This message does not appear on 4400 series controllers because the 4400 NPU filters ARP packets while all forwarding (multicast or otherwise) and multicast replication are done in the software on the 2106.
    This message appears when too many multicast messages are sent to the CPU. In controller software releases prior to 5.1, multicast, CDP, and ARP packets share the same queue. However, in software releases 5.1 and later, these packets are separated into different queues. There are currently no controller commands that can be entered to determine if the multicast receive queue is full. When the queue is full, some packets are randomly discarded.
    Do you have a lot of clients using multicast ?
    "Satisfaction does not come from knowing the solution, it comes from knowing why." - Rosalind Franklin

  • Tuxedo process IPC queue monitoring

    Hi
    This question is aimed mostly at Oracle techies who have experience supporting Tuxedo sites.
    We've been developing on Tuxedo for a couple of years now and are about to go live with a Tuxedo based transactional system.
    Throughout the development and testing phase it has become pretty apparent to us that we need to be able to keep an eye on IPC request queues for processes running under Tuxedo.
    For example, we have 10 instances of server "A" running within Tuxedo. As long as these ten instances can cope with the load, we are getting good end-to-end response times. When we overload the system whereby our requests start being queued up on IPC queues for these processes, quite predictably our end-to-end response times go up. We would like to get an early warning when situations like this occur.
    Sometimes there are obvious reasons for this behaviour (i.e. we're sending more requests per seconds than our application can cope with) and at other times, it is a degradation in performance of a downstream process (e.g. database query taking longer and longer to return as time goes by) that is causing our requests to start backlogging. It is this latter scenario that we would like to be alerted to before our clients ring us up to tell us that our service is not performing as well as it should.
    However, there seems to be a distinct lack of importance placed on server request queue monitoring within Tuxedo infrastructure and API set. Perhaps we are missing something obvious?
    We would like to be able to have a more proactive monitoring and reporting facility in place to help us observe queuing behaviour. It seems that there are two ways to obtain any queuing diagnostics:
    1) Use a shell script to call tmadmin/pq from time to time to take snapshots or use OS commands to monitor IPC queues
    2) Use Tuxedo MIB and poll the queue OIDs to get queue depth snapshots.
    We would like to be able to get a lot more than that. For example:
    * to be able to plot a graph of queue depths over a period of time
    * get alerted when queue depths cross a certain limit (trap vs poll)
    * work out how long an item spent on the queue before it got processed
    So, are we placing too much emphasis on something that is not important to other Tuxedo users or is there a suite of tools one can use to obtain this sort of information that we are not aware of?
    How important is request queuing to a typical Tuxedo site.
    Regards,
    Mike

    Hi Mike,
    You might want to look at the Oracle Tuxedo System and Application Monitor (TSAM). It is an add-on product from Oracle that provides system and application monitoring for Tuxedo applications. Although TSAM doesn't currently give you the kind of graph you are looking for, it does collect that information and place it in the TSAM monitoring database. TSAM provides the schema of the monitoring database, so creating whatever graphs you want should be pretty easy.
    Specifically what I would look at though is TSAM's alert generating mechanism. Alert information can either be checked at the monitoring console, or it can be checked at the monitored nodes plug-in. The later offers the option of simply checking for an alert without having to report the data back to the console unless an alert is triggered. This tends to have much less impact on the monitored system and help keep the monitoring console from being flooded with data. In the Call Path and Service monitoring categories, you can define alerts that check if the number of queued messages exceed a certain amount. This is quite efficient and would have minimal impact on system performance, yet warn you when queues are starting to back up.
    These two methods would give you historical data to look for trends as well as real time alerting to be notified that backups are occurring.
    Regards,
    Todd Little
    Oracle Tuxedo Chief Architect

  • Help in resolving the error - Incident Queue is full

    I've been given a 4.2.4 system to look into since the previous admin left some time ago.
    I have found we are seeing the following
    Error pn datax IncidentXmlNotificationServlet PN-1100: Java message: Error sending XML Notification. Incident Queue is full.
    and we no longer get the previously configured XML emailed actions when a rule gets hit.
    I have been looking trying to determine what and how needs to be cleared.
    Any insight would be appreciated.

    Thanks,
    Actually that is what I did (reboot) and functionality has returned.
    Thanks for suggesting something so obvious that I didn't think about that. I guess I wanted to make the issue and resolution more complex that it needed to be.

  • TCP Provider: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.

    Hi All,
    We have snapshot replication . Job is failing in between due to below error.
    Error messages:
    The process could not connect to Subscriber 'XX:SD'. (Source: MSSQL_REPL, Error number: MSSQL_REPL20084)
    Get help: http://help/MSSQL_REPL20084
    TCP Provider: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full. (Source: MSSQLServer, Error number: 10055)
    Get help: http://help/10055
    A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information
    see SQL Server Books Online. (Source: MSSQLServer, Error number: 10055)
    Get help: http://help/10055
    Login timeout expired (Source: MSSQLServer, Error number: HYT00)
    Get help: http://help/HYT00
    Please suggest

    Hi Khushi，
    According to your description, your snapshot replication job fails and you come across the error related to TCP Provider. Based on my research, the issue could occur as the following two situations.
    1. OS runs out of memory for TCP buffers. When people use the two switch: the /PAE switch in the c:\boot.ini file and the /3gb switch in the boot.ini file. If the applications require many OS resources, such as by opening many TCP connections, this can cause
    the OS to run out of memory for resources like TCP buffers.
    2.OS runs out of available TCP “ephemeral” ports. When the client machine is opening many TCP connections and is running Windows Server 2003, Windows XP, or any earlier version of Windows, it may run out of TCP “ephemeral” ports.
    To work around the issue, please follow the two solutions as below.
    1.Remove the /3gb switch from c:\boot.ini. The root problem in this case is memory pressure on the OS, so removing the /3gb switch will give more memory to the OS and will alleviate this problem.
    2.Make more ephemeral ports available following the steps in the article:
    http://support.microsoft.com/kb/196271
    For more information about the process, please refer to the article:
    http://blogs.msdn.com/b/sql_protocols/archive/2009/03/09/understanding-the-error-an-operation-on-a-socket-could-not-be-performed-because-the-system-lacked-sufficient-buffer-space-or-because-a-queue-was-full.aspx
    Regards,
    Michelle Li

  • How to recover when Delta Queues have been emptied by error ?

    Hi All,
    Here is our problem : the ECC Delta Qs have been emptied by error some days ago, and for the moment we have frozen every BW updates from ECC, but the Delta Queues have been being filled in ECC since that.
    We have so smart ideas so far...
    Thx 2 all in advance

    Hello ThoLei ,
    In case your normal delta is running file in BW and the only thing you are concerned about is the data missed during the day when the queue got deleted, then yes you can go ahead with Repair Full option. If you refresh the missing data and no need to worry about delta.
    Regards,
    Pratap Sone

  • "CPU Receive Multicast Queue is full"

    Hello.
    I am getting a Critical controller error on my WCS panel that says "CPU Receive Multicast Queue is full on Controller 'xxx.xxx.xxx.xxx'. "
    I have blanked out the IP above- it's the controller's IP address.
    What exactly does this error mean, and what can I do to fix this? I haven't had much luck in finding information on this particular error and it's starting to bother me.
    It triggers a couple times a day, or sometimes several days apart.
    My controller is running version 4.1.171.0 and my WCS is 4.1.83.0.
    Thanks!

    Hi Peter,
    There is a bug filed for the same CSCsf27201. Sometimes even when you are not using multicast traffic ,ARP and CDP packet keep hitting controller CPU and reports this message. Although these packets are dropped when que gets filled but WLC reports this message which will is already resolved in upcoming release.
    This message will not affect your network in anyway.
    HTH
    Ankur
    *Pls rate all helpfull post

  • Wism 7.0.230 Alarm: CPU Receive Multicast Queue is full on Controller

    Hello
    How can I get rid of this message ?
    "Cpu Receive Multicast Queue is full on Controller"
    This alarm appears every 5 minutes.
    I have this message only with Wism2, not with Ctrl 5508.
    Do you know if this affect controller performance ?
    Regards
    Michel Misonne

    There have been many post similar to yours, except you have the WiSM2. From other post, it seems like it doesn't affect performance but more of an eye sore. I guess you can wait to see if one of the Cisco engineer respond to your post and maybe try will be able to clarify why you see than my assumption of an "eye sore":)
    Here is from an older doc
    Q. I see a lot of "CPU Receive Multicast Queue is full on Controller" messages on the 2006 Wireless LAN Controller (WLC), but not on the 4400 WLCs. Why? I have disabled multicast on the controllers. What is the difference in the Multicast Queue Limit between the 2006 and 4400 WLC platforms?
    A. Because multicast is disabled on the controllers, the messages that cause this alarm might be Address Resolution Protocol (ARP) messages. There is no difference in queue depth (512 packets) between the 2000 WLCs and the 4400 WLCs. The difference is that the 4400 NPU filters ARP packets whereas everything is done in software on the 2006. This explains why the 2006 WLC sees the messages but not the 4400 WLC. A 44xx WLC processes multicast packets via hardware (through CPU). A 2000 WLC processes multicast packets via software. CPU processing is more efficient than software. Therefore, the 4400's queue is cleared faster, whereas the 2006 WLC struggles a bit when it sees a lot of these messages.
    Sent from Cisco Technical Support iPhone App

  • Android Back Button when FLVPlayback component in full screen

    I'm launching an FLVPlayback component in to full screen by setting fullScreenTakeOver = true and setting StageDisplayState to FULL_SCREEN in an air for android app. This works fine. The video plays in full screen
    But when I press the device Back button, it quits out of the app.
    Everywhere else in the app I can catch the key_down event and do preventDefault(), stopImmediatePropogation(), but the event doesn't seem to fire when FLVPlayback is in full screen.
    I've tried adding the Key_Down event listener to NativeApplication.nativeApplication, to stage and to the FLVPlayback component itself. None of these seem to work.
    I do, however, get the correct behaviour when I test the app in Flash IDEs simulator
    Thanks
    Ben
    EDIT:
    Also worth noting. Using latest Flash CC and AIR 17 testing on an S4

    www.giancarlogenise.com
    the video is in the promo section.
    if you navigate the site without pressing the Full screen button, everything is fine.
    If you click the full screen , strange behaviours every time
    I bought this template, but I have to add the video section by myself.

  • %ACE-2-901001 kernel: Warning:-MTS queue is full messages on ACE 4710

    I am getting the following messages repeated continually in my Admin context
    Apr 18 2012 14:42:27 CHIACE01: %ACE-2-901001 kernel: Warning:- MTS queue is full for opcode 4062 sap 63994 pid 18847 clear idle debug plugin sessions or telnet/ssh connections to recover
    How do i clear them? 
    How can I check whether i have idle debug sessions?
    or telnet/ssh sessions ( as far as i am aware no open telnet sessions, as they alway time out after 5 minutes)!

    Hi,
    Could be related to CSCtn85846
    Symptom:
    A warning message is displayed as follows: "Warning:- MTS queue is full for opcode %d sap %d pid %d clear idle debug plugin sessions or telnet/ssh connections to recover"
    Conditions:
    MTS messages to command shell are not processed as it executes user commands. Because a user command may take a while to terminate, the command shell can receive many MTS messages which won't be processed and run out of buffer space for the MTS queue. When this happens, a warning message is displayed.
    Workaround:
    This warning can be ignored. User can close all debug plugin sessions and terminate command execution in all telnet/ssh connections to stop display of warnings.
    Dan

  • Pq versus IPC queue monitoring (ipca)

    morning....
    There seems to be a discrepancy between what is reported via pq and what is visible through ipcs (the actual ipc queues). We have a script that polls pq (via tmadmin) and pumps statistics to a tool called Introscope used for presenting system stats historically. We also through a Java agent pump stats from Weblogic's WTC. On the same service without queue building in IPC there is in pq and a clear time difference (100+ms) between the imported WTC service and the actual Tuxedo service.
    This we have confirmed several times in our QA and production.
    My theory is that despite no OS queue building (ipc) Tuxedo as a container reacts to something internal to itself. If there is no queue building outside the container then it has to be inside?
    Another thing to note is I don't have any skills on how Tuxedo does the actual service call. Assuming it is a "ipc message queue" only that holds the call inclusive data (in-argument) while the pool of services offering the called service are busy. Or are semaphores involved to help out with the in-argument storage to keep the size of the data on the message queues down. What I am getting at is that the Tuxedo implementation is what gives rise to the diff between pq and ipcs.
    Edited by: MatthewDavidYoung on 2012-nov-15 00:35

    Hello Matthew,
    It is a common misconcpetion that the Tuxedo tmadmin pq command is the same as the operating system ipcs command.
    To force Tuxedo to do this there are some things that are needed that depend on the server you are interested in, the release of Tuxedo,
    and the rolling patch level of Tuxedo.
    For general application servers. Tuxedo 10.3 RP091 (Bug 13109309) enhances environment variable TM_SVRLOAD_REMEDY_ROUNDS
    to use ipcs results (e.g if set==50). See KM Doc ID 1463650.1
    For the GWTDOMAIN server. Tuxedo 8.0 RP391 (Bug 8130933),Tuxedo 8.1 RP298, Tuxedo 9.0 RP046 add environment variable
    TM_GWT_READIPCQUEUE. When set to 'y' tmadmin pq will provide ipcs results. See KM Doc ID 777521.1.
    Regards,
    Bob Finan
    Edited by: Bob Finan on Dec 5, 2012 5:01 PM

  • ThreadpoolExecutor not creating threads unless queue is full

    Hi I want to use the threadpool provided in java. i.e Threadpoolexecutor.
    The requirement is:
    1. I need to have a corepool size(1)
    2. I need to have Maxpoolsize(40)
    3. I need to have a Queue without predefined capacity. I can use a LinkedBlockingQueue for this.
    But the problem is, if i construct a threadpool,
    ThreadPoolExecutor threadPool = new ThreadPoolExecutor(1,40,600, TimeUnit.SECONDS,new LinkedBlockingQueue<Runnable>());
    No new threads are created until the queue is full. I.e the no more than corePoolSize threads will ever be created. (And the value of the maximumPoolSize therefore doesn't have any effect.). But what i need to have is everytime a task is submitted, it should create a thread until it reaches MaxpoolSize() and then it should start Queing requests.
    I know that this is a documented behaviour in javadoc. But i need to know if there is a proven workaround to do so.
    Is there a way to do So.....

    Please don't [double post|http://forums.sun.com/thread.jspa?threadID=5379100].

  • How do you "pause" an open application, like Mac automatically does when the memory is full?

    How do you "pause" an open application, like Mac automatically does when the memory is full?
    I would like to do so proactively when I am not using a certain program but need to keep it open, and I want to free up computer resources.
    In other words, I want to keep a given program open, but stop it from "thinking".

    There is an application that I can only open a set number of times before it expires.  Every time the program closes, you get one less use out of it.  I want to use it as long as possible, so I thought I would just keep it open all the time.  However, it consistently turns on the fan on the computer, so I imagine it uses a lot of resources.  I got the idea after the Force Quit program "paused" it once after getting overloaded with all the apps I had open.  It stopped the fan, and I was able to open it and run it again later with no problems.  The tricky thing with trying to intentionally overload the system is that it's a hit or miss on which open program the OS will pause (to my knowledge).

  • Can I recover emails that were blocked when my storage was full?

    The storage on icloud was full for me for perhaps a number of days. I bought extra storage (20GB) and my emails are now working, but how can I recover the emails that were blocked when my storage was full?

    Yes.  I tried the refresh and checked the uninstall tab.  Still nothing.  Another thing though - when I entered my Blackberry ID, it would not let me use my screen name because it was supposedly already in use.  But I knew that because it was in use by me.  It would only let me use the same name with a numerical suffix.  So I am wondering if it thinks I am still logged on from before the upgrade.  Also, is it possible the Playbook thinks I am some other use and won't let me access my files - like photos and videos, etc.

  • When I go into full screen mode in Safari 5.1 on LION it switches to a new Space, making my other displays useless. Anyone else annoyed that Apple has completely left out those with multiple displays?

    I had a fear that this was going to be the case, but you can't have two full screen apps open in two different displays. At least that's how Safari fullscreen works. When switching over to full screen mode in safari, it automatically creats a new space, turns on four finger gesture to swipe through full screen apps which I had turned off in place of App Switcher.
    I don't use spaces because of the lack of support for using spaces with multiple displays. I would use spaces and mission control if I had control of each display's mission control separately.
    The fix would be to unlink the displays so they can scroll through full screen apps by themselves. I want this feature so I can mix and match the full screen apps on my three different displays. I see this beneficial for more people with multiple displays. in the future when thunderbolt display links, people will want better support too.
    I know I would use spaces and mission control more if I could have this ability. I'd be interested to know people's opinion.

    No no, if you read what I said, it would be that each individual display would be customizable to the user. The way you have it, it sounds like your second display will be dedicated to scrolling through fullscreen apps not allowing you to do it on the home screen. This is silly because with my concept, you could do that, by dragging apps to fullscreen in your second display, but if you want you could tell another display to scroll through fullscreen apps (independently) if you wanted. Each display will have it's own mission control with it's own set of desktops. I also can see how some people would still want them linked together, so this could just be a little check box made in mission control prefs to "link" or "unlink" desktops with link checked by default.
    With your solution you're limited to your setup. but what if the person has three or four displays? Well then your solution would limit them to one dedicated display that scrolls through fullscreen apps where mine lets a person customize their desktops depending on their preference and setup.
    further example: if you have safari, address book and mail in one display, you could still have ichat, PS and Illustrator in your second, and a text editor, FTP Client and iTunes in your third. Then you would mix and match these apps but it would promote the use of mulitple displays as well as benefit people with three or more, which thunderbolt, Cinema Display and Pro computers support.

Maybe you are looking for