Tuxedo process IPC queue monitoring

Question

Hi
This question is aimed mostly at Oracle techies who have experience supporting Tuxedo sites.
We've been developing on Tuxedo for a couple of years now and are about to go live with a Tuxedo based transactional system.
Throughout the development and testing phase it has become pretty apparent to us that we need to be able to keep an eye on IPC request queues for processes running under Tuxedo.
For example, we have 10 instances of server "A" running within Tuxedo. As long as these ten instances can cope with the load, we are getting good end-to-end response times. When we overload the system whereby our requests start being queued up on IPC queues for these processes, quite predictably our end-to-end response times go up. We would like to get an early warning when situations like this occur.
Sometimes there are obvious reasons for this behaviour (i.e. we're sending more requests per seconds than our application can cope with) and at other times, it is a degradation in performance of a downstream process (e.g. database query taking longer and longer to return as time goes by) that is causing our requests to start backlogging. It is this latter scenario that we would like to be alerted to before our clients ring us up to tell us that our service is not performing as well as it should.
However, there seems to be a distinct lack of importance placed on server request queue monitoring within Tuxedo infrastructure and API set. Perhaps we are missing something obvious?
We would like to be able to have a more proactive monitoring and reporting facility in place to help us observe queuing behaviour. It seems that there are two ways to obtain any queuing diagnostics:
1) Use a shell script to call tmadmin/pq from time to time to take snapshots or use OS commands to monitor IPC queues
2) Use Tuxedo MIB and poll the queue OIDs to get queue depth snapshots.
We would like to be able to get a lot more than that. For example:
* to be able to plot a graph of queue depths over a period of time
* get alerted when queue depths cross a certain limit (trap vs poll)
* work out how long an item spent on the queue before it got processed
So, are we placing too much emphasis on something that is not important to other Tuxedo users or is there a suite of tools one can use to obtain this sort of information that we are not aware of?
How important is request queuing to a typical Tuxedo site.
Regards,
Mike

Answer 1

Hi Mike,
You might want to look at the Oracle Tuxedo System and Application Monitor (TSAM). It is an add-on product from Oracle that provides system and application monitoring for Tuxedo applications. Although TSAM doesn't currently give you the kind of graph you are looking for, it does collect that information and place it in the TSAM monitoring database. TSAM provides the schema of the monitoring database, so creating whatever graphs you want should be pretty easy.
Specifically what I would look at though is TSAM's alert generating mechanism. Alert information can either be checked at the monitoring console, or it can be checked at the monitored nodes plug-in. The later offers the option of simply checking for an alert without having to report the data back to the console unless an alert is triggered. This tends to have much less impact on the monitored system and help keep the monitoring console from being flooded with data. In the Call Path and Service monitoring categories, you can define alerts that check if the number of queued messages exceed a certain amount. This is quite efficient and would have minimal impact on system performance, yet warn you when queues are starting to back up.
These two methods would give you historical data to look for trends as well as real time alerting to be notified that backups are occurring.
Regards,
Todd Little
Oracle Tuxedo Chief Architect