Java Application server monitoring?

Question

We have a server farm for a project which involves custom and commercial (third parth proprietary) j2ee
web-applications on varied application servers such as Tomcat, GlassFish, JBOSS, Sun Web/Application
servers, etc. running under Solaris 10.
The hardware/OS parts are already monitored by SMC 4.0; we are now trying to understand whether (how)
we can monitor individual web-applications.
A bit too often we have some web-application misbehaving without anyone noticing for a long time, and
the server as a currently monitored object (an http engine or an OS application) works just fine.
So far I see a few ways to do it, perhaps some ways are already-known-wrong?
Or perhaps some of my assumptions below are incorrect...
1) Set up a "response filter" in app servers, so any error output from applications would be intercepted and
acted upon (i.e. snmp trap, email message, simple error message for users instead of stacktraces, etc.)
- this leaves the app server as a bottleneck which decides whether it has failed; if the failure is due to
lack of system resources, jvm bugs, gamma rays from outer space or phase of the moon - this "decider"
can fail and either give a wrong answer, or none at all.
- however failure of the application server as an OS application and/or generic HTTP-responder can be
discovered by other, more generic means...
2) Set up a web crawler (like Halcyon's HTTPping) to send specific requests to each web-application's
URL, parse the results and see whether they fit our expectations. For simplicity it can be a single
JSP page which runs some predefined tests and outputs status.
- each application's status-checking methods should be custom (URL, request parameters, expected
response template should be maintained) and probably should be part of the applcation's spec, just
like its any other code.
- the simple solution with just one URL and one piece of logic to test can be a shortcoming too, though
it can help catch lowlevel problems such as connection pool errors, etc.
- for reliability (decision not made within app server) some part of logic should be offloaded to an agent
which should implement/use a monitoring engine API (such as SMC)
3) Parse access/error log files of the servers (find HTTP 4xx/5xx results, stacktraces and other typical
error signs)
- an async method - we learn of the problem some time after the failure. Depending on the situation
this may be acceptable or not.
- not all servers maintain access logs (i.e. I haven't seen a Tomcat or Jboss installation write an access
log by default), and it is a recommended part of performance tuning on Sun App Server - to disable
access logging - so by default we don't have a simple means to grep for HTTP 4xx/5xx statuses
- parsing error logs seems like a nightmare because they are inherently so unstructured...
I am also not yet certain how we can feed results of any tests into SMC - as results of scripts which
are executed by SMC itself, as SNMP traps, SMTP/JMS messages or in some other manner, or all
of the above?
Thanks for any ideas and facts,
//Jim
Edited by: JimKlimov on May 5, 2009 1:21 PM
PS: I meant to mention one more implication. Many of these web-applications we need to monitor are not
quite user-interface guis. Most of them are SOAP webservices, speaking their protocol over HTTP.
Just in case this aspect matters...

Answer 1

Hi Jim,
You've mentioned that you've already seen Halcyons HTTPping utils, but if you haven't looked in awhile there are also addons specifically for Sun App Server (and Glassfish), as well as modules for regular JVM monitoring, all of which you can try as eval downloads or buy from your Sun rep.
Regards,
[email protected]