How to REALLY monitor a J2EE-engine?

Question

We recently started implementing an EP 6.0 project as well as a NWDI infrastructure to develop some thing for Internet Sales on CRM 2005.
I installed a NW2004s SR1 and patched the instance up to SP7 - so latest and greatest.
However, doing a search for "exception" in the log directory I get a number of 201670 (yes, you've read right).
Using the logviewer I see on each start and during runtime ~ 200 - 300 exceptions that are getting thrown, some of them often, some of them only one - so I counted about 80 - 90 different exceptions.
How should I as an administrator find out, which ones of them are crucial for the application, what is a real error and what is just an exception, because something is installed but not (yet) configured?
Setting up NetWeaver Administrator will not help much since I would get 90 mails every day with different errors and I would see there also those 90 different errors.
How to distinguish between relevant/irrelevant exceptions/errors? There's no source code around and I doubt, that it's necessary for an administrator, to learn the full J2EE API just to get those informations. It can't also be that I need to create OSS calls to get an answer from the development.
Where is some documentation what part of the engine is used where, what errors can be ignored and which of them needs to be followed?
Markus

Answer 1

Apologies, this thread is old but worth a mention.
No problem with that!
I was in ramp up for 630 J2EE and starting from the beginning it was clear that ERROR means there is some type of error that can even be of user cause and so ERROR was a rubbish thing to monitor (a bit like looking for red messages in SM21).
We don´t have many "red messages" in SM21 - and if, then there is really a problem (like a dump)
However, right from the start I have been alerting on CRITICAL errors as these mean trouble. Seems to work well..
There are less of them - true - but they are even less speaking:
#1.5^H#00118513360B002E000000250000399900044B606D5EC8B1#1208777430714#com.sap.engine.services.failover.st
orage#sap.com/irj#com.sap.engine.services.failover.storage#Guest#0##n/a##55b6fe700f9611ddbf4800118513360b
#SAPEngine_Application_Thread[impl:3]_31##0#0#Error##Java###null
[EXCEPTION]
{0}#1#com.sap.engine.services.failover.exceptions.PersistentStorageException: Exception occurred during
I/O operations.
        at com.sap.engine.services.failover.storage.DatabasePersistentStorage.serializeSession(DatabasePe
rsistentStorage.java:123)
        at com.sap.engine.services.servlets_jsp.server.container.ApplicationThreadDestroyer.serializeSess
ions(ApplicationThreadDestroyer.java:104)
        at com.sap.engine.services.servlets_jsp.server.container.ApplicationThreadDestroyer.run(Applicati
onThreadDestroyer.java:72)
or
#1.5^H#00118513360B00070000000100006B1100044D3854843446#1210804238299#com.sap.engine.services.security.po
licyconfiguration##com.sap.engine.services.security.policyconfiguration####n/a##60ab1ab0220511dda5f700118
513360b#SAPEngine_System_Thread[impl:5]_67##0#0#Warning#1#/System/Security/PolicyConfiguration#Java###Exc
eption occurred during remove of policy root [{0}].
[EXCEPTION]
{1}#2#service.iiop#com.sap.engine.frame.core.configuration.NameNotFoundException: A configuration entry
with the name "service.iiop" cannot be found in the configuration "security/configurations".
        at com.sap.engine.core.configuration.impl.ConfigurationDataCacheAllImpl.getValue(ConfigurationDat
aCacheAllImpl.java:98)
        at com.sap.engine.core.configuration.impl.ReadAccessConfiguration.getConfigEntry(ReadAccessConfig
uration.java:849)
        at com.sap.engine.services.security.server.PolicyRoots.removePolicyRoot(PolicyRo
I gave up on Java.
We run the instances and in case of a problem we restart all affected Java systems subsequently until the error vanishes because trying to "find" an error in a federated portal network (with some scenarios 6 or 7 machines affected) is not only cumbersome and very time consuming but in most cases meaningless because the support needs a reproducibility, a case, otherwise they can´t help - and recommend to "install the latest support packages". This is also true for production system. In case I open a prio-1 call I don´t only got hold off sleep because they are working on the call 24 h but we finally end in restarting the engines though.
There´s one call open since June 2006 - which is now two years old - about the dependencies in the packages and how to find out which service/applications needs which, to monitor only the really necessary ones. Since the development itself is not able to tell which dependencies they generate then, well, it´s better for everyones nerves and finally health to "just reboot" as you were (and maybe are) doing with Windows boxes if they don´t behave as they should. Maybe it´s by design/defnition like this, I don´t know.
We´re now in the process of putting all Java instances onto clusters to do an automatic switchover so the instances get restarted every few days and we avoid any problems that are caused by instances running too long.
I wish there was another solution or an idea of how to deal with that - I don´t know how others handle that but I have to take of about 50 instances (ABAP + Java) so I don´t just have the time of collecting dozens of logfiles of half a dozen machines and open an OSS call try to explain someone at the other side of the world in a foreign language (english is not my native language) what´s happening. The problem starts with "in which component do I open the call?"
Markus