Java APM Agent initialization failed: SEAgent.config.xml does not have a valid event sender configured

Question

Today I tried to configure JAVA APM on a CentOS 6.4 server with tomcat6 installed through the normal RPM package. Adding this server to SCOM was no problem and it shows itself as a 'Deep Monitored Configuration'. The last part of enabling
JAVA APM fails. I have followed the following process:
1. Imported the JAVA APM MP's
2. Extracted the JAVA APM Files
3. Copied those files to my CentOS server
4. Added the JAVA_OPTS to my tomcat configuration file
5. Restarted Tomcat
When I look for the log file which needs to be created through the installation of the APM agent it cannot be find. When I look into my Catalina.out file I see the following errors:
Mar 05, 2014 7:05:16 PM com.microsoft.ManagementServices.APMAgent.Log.SystemLogger error
SEVERE: Java APM Agent initialization failed: SEAgent.config.xml does not have a valid event sender configured
com.microsoft.ManagementServices.APMAgent.Exception.ApmException: SEAgent.config.xml does not have a valid event sender configured
        at com.microsoft.ManagementServices.APMAgent.AgentThreadManager.<init>(Unknown Source)
        at com.microsoft.ManagementServices.APMAgent.AgentThreadManager.<init>(Unknown Source)
        at com.microsoft.ManagementServices.APMAgent.Agent.Monitor.AgentContext.<init>(Unknown Source)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at com.microsoft.ManagementServices.APMAgent.Agent.Startup.AgentContextLoader.loadContext(Unknown Source)
        at com.microsoft.ManagementServices.APMAgent.Agent.Startup.Agent.premain(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:382)
        at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:397)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        ... 15 more
Caused by: com.microsoft.ManagementServices.APMAgent.Exception.ApmException: Error registering MBean: com.microsoft.ManagementServices.APMAgent:type=COUNTER
        at com.microsoft.ManagementServices.APMAgent.EventSender.MBeanEventLog.registerMetricMBeans(Unknown Source)
        at com.microsoft.ManagementServices.APMAgent.EventSender.MBeanEventLog.<init>(Unknown Source)
        at com.microsoft.ManagementServices.APMAgent.EventSender.MBeanEventLog.<init>(Unknown Source)
        at com.microsoft.ManagementServices.APMAgent.AgentThreadManager.<init>(Unknown Source)
        at com.microsoft.ManagementServices.APMAgent.AgentThreadManager.<init>(Unknown Source)
        at com.microsoft.ManagementServices.APMAgent.Agent.Monitor.AgentContext.<init>(Unknown Source)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at com.microsoft.ManagementServices.APMAgent.Agent.Startup.AgentContextLoader.loadContext(Unknown Source)
        at com.microsoft.ManagementServices.APMAgent.Agent.Startup.Agent.premain(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:382)
        at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:397)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        ... 15 more
Caused by: com.microsoft.ManagementServices.APMAgent.Exception.ApmException: Error registering MBean: com.microsoft.ManagementServices.APMAgent:type=COUNTER
        at com.microsoft.ManagementServices.APMAgent.EventSender.MBeanEventLog.registerMetricMBeans(Unknown Source)
        at com.microsoft.ManagementServices.APMAgent.EventSender.MBeanEventLog.<init>(Unknown Source)
        at com.microsoft.ManagementServices.APMAgent.EventSender.MBeanEventLog.<init>(Unknown Source)
        ... 19 more
Mar 05, 2014 7:05:16 PM com.microsoft.ManagementServices.APMAgent.Log.SystemLogger info
INFO: Java APM Agent loading failed (Build: 1.10.100.15, Label: NotFromLabel, BuildDate: 20131112)
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at com.microsoft.ManagementServices.APMAgent.Agent.Startup.AgentContextLoader.loadContext(Unknown Source)
        at com.microsoft.ManagementServices.APMAgent.Agent.Startup.Agent.premain(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:382)
        at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:397)
Caused by: java.lang.InstantiationException: Monitor initialization failed: SEAgent.config.xml does not have a valid event sender configured
        at com.microsoft.ManagementServices.APMAgent.Agent.Monitor.AgentContext.<init>(Unknown Source)
        ... 12 more
Does anyone have seen this error before? Can someone help me with this one?
Thnx,

Answer 1

I'm running tomcat on a CentOS release 6.4 (Final) server. I installed Tomcat 6 through an RPM from the CentOS sources. I have installed Apache Tomcat/6.0.24 wit JVM version 1.7.0_09-icedtea-mockbuild_2013_01_16_18_52-b00, the JVM Vendor is Oracle Corporation.
My tomcat configuration needs to be done in '/etc/tomcat6/tomcat6.conf'
I installed the APM Agent in the path '/opt/apm/' with the rights  40755 (root/root). The files have the following rights:
-rw-rw-r-- 1 root tomcat  32046 Mar  3 15:38 apm_facade.jar
-rw-rw-r-- 1 root tomcat 244819 Mar  3 15:38 apm_monitor.jar
-rw-rw-r-- 1 root tomcat  38110 Mar  3 15:38 apm_producers.jar
-rw-rw-r-- 1 root tomcat 207006 Mar  3 15:38 asm-all-3.3.1.jar
-rw-rw-r-- 1 root tomcat   4157 Mar  3 15:38 ASM_Third_Party_Notices.txt
-rw-rw-r-- 1 root tomcat 241368 Mar  3 15:38 DispatcherStub.dll
-rw-rw-r-- 1 root tomcat  36880 Mar  3 15:38 pmonitor.config.xml
-rw-rw-r-- 1 root tomcat   2625 Mar  3 15:38 SEAgent.config.xml
-rw-rw-r-- 1 root tomcat    830 Mar  5 15:19 Starter.properties
The above rights are equal to the /etc/tomcat6 directory with the tomcat configuration files.
In the tomcat6.conf I have added the following entries:
#Microsoft SCOM APM
JAVA_OPTS="${JAVA_OPTS} -Djava.library.path=/opt/apm"
JAVA_OPTS="${JAVA_OPTS} -javaagent:/opt/apm/apm_facade.jar -Xbootclasspath/p:/opt/apm/apm_producers.jar -Xbootclasspath/p:/opt/apm/apm_facade.jar"
When I look into the running tomcat process I see the following:
tomcat   17401     1  0 Mar05 ?        00:00:39 /usr/lib/jvm/jre/bin/java -Djava.library.path=/opt/apm -javaagent:/opt/apm/apm_facade.jar -Xbootclasspath/p:/opt/apm/apm_producers.jar -Xbootclasspath/p:/opt/apm/apm_facade.jar
-Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -Djava.library.path=/opt/apm -javaagent:/opt/apm/apm_facade.jar -Xbootclasspath/p:/opt/apm/apm_producers.jar -Xbootclasspath/p:/opt/apm/apm_facade.jar -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory
-classpath :/usr/share/tomcat6/bin/bootstrap.jar:/usr/share/tomcat6/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat6 -Dcatalina.home=/usr/share/tomcat6 -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat6/temp
-Djava.util.logging.config.file=/usr/share/tomcat6/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
My Starter.Properties looks like this:
# Jars to load by inner classloader.
# Use semicolons to separate the JAR entries
ClassPath=apm_monitor.jar;asm-all-3.3.1.jar;apm_producers.jar
# Path to pmonitor.config
MonitorConfigFileName=pmonitor.config.xml
# Path to SEAgent.config
AgentConfigFileName=SEAgent.config.xml
# Log file name
# Example for full-path location:
#                  LogFileName=c:\\tracelog\\apm-java-agent.log
# Example for relative path (to agent jar files):
#                  LogFileName=apm-java-agent.log
LogFileName=apm-java-agent.log
# Log level (SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST)
# A level of FINER or FINEST may expose Personally identifiable
# information (PII) in the log file.  The levels below INFO are
# intended for debugging and diagnostic use only.
LogLevel=FINEST
The 'apm-java-agent.log' is not created on the system. The only logging is done in the catalina.out. When restarting tomcat I see the following entries in te log file:
INFO: Initializing Java APM Agent (Build: 1.10.100.15, Label: NotFromLabel, BuildDate: 20131112)
Mar 06, 2014 9:35:21 AM com.microsoft.ManagementServices.APMAgent.Log.SystemLogger info
INFO: Java APM Agent loading succeeded (Build: 1.10.100.15, Label: NotFromLabel, BuildDate: 20131112)
Mar 06, 2014 9:35:21 AM com.microsoft.ManagementServices.APMAgent.Log.SystemLogger info
INFO: Initializing Java APM Agent (Build: 1.10.100.15, Label: NotFromLabel, BuildDate: 20131112)
Mar 06, 2014 9:35:21 AM com.microsoft.ManagementServices.APMAgent.Log.SystemLogger warning
WARNING: Failed registering the COUNTER MBean
Mar 06, 2014 9:35:21 AM com.microsoft.ManagementServices.APMAgent.Log.SystemLogger error
SEVERE: Java APM Agent initialization failed: SEAgent.config.xml does not have a valid event sender configured
Mar 06, 2014 9:35:21 AM com.microsoft.ManagementServices.APMAgent.Log.SystemLogger info
INFO: Java APM Agent loading failed (Build: 1.10.100.15, Label: NotFromLabel, BuildDate: 20131112)
Hope this gives your some more background around my problem, I double checked the permissions..