How to set up automatic syslog configuration?

Question

A colleague recently asked: "I am wondering whether I need to specify
the "appid" (something like "nqcg") when I connect the collector to the
syslog event source. If yes, do I specify it in the "Syslog Message
Filter" dialog? It's asking for a message filter pattern. Do I put in
something like nqcg:*? Or, do I specify my "appid" as the "Applications"
property value in the Syslog:Map Output(appid) connection mode XML node
in the "connectionMethods.xml" file?
The answer to this question lies in the Syslog Connector documentation,
mostly, and some samples are provided in the template, but it's left up
to the developer, somewhat, to figure out how to put all the pieces
together.
DCorlette
DCorlette's Profile: http://forums.novell.com/member.php?userid=4437
View this thread: http://forums.novell.com/showthread.php?t=441089

Answer 1

OK, so first some background:
- The Syslog Connector is pretty sophisticated - it will automatically
apply a proper syslog header if the event source does not provide one,
and will also pre-parse out a number of key pieces of information from
syslog headers (generated and provided) to provide metadata about the
event records received.
- The Connector will, for example, extract the facility etc, date/time,
and the Observer information from the syslog header. It will also figure
out which system is sending it the data (which may or may not be the
same as the actual Observer) and report that as well. All this metadata
is attached to the actual received record in various record fields like
's_syslogRelayIP' etc (see the Connector doc for details, which vary
slightly between different versions of the Connector).
- In addition, the Syslog Connector will take a peek at the first chunk
of data in the actual record body (usually up to the first ':'
character, and based on that set the 's_AppId' field. This often
corresponds to the name of the application that generated the actual
event record.
- So, the Syslog Connector extracts all this useful stuff - what does
it do with it? Well, this is where the Collector comes in - essentially,
the Connector can "register" to handle specific sets of data. There are
three layers here:
1) The Collector can declare itself to be the
'UniversalSyslogCollector'. In this case, all event records that are
otherwise unallocated will come to this Collector. Obviously, this
should only be one Collector in the system at a time, and is typically
our provided Generic Event Collector.
2) The Collector can register to handle specific applications. In this
case, for each and every event received if the Syslog Connector detects
the associated application ID, it will route that data to the Collector
that knows how to handle that data. If necessary, it will auto-create a
new Event Source node in ESM to represent the single application running
on the single Observer source that is sending data to that Collector.
3) The Collector can supply a regular expression matching pattern to
help the Connector identify the Collector to which Observer data should
be sent. In this scenario, all input records are checked against the
regex until the match is found; then all the Observer's data is sent to
the associated Collector.
- The way that these nest can be a little confusing, so I'll explain a
bit further here in terms of the chain of actions that take place when a
new event source starts sending data to Sentinel:
a) When a new connection is detected from a new event source
(determined from the syslog header hostid, which is either an IP or
hostname), a new Event Source node is immediately created in ESM to
represent the new source (well, actually, this is determined by policy
on the Event Source Server). This node is attached to the Generic Event
Collector initially, since we don't yet know anything about the source.
b) Inbound records will be compared against all the available
registered regex matching patterns for all installed Collectors; if a
match is found, then the Event Source is determined to be the product
for which the associated Collector was written - so for example it might
figure out that the stream of data coming from a particular server is
SUSE Liinux data. The Syslog Connector will then instantiate a new
instance of the SUSE Linux Collector (if necessary), and *move* the
existing Event Source node to be handled by that Collector.
c) At the same time, all inbound records are compared against the set
of registered application IDs, and if a match is found, another Event
Source node is created and attached to a new instance of the associated
Collector.
It might not be obvious why the system works this way, but the reason
is that we often have scenarios where a particular application can
actually run on several different platforms. So, for example, Apache Web
Server can run on both SUSE Linux and Solaris - we have Collectors for
all three. A new event source will start sending data such as logins,
account creates, etc and eventually the Syslog Connector will figure out
"hey, that's a Solaris box" and route the data appropriately. At the
same time, however, if events from 'httpd' (Apache) arrive intermingled
with the same event stream, then they will be routed to the Apache
Collector instead of the Solaris Collector.
DCorlette
DCorlette's Profile: http://forums.novell.com/member.php?userid=4437
View this thread: http://forums.novell.com/showthread.php?t=441089