Automatic log files monitoring

Hi all!
is there any tool available which is capable of checking or monitoring my log files, maybe against some (pre)defined rules, and alerts me if some error/failure occurs?
Best regards!

Splunk is nice, but it is non-free. It looks like fluentd (AUR) can do the same stuff, though. I haven't tried fluentd personally, but it looks pretty cool.

Similar Messages

Text log file Monitor

Hi Team,
i have a task to create a log file example SCOM.log, the log has all text entries and the pattern is like below
"[0014 20140724 094527069 SCOM E] ProcessDeposits(), DMGATEWAY internal error in plugin: An exception occurred while processing terminal transaction with HostTransactionID: '4143', tracking fact Id: '3' in the Deposit Gateway Processor task 7. The transaction
will be NOT be re-queued. // "
we need to monitor anything which has "DMGATEWAY internal error in plugin:" string.
sometimes we have greater then 10 entries with in few seconds.
once the log file reaches 10MB all the entries are moved to an archive log file that means the present SCOM.log file is empty and start from line one again.
any suggestions \ solution will be a great help and also let me know if you need any further information
-Vrkumar01
RajKumar

Hi Raj,
Have a read through this post it highlights issues with log monitoring: http://social.technet.microsoft.com/Forums/systemcenter/en-US/827464fd-ff06-495d-8ac6-4a6e337314d3/bug-in-scom-log-file-monitor?forum=operationsmanagergeneral
Have a look at creating a script monitor to monitor the log file:
http://www.opsmanager.se/2012/11/06/text-log-monitoring-part-1/
http://www.opsmanager.se/2012/12/17/text-log-monitoring-in-operations-manager-part-2/
The issue you are experiencing is a limitation of scom which is documented here:
http://support.microsoft.com/kb/2691973/en-us
Snippet from KB:
Additional Information
When monitoring a log file, Operations Manager remembers the last line read within the file (a 'high water mark'). It will not re-read data before
this point unless the file is deleted and recreated, or renamed and recreated, which will reset the high water mark.
If a logfile is deleted and recreated with the same name within the same minute, the high water mark will not be reset, and log entries will
be ignored until the high water mark is exceeded.
An implication of this is that log files that are cleared periodically without being renamed and recreated, or deleted and recreated, will not have entries in them processed until the high water mark from before the log is cleared is exceeded.
Operations Manager cannot monitor 'circular log files' (i.e. log files that get to a certain size or line count, then start writing the newest entries at the beginning of the log) for the same reason. The log file must be deleted or renamed and then recreated,
or the application configured to write to a new log once the current log is filled.
Cheers,
Martin
Blog:
http://sustaslog.wordpress.com
LinkedIn:
Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

Setting up a Log File monitor to inactivity for a set amount of time

I have set up a number of log file monitors to alert when certain conditions apply, such as the word "ERROR" or "exception". Now I have a request to set up an alert if the log file has not changed for 20 minutes. I have been
searching and have not found any information on how or if this can be done. Anyone???
I am running Operations Manager 2012 SP1
The log files are simple text files.

Hi!
You could create a timer reset monitor that reads the log file every 19 minutes for a wildcard pattern (everything matches) and configure the successful search to healthy. Further, you've to configure the timer reset to 20 minutes and configure the
timer reset state to unhealthy (warning/critical).
Keep in mind that SCOM reads from the last line from the previous run every time. If your file rotates (based on a schedule or size) SCOM will not read the lines until the latest line is reached. For more information refer to
http://www.systemcenterrocks.com/2011/06/log-file-monitoring.html
HTH, Patrick
Please 'Propose/Mark as answer' if this post solved your problem. <br/> <br/> http://www.syliance.com | http://www.systemcenterrocks.com

Alert Log File Monitoring of 8i and 9i Databases with EM Grid Control 10g

Is it possible to monitor alert log errors in Oracle 8i/9i Databases with EM Grid Control 10g and EM 10g agents? If yes, is it possible to get some kind of notification?
I know that in 10g Database, it is possible to use server generated alerts, but what about 8i/9i?
Regards,
Martin

Hello
i am interested in a very special feature: is it possible to get notified if alerts occur in alert logs in an 8i/9i database when using Grid control and the 10g agent on the 8i/9i systems?
Moreover, the 10g agent should be able to get Performance Data using the v$ views or direct sga access without using statspack, right?
Do you know where I can find documentation about the supported features when using Grid Control with 8i/9i databases?

Log files can't be removed automatically in HA environment

Hi BDB experts,
I am writing db HA application based on bdb version 4.6.21. Two daemons run on two machines, one as master which will read/write db, one as client/backup will only read db. There is one thread in master daemon that run checkpoint every 1 second: dbenv->txn_checkpoint(dbenv, 1, 1, 0), and dbenv->log_archive(dbenv, NULL, DB_ARCH_REMOVE) will be called after runnng checkpoint each time. The env was open with flag: DB_CREATE | DB_INIT_TXN | DB_INIT_LOCK | DB_INIT_LOG | DB_REGISTER | DB_RECOVER | DB_INIT_MPOOL | DB_THREAD | DB_INIT_REP; Autoremove flag was set by: envp->set_flags(uid_dbs.envp, DB_LOG_AUTOREMOVE, 1) before open env.
I found this thread https://forums.oracle.com/message/10945602#10945602 which discussed about non-ha environment, and I tested my code in a non-ha env without DB_INIT_REP, it worked. However in HA env those log files were never removed. Could you help on this issue? Does the client need to run checkpoint? May there be a bdb bug?
Thanks,
Min

There is one thread in master daemon that run checkpoint every 1 second: dbenv->txn_checkpoint(dbenv, 1, 1, 0), and dbenv->log_archive(dbenv, NULL, DB_ARCH_REMOVE) will be called after runnng checkpoint each time. The env was open with flag: DB_CREATE | DB_INIT_TXN | DB_INIT_LOCK | DB_INIT_LOG | DB_REGISTER | DB_RECOVER | DB_INIT_MPOOL | DB_THREAD | DB_INIT_REP; Autoremove flag was set by: envp->set_flags(uid_dbs.envp, DB_LOG_AUTOREMOVE, 1) before open env.
I am not saying that this is causing a problem, but doing the DB_ENV->log_archive(DB_ARCH_REMOVE) in your thread and setting DB_ENV->set_flags(DB_LOG_AUTOREMOVE) is redundant. In your thread, you control the timing. The DB_ENV->set_flags(DB_LOG_AUTOREMOVE) option checks for and removes unneeded log files when we create a new log file.
Did you see in the documentation for DB_ENV->set_flags(DB_LOG_AUTOREMOVE) that we don't recommend doing automatic log file removal with replication? Although this warning is not repeated in DB_ENV->log_archive(DB_ARCH_REMOVE), it also applies to this option. You should reconsider using this option, particularly if it is possible that your client could go down for a long time.
But this is only a warning and automatic log removal should work. My first thought here is to ask whether your client has recently gone through a sync? Internally, we block archiving on the master during some parts of a client sync to improve the chances that we will keep around all logs needed by the syncing client. We block archiving for up to 30 seconds after the client sync.
I found this thread https://forums.oracle.com/message/10945602#10945602 which discussed about non-ha environment, and I tested my code in a non-ha env without DB_INIT_REP, it worked. However in HA env those log files were never removed.
This thread is discussing a different issue. The reason for our warning in BDB 4.6 against using automatic log removal with replication is that it doesn't take into account all the sites in your replication group, so we could remove a log from the master that a client still needs.
We added replication group-aware automatic log removal in BDB 5.3 Replication Manager, and this discussion is about a change of behavior from this addition. With this addition, we no longer need to recommend against using automatic log removal with replication in BDB 5.3 and later releases.
Could you help on this issue? Does the client need to run checkpoint? May there be a bdb bug?
I'm not sure the client needs to run its own checkpoints because it performs checkpoints when it receives checkpoint log records from the master.
But none of the log removal options on the master does anything to remove logs on the client. You will need to perform steps to archive logs separately on the client and the master.
Paula Bingham
Oracle

Alert Email notification for Log file alerts

Hi,
Scenario: SCOM 2012 R2 UR4.
There are created unix/linux log file monitoring objects. In SCOM console I can view alerts related to unix/linux log file monitoring. Email notification is: Warning or Critical for severity, and, Medium or High for priority. The alerts for unix/linux log
file are severity warning and priority medium.
In my inbox there are emails for alerts (Warning or Critical for severity, and, Medium or High for priority) except for unix/linux monitoring.
The question is:
How to enable email notification for unix/linux log file monitoring?
Thanks in advance!

Hello,
If you go into the "Subscription" in the Notifications section of the Operations Console\Administration, you should be able to see the Description of the subscription criteria. Could you copy paste that in a reply?
Thanks,
Kris
www.operatingquadrant.com

Getting Log File Pattern Matched Line Count metric to work ?

Hi
has anyone been able to get this to work with more complex Perl expressions ?
Basically I can get simple, single expressions to match.
EG *(does not exist)* will match the text *"does not exist"* anywhere in a file.
However, if I want to match either does not exist OR file not found I should be able to do something like
*(does not exist)|(file not found)* OR *(does not exist|file not found)* but this just doesn't work.
I want to be able to do more complex expressions, using *\i* (ignore case), and *^* (start of line) *$* (end of line) expressions too.
I can test the matching functionality using a simple perl program, and I know the expression works in Perl.
Oracle is supposed to be using a perl pattern match but seems to fail unless it is a single simple expression.
Anyone been able to use this functionality at all.
Many thanks.

I have a chance to look into the parse-log1.pl script which is responsible for monitoring the log files and generating the alerts for EMGC. I am just pasting the comments given in this file
# This script is used in EMD to parse log files for critical and
# warning patterns. The script holds the last line number searched
# for each file in a state file for each time the script is run. The
# next run of the script starts from the next line. The state file name
# is read from the environment variable $EM_STATE_FILE, which must
# be set for the script to run.
but in my case this is not happending according to log files it is storing the lst read line of the log file but it is not using that info in its next run. The file will be scanned from the begining again but this is not the case with emagent.log file monitoring its working fine as expected and explained in the script file.
According to my observation this is becasue of the script is rotating my log file for each run i dont know how stop it. I just want to scan my log file I dont want to rotate my log file for each run of the script. Could any one please help me to solve this problem
Thanks
Ashok Chava.

How to Monitor the size of log file (Log4j) During its Generation.

i have made a program using log4j technology. Now i want to know that is there any method that i can restrict the size of the log file. for example if the size of generating log file exceeds the restricted size then it should be placed in backup and a new file should be automatically generate and remaining contents should be written in that file.
is there any method that can monitor the size of file during the generation of that (say) log file.
Waiting for ur Urgent response

I have wrote that code
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="appender" class="org.apache.log4j.FileAppender">
<param name="File" value="c:\\abc.txt"/>
<param name="MaxFileSize" value="100B"/>
<param name="MaxBackupIndex" value="3"/>
<param name="Append" value="false"/>
<layout class="org.apache.log4j.SimpleLayout"></layout>
</appender>
<root>
<priority value ="debug"/>
<appender-ref ref="appender"/>
</root>
</log4j:configuration>
When i run it it gave me error Message
log4j:WARN No such property [maxFileSize] in org.apache.log4j.FileAppender.
log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.FileAppender.

I have set up two users, one for myself and one for children. The computer automatically logs in for the children with no password required. When the children go to spotlight and type in a search criteria all of my files show up. How do I prevent this?

I have set up two users, one for myself and one for children. The computer automatically logs in for the children with no password required. When the children go to spotlight and type in a search criteria all of my files show and open up. How do I prevent this?

Log in to your account, and move all your files to your home folder. No other users should be able to access them there and they won't show up with a Spotlight search.
Make sure your kids' account(s) do not have admin privileges.

Rule created to monitor a single line entries in a text.log file does not work

Hi All,
I have this strange issue. I created a script which generates .log file and i have configured a rule to monitor it. Whenever the .log is altered the alert does not come at all in SCOM 2012 R2.
I want this alert to be raised when one specific line in the center is altered from LISTENING to NOT LISTENING.
I have configured it. It triggered a alert for the first time and again it did not trigger at all.
I created this rule and disabled it and overrided the value to true only to the MS acting as the watcher for this log file.
The log file generates in the local drive of the MS itself.
Changed the log watcher to a different server and also mentioned the application data source to a network location when the watcher was changed so it can pull the log accordingly.
The log is generated in the MS itself. Tried using both local location where the log is located as well as converted the same to a network location still didn't help.
C:\Port_checker is the directory where the .log file is located also there is no other log file present only 1.
I also changed the parameters such as "Contains, Wildcard matches etc but nothing worked.
Screenshots:
2.
The SCOM Action account has Full permissions on all servers over the entire forest itself.
Target used to create this rule is "Windows server operating system"
Can any one help me please.
Gautam.75801

Since you have a script that updates a file line from "LISTENING" to "NOT LISTENING"
you might want to try and configure a Two State Script Unit Monitor rather then a rule. So your script just need to check say every 5 minutes the content of the log file and generate an alert when it matches "Not Listening" and clear when
it changes to "listening".
http://www.systemcentercentral.com/wp-content/uploads/2009/04/HOW-TO_2-state_ScriptMonitor.pdf
Cheers,
Martin
Blog:
http://sustaslog.wordpress.com
LinkedIn:
Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

How to Monitor oracle Application in unix through log files?

Hello Every Body,
I Would Like To Know The Log Files locations to monitor the start-up and stop error entries in the logs , like web server where its log files located,...forms server ...etc
is it in $COMMON_TOP ..if yes where..?
one last Q: what should i do to be a pro-active apps dba?
using unix -AIX commands
Thanks and Regards,

Startup/Shutdown logs
$COMMON_TOP/admin/log/<SID_hostname>
Apache logs
$APACHE_TOP/Apache/Jserv/logs
$APACHE_TOP/Apache/Jserv/logs/jvm
Concurrent Manager Log Files
$APPLCSF/$APPLLOG
$APPLCSF/$APPLOUT
Database Log Files
$ORACLE_HOME/admin/<SID_hostname>/bdump
You can also use Oracle Applcations Manager (OAM), with OAM you can view information on general system activity including the statuses of the database, concurrent managers and other services, concurrent requests, and Oracle Workflow processes. You can also start or stop services, and submit concurrent requests. You can also view configuration information, such as initialization parameters and profile options .. etc.

Can I monitor a Log file using EMGC 10.2.0.2?

Hi,
I am thinking of monitoring my web application log file using EMGC by creating a generic service is that possible. Now we are using some shell scripts to do that but its bit difficult to maintain all these shell scripts on each of the host. Is there any in built in mechanism that enable me to monitor the log file and when a perticular pattern match I would like to send a email notification to concerned people say application admins if there is no out of box option for this do we have plugins to do this. please let me know the possibility of implementing this using EMGC or extencibility plug-ins.
Ashok Chava.

Hi,
I have used "Log File Pattern Matched Line Count" of host to monitor the log files and below is the pattern I have defined for the log file. But i could not find any alerts even there are so many such exceptions in the log file matching the patteren given in EMGC.
/u01/app/oracle/product/IAS904/sysman/log/emias.log;%oracle.sysman.emSDK.util.jdk.EMException;%
I have even add the log file in agent_home/sysman/config/lfm_ifiles file as given in the documentation but I could not see any alerts as expected am I doint anything wrong in my setup.
Please let me know.
Thanks,
Ashok Chava

How often do log monitors check log files?

Hello all, just wondering if there was a set interval that SCOM checks log files when you set up a simple log monitor. Is there an interval somewhere I'm not seeing? Or how does that work?

Please see: https://social.technet.microsoft.com/Forums/systemcenter/en-US/ee74e8b6-623a-41ee-abb4-54a98d017000/scom-2012-polling-interval-faster-for-monitoring-network
But not every param of monitor can be changed and it can be hardcoded in MP...and only way to change this param is to recreate MP ..
usually Param Interval (sec) or Frequency is
used to modify that property (polling interval) in overrides."
Cheers,
Martin
Blog:
http://sustaslog.wordpress.com
LinkedIn:
Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

SCOM monitor log file

Ｗe'd like to user SCOM 2012 R2 to mointor log file and send alert.
We follow this blog:
http://jimmoldenhauer.blogspot.tw/2013/03/scom-2012-how-to-generate-alert-from.html
However, when we completed the monitoring setting, OM did not show alert on console.
Our OM agent is healthy, so is there anything I miss to send log file alert?

Hi,
I suggest you refer to the TechNet library docuement to create the rule again for a test. Thanks for your time.
http://technet.microsoft.com/en-us/library/hh457567.aspx
Niki Han
TechNet Community Support

Is there a way to automatically collect the log files from a AirPort or TimeCapsule base station?

Hi there,
the headline says basically all: Is there a way to collect the log files from a Time Capsule 7 AirPort from time to time? They are overwritten quite soon but I want a complete log of all activities to my access point.
The AirPort Utility says something about SNMP. Is this the way to go? Some kind of demon on my Mac that retrieves and saves the logs like every two days?
I want to copy the access logs, I am not interested in Time Machine backup logs.
Thanks.

There is a way to do this via Syslog. On the Logging & Statistics panel (within the AirPort Utility), you can point the AirPort's system logs to a Syslog "server."
This would require a "dedicated" network client to receive the logs.
Unfortunately, setting up a Syslog server is a bit intensive initially, but is simple to operate and maintain.
Please check out this Apple Support Communities thread: Directing Syslog message to a file
ref: Enable an Apple Mac OS machine as a syslog server

Automatic log files monitoring

Similar Messages

Maybe you are looking for