Linux logfile monitoring does not work after using "privileged datasource"

Question

Hello!
I have noticed a strange behaviour on one of my Linux Agents (lets call it server_a) regarding logfile monitoring with the "Microsoft.Unix.SCXLog.Datasource" and the "Microsoft.Unix.SCXLog.Privileged.Datasource".
After successfully testing monitoring of /var/log/messages on server_a with the "Privileged Datasource". This test has been on server_a and the MP containing this rule has been delete from the management gorup before the following tests.
I wanted to test another logfile (lets call it logfile_a) using the normal datasource "Microsoft.Unix.SCXLog.Datasource" on server_a. So I created the usual logfile rule (rule_a) in XML (which I have done countless times before) for monitoring
logfile_a. Logfile_a has been created by the "Linux Action Account User" with reading rights for everyone. After importing the Management Pack with the monitoring  for logfile_a I had the following warning alert in the scom console managing
server_a:
  Fehler beim Überprüfen der Protokolldatei "/home/ActionAccountUser/logfile_a" auf dem Host "server_a" als Benutzer "<SCXUser><UserId>ActionAccountUser</UserId><Elev></Elev></SCXUser>";
An internal error occurred.  (the userid has been changed to keep the anonimity of our action account).
To make sure I did not make any mistakes in the XML i have created a new logfile rule (rule_b) monitoring "logfile_b" on "server_a" using the "Logfile Template" under the authoring tab. logfile_b was also created by the "Linux
Action Account User" and had reading rights for everyone. Unfortunately this logfile rule created the same error:
  Fehler beim Überprüfen der Protokolldatei "/home/ActionAccountUser/logfile_b" auf dem Host "server_a" als Benutzer "<SCXUser><UserId>ActionAccountUser</UserId><Elev></Elev></SCXUser>";
An internal error occurred.  (the userid has been changed to keep the anonimity of our action account).
Although both rules (rule_a and rule_b) used the "Microsoft.Unix.SCXLog.Datasource" which uses the Action Account for monitoring logfiles, the above error looks to me as SCOM wants to use the privileged user, which in this case it not necessary
as the Action Account can read logfile_a and logfile_b without any problems.
So after a few unsuccessfull tries to get both rules to raise an alert I tried to use the "Microsoft.Unix.SCXLog.Privileged.Datasource" for rule_a as last resort. Then suddenly after importing the updated Management Pack I finally received the
alert I desperately waited for this whole time.
Finally after lot of text here are my questions:
Could it be that the initial test with the Privileged Log Datasource somehow screwed up the agent on server_a so it could not monitor logfiles with the standard log datasource? Or may anyone of you might have an idea what went wrong here.
Like I said both logfile could be accessed and changed by the normal Action Account without any problems. So privileged right are not needed. I even restarted the scom agent in case something hanged.
I hope I could make the problm clear to you. If not, don´t hesitate to ask any questions.
Thank you and kind regards,
Patrick

Answer 1

Hello!
After all that text, I fogrot the most essential information..
We are currently using OpsMgr 2012 SP1 UR4, the monitored server (server_a) has agent version 1.4.1-292 installed.
Thanks for the explanation of how the logprovider works. I tried to execute the logfilereader just to see if there are any errors and everything looks fine to me:
ActionAccount @server_a:/opt/microsoft/scx/bin> ./scxlogfilereader -v
Version: 1.4.1-292 (Labeled_Build - 20130923L)
Here are the latest entry in the scx.log file:
* Microsoft System Center Cross Platform Extensions (SCX)
* Build number: 1.4.1-292 Labeled_Build
* Process id: 23186
* Process started: 2014-03-31T08:29:09,136Z
* Log format: <date> <severity>     [<code module>:<process id>:<thread id>] <message>
2014-03-31T08:29:09,138Z Warning    [scx.logfilereader.ReadLogFile:23186:140522274359072] scxlogfilereader - Unexpected exception: Could not find persisted data: Failed to access filesystem item /var/opt/microsoft/scx/lib/state/ActionAccount/LogFileProvider__ActionAccount_shome_sActionAccount_slogfilewithoutsudo.txtEDST02
2014-03-31T08:29:09,138Z Warning    [scx.core.providers.logfileprovider:5209:140101980321536] LogFileProvider InvokeLogFileReader - Exception: Internal Error: Unexpected return code running '/opt/microsoft/scx/bin/scxlogfilereader -p': 4
2014-03-31T08:29:09,138Z Warning    [scx.core.providers.logfileprovider:5209:140101980321536] BaseProvider::InvokeMethod() - Internal Error: Unexpected return code running '/opt/microsoft/scx/bin/scxlogfilereader -p': 4 - [/home/serviceb/ScxCore_URSP1_SUSE_110_x64/source/code/providers/logfile_provider/logfileprovider.cpp:442]
* Microsoft System Center Cross Platform Extensions (SCX)
* Build number: 1.4.1-292 Labeled_Build
* Process id: 23284
* Process started: 2014-03-31T08:30:06,139Z
* Log format: <date> <severity>     [<code module>:<process id>:<thread id>] <message>
2014-03-31T08:30:06,140Z Warning    [scx.logfilereader.ReadLogFile:23284:140016517941024] scxlogfilereader - Unexpected exception: Could not find persisted data: Failed to access filesystem item /var/opt/microsoft/scx/lib/state/ActionAccount/LogFileProvider__ActionAccount_shome_sActionAccount_stest.txtEDST02
2014-03-31T08:30:06,142Z Warning    [scx.core.providers.logfileprovider:5209:140101980321536] LogFileProvider InvokeLogFileReader - Exception: Internal Error: Unexpected return code running '/opt/microsoft/scx/bin/scxlogfilereader -p': 4
2014-03-31T08:30:06,143Z Warning    [scx.core.providers.logfileprovider:5209:140101980321536] BaseProvider::InvokeMethod() - Internal Error: Unexpected return code running '/opt/microsoft/scx/bin/scxlogfilereader -p': 4 - [/home/serviceb/ScxCore_URSP1_SUSE_110_x64/source/code/providers/logfile_provider/logfileprovider.cpp:442]
Strangely I could not acces the "Action Account User" directory under /var/opt/microsoft/scx/log as "ActionAccount" user. Is it ok for the directory to have the following rights:  drwx------ 2 1001 users? Instead of "1001" it should say "ActionAccount",
right?
This could be a bit far fetched, but perhaps the logfile provider can´t access logfiles as the "ActionAccount" on this server because it needs to write in the scx.log file. But as the "ActionAccount" can´t access the file, the logfile provider throws
an error. And as "Privileged Account" the rule works flawlessly, as the logfile provider running in root context can access everything.
Don´t know if that makes sense, but right now it sounds logical to me.