Parse Webi Logs

Has anyone ever attempted to parse a BO webi log file with powershell (or other windows scripting language) to get something in a more friendly output that could be passed on to an internal support team for review?  For example, when I review our log files I see a variety of error messages that might be something I need to pay attention to as a system admin or they might be 'training' type errors that I can hand off to our support team to get with the user.  I want to parse out the 'useful' info and ignore the noise ( such as excess info, 'soft errors', or successes).  Plus, the support team does not have access to our webi logs/webi servers.
I am self-teaching as I go with powershell and since I have other scripts running in powershell, I started with that (there could be a better way).  I would just like to extract the date, server, full error information and username and format it to a list.  There is just no easy way that I can figure out to identify a single log entry line... or to know where the error information 'really' ends.  Maybe I'm overlooking the obvious though - which is very possible as these webi logs make my eyes go buggy!
Has anyone ever tried anything like this and can share some insight? we are on 4.0 sp7 patch 7 if it makes a difference in log format or anything.
thanks in advance,
Missy

Hi Mani,
Thanks for the link to the trace settings - yet another good resource you have provided.  Our ini file is set as follows:
active = true;
importance = xl;
alert = true;
severity = 'E';
//keep = false;
//size = 100 * 1000;
So, mostly defaults.  Not sure the exact difference in context between s, m, l, and xl, but even with these settings noted above, in our webiserver log, for example, I will get a line that looks like this:
|12671c8e-cc1c-5164-2b7f-d36028aea63d|2014 07 30 17:14:58:479|-0400|Error| |>>| | |webiserver_SERVERA.WebIntelligenceProcessingServer| 9984|3984|| |2|0|2|0|BIlaunchpad.WebApp|SERVERA:7616:65.106749:1|Webi SDK.CorbaServerImpl.doProcess()|SERVERA:7616:65.106749:2|webiserver_SERVERA.WebIntelligenceProcessingServer.loadStateMDP|localhost:9984:3984.119276:1|CtC8xxN8nEPIkk05usjbm5s1a0fb|||||||||||AsyncCaller:WebiSession[AS6LS2uphS9GilVoKXDMP9k]: loadState by user 'userA' on document 'Webi Report A'  AsyncCaller.cpp:181:bool __cdecl async::AsyncCaller::startRequest(const char *,int): TraceLog message 21020
So what part of this is an error? To me, this looks like normal processing steps. Another example, would be:
|1404a918-2cad-9184-6a6b-decec93efe1e|2014 07 30 17:58:06:861|-0400|Error|Error|>>|E| |webiserver_SERVERA.WebIntelligenceProcessingServer| 37332|35556|| |12|0|2|0|BIlaunchpad.WebApp|SERVERB:8700:75751.68935664:1|Webi SDK.CorbaServerImpl.doProcess()|SERVERB:8700:75751.68935664:3|webiserver_SERVERA.WebIntelligenceProcessingServer3.loadStateMDP|localhost:37332:35556.58574:1|CoWFD4_SSEn5rjTHJ2XQgkU41bdff6|||||||||||**ERROR:RequestProc:user: userC, doc: "", error stream: [kctRequestProc.cpp;777]  SharedContextImpl.cpp:227:__cdecl SharedContextImpl::~SharedContextImpl(void): TraceLog message 38279
That one, I can see how it is an error, generic or otherwise, it is clearly an error.
What we have been doing with our vb script, is parsing the log files by searching line by line for the word 'user' and then outputting a subset of that line to excel, figuring this gives us errors the user experienced.  What we were surprised to see is the sheer number of errors that the user is supposedly experiencing.  What is also surprising, is that it doesn't appear that the user sees these messages on screen all the time, given that they don't complain and/or we having someone working with the user who verifies no messages on screen appeared, but I can match up timestamps with the log files and their actions performed and see that an error was hit.
Our initial goal with this was to see what users may need more training, but we also have been using it as a quick way to monitor things too.  So for example, if a user calls and complains they got an error, I'll use this first to help identify which server they were on and check out server related 'things' and then I use the glf viewer to view the full log if I need to analyze things further.
Also, here's the link I have for the error message guide:
http://help.sap.com/businessobject/product_guides/boexir4/en/xi4_error_messages_en.pdf
For example, WIS 30600 is not in there.  Is there a different guide?
Thanks,
Missy

Similar Messages

  • Parsing a log file on Weblogic

    Hi!
    I'd like to know how to get started on parsing a log file present in the default directory of Weblogic (ver 6.1 to be precise).
    I thought of using regular expressions, and use java.util.regex , but that is supported from JDK1.5 onwards, whereas WL6.1 supports JDK1.3.
    If u can also provide the code template for the same , that would be nice.
    Thanks in advance,
    Deepthy.

    uncle_alice wrote:
    String regex = "([^\"\\\\]++|\\\\.)++"{code} The trick is to match anything except a quotation mark or a backslash, OR match a backslash followed by anything (because the backslash is usually used to escape other characters as well, including backslashes).Superb! Thanks! I have to admit I've never used the ++ before (only the greedies), but that's the thing I was looking for.
    Just for the completeness, this is the whole thing that's able to parse a log line:
    {code}
    public class LogParser {
    private static final String NOSPACE_PARAM = "([^ ]++)";
    private static final String DATE_PARAM = "([^\\]]++)";
    private static final String ESCAPED_PARAM = "((?:[^\"\\\\]++|\\\\.)++)";
    private static final String PATTERN_STRING = NOSPACE_PARAM
    + " " + NOSPACE_PARAM
    + " " + NOSPACE_PARAM
    + " \\[" + DATE_PARAM + "\\]"
    + " \"" + ESCAPED_PARAM + "\""
    + " " + NOSPACE_PARAM
    + " " + NOSPACE_PARAM
    + " \"" + ESCAPED_PARAM + "\""
    + " \"" + ESCAPED_PARAM + "\""
    + " \"" + ESCAPED_PARAM + "\""
    + " \"" + ESCAPED_PARAM + "\""
    + " \"" + ESCAPED_PARAM + "\""
    + " " + NOSPACE_PARAM
    + " \"" + ESCAPED_PARAM + "\""
    + " \"" + ESCAPED_PARAM + "\""
    + " \"" + ESCAPED_PARAM + "\""
    + " \"" + ESCAPED_PARAM + "\"";
    private static final Pattern PATTERN = Pattern.compile(PATTERN_STRING);
    public static String[] parse(String line) {
    Matcher m = PATTERN.matcher(line);
    if (m.matches()) {
    String[] result = new String[m.groupCount()];
    for (int i = 0; i < m.groupCount();) {
    result[i] = m.group(++i);
    return result;
    return null;
    {code}
    Any idea about the efficiency of this thing?

  • Parsing OCSP logs

    Hello,
    am using the following script to parse OCSP Logs(ADCS), which is resulting fine.
    but am not satisfied with the speed of parsing - any one can help me to make it quick or other alternative way.
    on Error resume Next
    Set objFS = CreateObject("Scripting.FileSystemObject")
    Set WshShell = WScript.CreateObject("WScript.Shell")
    set objFileU = objFS.OpenTextFile("u_ex141113.log",1) 'IIS Log file input
    set objFileRW = objFS.CreateTextFile("Result.txt",2)
    i=-10
    do until objFileU.atendofstream
    strLine = objFileU.readLine
    arrLine = split(strLine," ")
    arrLine1 = split(arrLine(4),"/")
    if arrLine1(1) = "ocsp" Then
    'wscript.echo arrLine1(2) & "/" & arrLine1(3) & "/" & arrLine1(4)
    set objFileT = objFS.CreateTextFile(i&".txt",2,False)
    objFileT.WriteLIne arrLine1(2) & "/" & arrLine1(3) & "/" & arrLine1(4)
    objFileT.close
    strCommand =  "cmd /c certutil " & i &".txt | findstr Serial"
    set objExec = wshShell.exec(strCommand)
    strOut = objexec.stdout.ReadAll
    if instr(strOut,":") Then
    wscript.echo split(replace(replace(strout," ","",1,-1),vbCrLf,""),":")(1) & "," & arrLine(8)
    objFileRW.WriteLine split(replace(replace(strout," ","",1,-1),vbCrLf,""),":")(1) & "," & arrLine(8)
    end if
    objFS.deleteFile i & ".txt"
    end if
    i=i+1
    Loop
    objFileRW.close
    'thanks

    On Sat, 15 Nov 2014 14:33:44 +0000, Jagadeesh_ wrote:
    am using the following script to parse OCSP Logs(ADCS), which is resulting fine.
    I know that you're asking about OCSP here, and it would appear that this is
    the correct forum to post to, however, what you're really asking is for
    help in optimizing a script, and that question is really independent of the
    input file type.
    You should really post your question where the scripting experts hang out:
    https://social.technet.microsoft.com/Forums/scriptcenter/en-US/home?forum=ITCG
    Paul Adare - FIM CM MVP
    "Be pedantic in what you accept,
    and arbitrarily brutal in what you send" -- Malcolm Ray

  • Tracking User Activity with Standard Web Logs and Tools Like WebTrends

    We are running EP 6 SP14 on UNIX and I'm looking to track user activity.  Not just how many people have logged in, but who is accessing what documents in KM.  For example, what documents are the most popular and how much use the system is getting.
    I was hoping that EP 6 (Unix platform) ran off us some standard web engine (like Apache) and we could just turn web logs on, but from the lack of discussion on SDN, I'm not encouraged.
    All I've seen is unofficial link to a Portal Activity Report, which would get us part of the way, but is not full web reporting.
    Portal Activity Report:
    https://www.sdn.sap.com/irj/sdn?id=/library/uuid/0101b690-0201-0010-6584-a02730ad5edd
    Does anyone have any insight on this?

    HI llise
    Did you get any answers? We are in the same quandry as the exisitng tracking reports are not sufficient for our needs as our users are used to using more sophisticated tools like livestat.

  • Defacto Web Log Analyzer?

    Wondering if there's a defacto apache web log analyzer that most of you use for Mac OS X Server?
    Seems to be minimal simple options out there to just install and work. I'm not a Unix admin by any stretch, so something with a GUI or Web admin front end would be perfect.
    Cheers
    Brendan

    There is one one 'defacto' analyzer -- the one built into the traffic graphs of the SA.
    If you are instead asking what folks like to use, that varies greatly. You can run them yourself or use a 'service' such as Google.
    Here's a starter list for you to explore:
    Accrue Insight
    Analog
    AWstats
    FunnelWeb
    Sawmill
    Summary
    Urchin / Google Analytics
    Webalizer
    Wusage
    So explore all of those and report back here -- let us know what works best for you, and why.

  • Web log analyzer

    Does anyone know of a decent free web log analyzer for the Mac. I don't need anything fancy but all the ones I've seen are for Windows. Thanks.
    powerbook G4 12   Mac OS X (10.4)  

    There is one one 'defacto' analyzer -- the one built into the traffic graphs of the SA.
    If you are instead asking what folks like to use, that varies greatly. You can run them yourself or use a 'service' such as Google.
    Here's a starter list for you to explore:
    Accrue Insight
    Analog
    AWstats
    FunnelWeb
    Sawmill
    Summary
    Urchin / Google Analytics
    Webalizer
    Wusage
    So explore all of those and report back here -- let us know what works best for you, and why.

  • Parse robocopy Log File - new value

    Hello,
    I have found a script, that parse the robocopy log file, which looks like this:
       ROBOCOPY     ::     Robust File Copy for Windows                             
      Started : Thu Aug 07 09:30:18 2014
       Source : e:\testfolder\
         Dest : w:\testfolder\
        Files : *.*
      Options : *.* /V /NDL /S /E /COPYALL /NP /IS /R:1 /W:5
         Same          14.6 g e:\testfolder\bigfile - Copy (5).out
         Same          14.6 g e:\testfolder\bigfile - Copy.out
         Same          14.6 g e:\testfolder\bigfile.out
                   Total    Copied   Skipped  Mismatch    FAILED    Extras
        Dirs :         1         0         1         0        
    0         0
       Files :         3         3         0         0        
    0         0
       Bytes :  43.969 g  43.969 g         0         0         0         0
       Times :   0:05:44   0:05:43                       0:00:00   0:00:00
       Speed :           137258891 Bytes/sec.
       Speed :            7854.016 MegaBytes/min.
       Ended : Thu Aug 07 09:36:02 2014
    Most values at output file are included, but the two speed paramter not.
    How can I get this two speed paramters at output file?
    Here is the script:
    param(
    [parameter(Position=0,Mandatory=$true,ValueFromPipeline=$false,HelpMessage='Source Path with no trailing slash')][string]$SourcePath,
    [switch]$fp
    write-host "Robocopy log parser. $(if($fp){"Parsing file entries"} else {"Parsing summaries only, use -fp to parse file entries"})"
    #Arguments
    # -fp File parse. Counts status flags and oldest file Slower on big files.
    $ElapsedTime = [System.Diagnostics.Stopwatch]::StartNew()
    $refreshrate=1 # progress counter refreshes this often when parsing files (in seconds)
    # These summary fields always appear in this order in a robocopy log
    $HeaderParams = @{
    "04|Started" = "date";
    "01|Source" = "string";
    "02|Dest" = "string";
    "03|Options" = "string";
    "07|Dirs" = "counts";
    "08|Files" = "counts";
    "09|Bytes" = "counts";
    "10|Times" = "counts";
    "05|Ended" = "date";
    #"06|Duration" = "string"
    $ProcessCounts = @{
    "Processed" = 0;
    "Error" = 0;
    "Incomplete" = 0
    $tab=[char]9
    $files=get-childitem $SourcePath
    $writer=new-object System.IO.StreamWriter("$(get-location)\robocopy-$(get-date -format "dd-MM-yyyy_HH-mm-ss").csv")
    function Get-Tail([object]$reader, [int]$count = 10) {
    $lineCount = 0
    [long]$pos = $reader.BaseStream.Length - 1
    while($pos -gt 0)
    $reader.BaseStream.position=$pos
    # 0x0D (#13) = CR
    # 0x0A (#10) = LF
    if ($reader.BaseStream.ReadByte() -eq 10)
    $lineCount++
    if ($lineCount -ge $count) { break }
    $pos--
    # tests for file shorter than requested tail
    if ($lineCount -lt $count -or $pos -ge $reader.BaseStream.Length - 1) {
    $reader.BaseStream.Position=0
    } else {
    # $reader.BaseStream.Position = $pos+1
    $lines=@()
    while(!$reader.EndOfStream) {
    $lines += $reader.ReadLine()
    return $lines
    function Get-Top([object]$reader, [int]$count = 10)
    $lines=@()
    $lineCount = 0
    $reader.BaseStream.Position=0
    while(($linecount -lt $count) -and !$reader.EndOfStream) {
    $lineCount++
    $lines += $reader.ReadLine()
    return $lines
    function RemoveKey ( $name ) {
    if ( $name -match "|") {
    return $name.split("|")[1]
    } else {
    return ( $name )
    function GetValue ( $line, $variable ) {
    if ($line -like "*$variable*" -and $line -like "* : *" ) {
    $result = $line.substring( $line.IndexOf(":")+1 )
    return $result
    } else {
    return $null
    function UnBodgeDate ( $dt ) {
    # Fixes RoboCopy botched date-times in format Sat Feb 16 00:16:49 2013
    if ( $dt -match ".{3} .{3} \d{2} \d{2}:\d{2}:\d{2} \d{4}" ) {
    $dt=$dt.split(" ")
    $dt=$dt[2],$dt[1],$dt[4],$dt[3]
    $dt -join " "
    if ( $dt -as [DateTime] ) {
    return $dt.ToStr("dd/MM/yyyy hh:mm:ss")
    } else {
    return $null
    function UnpackParams ($params ) {
    # Unpacks file count bloc in the format
    # Dirs : 1827 0 1827 0 0 0
    # Files : 9791 0 9791 0 0 0
    # Bytes : 165.24 m 0 165.24 m 0 0 0
    # Times : 1:11:23 0:00:00 0:00:00 1:11:23
    # Parameter name already removed
    if ( $params.length -ge 58 ) {
    $params = $params.ToCharArray()
    $result=(0..5)
    for ( $i = 0; $i -le 5; $i++ ) {
    $result[$i]=$($params[$($i*10 + 1) .. $($i*10 + 9)] -join "").trim()
    $result=$result -join ","
    } else {
    $result = ",,,,,"
    return $result
    $sourcecount = 0
    $targetcount = 1
    # Write the header line
    $writer.Write("File")
    foreach ( $HeaderParam in $HeaderParams.GetEnumerator() | Sort-Object Name ) {
    if ( $HeaderParam.value -eq "counts" ) {
    $tmp="~ Total,~ Copied,~ Skipped,~ Mismatch,~ Failed,~ Extras"
    $tmp=$tmp.replace("~","$(removekey $headerparam.name)")
    $writer.write(",$($tmp)")
    } else {
    $writer.write(",$(removekey $HeaderParam.name)")
    if($fp){
    $writer.write(",Scanned,Newest,Summary")
    $writer.WriteLine()
    $filecount=0
    # Enumerate the files
    foreach ($file in $files) {
    $filecount++
    write-host "$filecount/$($files.count) $($file.name) ($($file.length) bytes)"
    $results=@{}
    $Stream = $file.Open([System.IO.FileMode]::Open,
    [System.IO.FileAccess]::Read,
    [System.IO.FileShare]::ReadWrite)
    $reader = New-Object System.IO.StreamReader($Stream)
    #$filestream=new-object -typename System.IO.StreamReader -argumentlist $file, $true, [System.IO.FileAccess]::Read
    $HeaderFooter = Get-Top $reader 16
    if ( $HeaderFooter -match "ROBOCOPY :: Robust File Copy for Windows" ) {
    if ( $HeaderFooter -match "Files : " ) {
    $HeaderFooter = $HeaderFooter -notmatch "Files : "
    [long]$ReaderEndHeader=$reader.BaseStream.position
    $Footer = Get-Tail $reader 16
    $ErrorFooter = $Footer -match "ERROR \d \(0x000000\d\d\) Accessing Source Directory"
    if ($ErrorFooter) {
    $ProcessCounts["Error"]++
    write-host -foregroundcolor red "`t $ErrorFooter"
    } elseif ( $footer -match "---------------" ) {
    $ProcessCounts["Processed"]++
    $i=$Footer.count
    while ( !($Footer[$i] -like "*----------------------*") -or $i -lt 1 ) { $i-- }
    $Footer=$Footer[$i..$Footer.Count]
    $HeaderFooter+=$Footer
    } else {
    $ProcessCounts["Incomplete"]++
    write-host -foregroundcolor yellow "`t Log file $file is missing the footer and may be incomplete"
    foreach ( $HeaderParam in $headerparams.GetEnumerator() | Sort-Object Name ) {
    $name = "$(removekey $HeaderParam.Name)"
    $tmp = GetValue $($HeaderFooter -match "$name : ") $name
    if ( $tmp -ne "" -and $tmp -ne $null ) {
    switch ( $HeaderParam.value ) {
    "date" { $results[$name]=UnBodgeDate $tmp.trim() }
    "counts" { $results[$name]=UnpackParams $tmp }
    "string" { $results[$name] = """$($tmp.trim())""" }
    default { $results[$name] = $tmp.trim() }
    if ( $fp ) {
    write-host "Parsing $($reader.BaseStream.Length) bytes"
    # Now go through the file line by line
    $reader.BaseStream.Position=0
    $filesdone = $false
    $linenumber=0
    $FileResults=@{}
    $newest=[datetime]"1/1/1900"
    $linecount++
    $firsttick=$elapsedtime.elapsed.TotalSeconds
    $tick=$firsttick+$refreshrate
    $LastLineLength=1
    try {
    do {
    $line = $reader.ReadLine()
    $linenumber++
    if (($line -eq "-------------------------------------------------------------------------------" -and $linenumber -gt 16) ) {
    # line is end of job
    $filesdone=$true
    } elseif ($linenumber -gt 16 -and $line -gt "" ) {
    $buckets=$line.split($tab)
    # this test will pass if the line is a file, fail if a directory
    if ( $buckets.count -gt 3 ) {
    $status=$buckets[1].trim()
    $FileResults["$status"]++
    $SizeDateTime=$buckets[3].trim()
    if ($sizedatetime.length -gt 19 ) {
    $DateTime = $sizedatetime.substring($sizedatetime.length -19)
    if ( $DateTime -as [DateTime] ){
    $DateTimeValue=[datetime]$DateTime
    if ( $DateTimeValue -gt $newest ) { $newest = $DateTimeValue }
    if ( $elapsedtime.elapsed.TotalSeconds -gt $tick ) {
    $line=$line.Trim()
    if ( $line.Length -gt 48 ) {
    $line="[...]"+$line.substring($line.Length-48)
    $line="$([char]13)Parsing > $($linenumber) ($(($reader.BaseStream.Position/$reader.BaseStream.length).tostring("P1"))) - $line"
    write-host $line.PadRight($LastLineLength) -NoNewLine
    $LastLineLength = $line.length
    $tick=$tick+$refreshrate
    } until ($filesdone -or $reader.endofstream)
    finally {
    $reader.Close()
    $line=$($([string][char]13)).padright($lastlinelength)+$([char]13)
    write-host $line -NoNewLine
    $writer.Write("`"$file`"")
    foreach ( $HeaderParam in $HeaderParams.GetEnumerator() | Sort-Object Name ) {
    $name = "$(removekey $HeaderParam.Name)"
    if ( $results[$name] ) {
    $writer.Write(",$($results[$name])")
    } else {
    if ( $ErrorFooter ) {
    #placeholder
    } elseif ( $HeaderParam.Value -eq "counts" ) {
    $writer.Write(",,,,,,")
    } else {
    $writer.Write(",")
    if ( $ErrorFooter ) {
    $tmp = $($ErrorFooter -join "").substring(20)
    $tmp=$tmp.substring(0,$tmp.indexof(")")+1)+","+$tmp
    $writer.write(",,$tmp")
    } elseif ( $fp ) {
    $writer.write(",$LineCount,$($newest.ToString('dd/MM/yyyy hh:mm:ss'))")
    foreach ( $FileResult in $FileResults.GetEnumerator() ) {
    $writer.write(",$($FileResult.Name): $($FileResult.Value);")
    $writer.WriteLine()
    } else {
    write-host -foregroundcolor darkgray "$($file.name) is not recognised as a RoboCopy log file"
    write-host "$filecount files scanned in $($elapsedtime.elapsed.tostring()), $($ProcessCounts["Processed"]) complete, $($ProcessCounts["Error"]) have errors, $($ProcessCounts["Incomplete"]) incomplete"
    write-host "Results written to $($writer.basestream.name)"
    $writer.close()
    I hope somebody can help me,
    Horst
    Thanks Horst MOSS 2007 Farm; MOSS 2010 Farm; TFS 2010; TFS 2013; IIS 7.5

    Hi Horst,
    To convert mutiple robocopy log files to a .csv file with "speed" option, the script below may be helpful for you, I tested with a single robocopy log file, and the .csv file will output to "D:\":
    $SourcePath="e:\1\1.txt" #robocopy log file
    write-host "Robocopy log parser. $(if($fp){"Parsing file entries"} else {"Parsing summaries only, use -fp to parse file entries"})"
    #Arguments
    # -fp File parse. Counts status flags and oldest file Slower on big files.
    $ElapsedTime = [System.Diagnostics.Stopwatch]::StartNew()
    $refreshrate=1 # progress counter refreshes this often when parsing files (in seconds)
    # These summary fields always appear in this order in a robocopy log
    $HeaderParams = @{
     "04|Started" = "date"; 
     "01|Source" = "string";
     "02|Dest" = "string";
     "03|Options" = "string";
     "09|Dirs" = "counts";
     "10|Files" = "counts";
     "11|Bytes" = "counts";
     "12|Times" = "counts";
     "05|Ended" = "date";
     "07|Speed" = "default";
     "08|Speednew" = "default"
    $ProcessCounts = @{
     "Processed" = 0;
     "Error" = 0;
     "Incomplete" = 0
    $tab=[char]9
    $files=get-childitem $SourcePath
    $writer=new-object System.IO.StreamWriter("D:\robocopy-$(get-date -format "dd-MM-yyyy_HH-mm-ss").csv")
    function Get-Tail([object]$reader, [int]$count = 10) {
     $lineCount = 0
     [long]$pos = $reader.BaseStream.Length - 1
     while($pos -gt 0)
      $reader.BaseStream.position=$pos
      # 0x0D (#13) = CR
      # 0x0A (#10) = LF
      if ($reader.BaseStream.ReadByte() -eq 10)
       $lineCount++
       if ($lineCount -ge $count) { break }
      $pos--
     # tests for file shorter than requested tail
     if ($lineCount -lt $count -or $pos -ge $reader.BaseStream.Length - 1) {
      $reader.BaseStream.Position=0
     } else {
      # $reader.BaseStream.Position = $pos+1
     $lines=@()
     while(!$reader.EndOfStream) {
      $lines += $reader.ReadLine()
     return $lines
    function Get-Top([object]$reader, [int]$count = 10)
     $lines=@()
     $lineCount = 0
     $reader.BaseStream.Position=0
     while(($linecount -lt $count) -and !$reader.EndOfStream) {
      $lineCount++
      $lines += $reader.ReadLine()  
     return $lines
    function RemoveKey ( $name ) {
     if ( $name -match "|") {
      return $name.split("|")[1]
     } else {
      return ( $name )
    function GetValue ( $line, $variable ) {
     if ($line -like "*$variable*" -and $line -like "* : *" ) {
      $result = $line.substring( $line.IndexOf(":")+1 )
      return $result
     } else {
      return $null
    }function UnBodgeDate ( $dt ) {
     # Fixes RoboCopy botched date-times in format Sat Feb 16 00:16:49 2013
     if ( $dt -match ".{3} .{3} \d{2} \d{2}:\d{2}:\d{2} \d{4}" ) {
      $dt=$dt.split(" ")
      $dt=$dt[2],$dt[1],$dt[4],$dt[3]
      $dt -join " "
     if ( $dt -as [DateTime] ) {
      return $dt.ToStr("dd/MM/yyyy hh:mm:ss")
     } else {
      return $null
    function UnpackParams ($params ) {
     # Unpacks file count bloc in the format
     # Dirs :      1827         0      1827         0         0         0
     # Files :      9791         0      9791         0         0         0
     # Bytes :  165.24 m         0  165.24 m         0         0         0
     # Times :   1:11:23   0:00:00                       0:00:00   1:11:23
     # Parameter name already removed
     if ( $params.length -ge 58 ) {
      $params = $params.ToCharArray()
      $result=(0..5)
      for ( $i = 0; $i -le 5; $i++ ) {
       $result[$i]=$($params[$($i*10 + 1) .. $($i*10 + 9)] -join "").trim()
      $result=$result -join ","
     } else {
      $result = ",,,,,"
     return $result
    $sourcecount = 0
    $targetcount = 1
    # Write the header line
    $writer.Write("File")
    foreach ( $HeaderParam in $HeaderParams.GetEnumerator() | Sort-Object Name ) {
     if ( $HeaderParam.value -eq "counts" ) {
      $tmp="~ Total,~ Copied,~ Skipped,~ Mismatch,~ Failed,~ Extras"
      $tmp=$tmp.replace("~","$(removekey $headerparam.name)")
      $writer.write(",$($tmp)")
     } else {
      $writer.write(",$(removekey $HeaderParam.name)")
    if($fp){
     $writer.write(",Scanned,Newest,Summary")
    $writer.WriteLine()
    $filecount=0
    # Enumerate the files
    foreach ($file in $files) { 
     $filecount++
        write-host "$filecount/$($files.count) $($file.name) ($($file.length) bytes)"
     $results=@{}
    $Stream = $file.Open([System.IO.FileMode]::Open,
                       [System.IO.FileAccess]::Read,
                        [System.IO.FileShare]::ReadWrite)
     $reader = New-Object System.IO.StreamReader($Stream)
     #$filestream=new-object -typename System.IO.StreamReader -argumentlist $file, $true, [System.IO.FileAccess]::Read
     $HeaderFooter = Get-Top $reader 16
     if ( $HeaderFooter -match "ROBOCOPY     ::     Robust File Copy for Windows" ) {
      if ( $HeaderFooter -match "Files : " ) {
       $HeaderFooter = $HeaderFooter -notmatch "Files : "
      [long]$ReaderEndHeader=$reader.BaseStream.position
      $Footer = Get-Tail $reader 16
      $ErrorFooter = $Footer -match "ERROR \d \(0x000000\d\d\) Accessing Source Directory"
      if ($ErrorFooter) {
       $ProcessCounts["Error"]++
       write-host -foregroundcolor red "`t $ErrorFooter"
      } elseif ( $footer -match "---------------" ) {
       $ProcessCounts["Processed"]++
       $i=$Footer.count
       while ( !($Footer[$i] -like "*----------------------*") -or $i -lt 1 ) { $i-- }
       $Footer=$Footer[$i..$Footer.Count]
       $HeaderFooter+=$Footer
      } else {
       $ProcessCounts["Incomplete"]++
       write-host -foregroundcolor yellow "`t Log file $file is missing the footer and may be incomplete"
      foreach ( $HeaderParam in $headerparams.GetEnumerator() | Sort-Object Name ) {
       $name = "$(removekey $HeaderParam.Name)"
                            if ($name -eq "speed"){ #handle two speed
                            ($HeaderFooter -match "$name : ")|foreach{
                             $tmp=GetValue $_ "speed"
                             $results[$name] = $tmp.trim()
                             $name+="new"}
                            elseif ($name -eq "speednew"){} #handle two speed
                            else{
       $tmp = GetValue $($HeaderFooter -match "$name : ") $name
       if ( $tmp -ne "" -and $tmp -ne $null ) {
        switch ( $HeaderParam.value ) {
         "date" { $results[$name]=UnBodgeDate $tmp.trim() }
         "counts" { $results[$name]=UnpackParams $tmp }
         "string" { $results[$name] = """$($tmp.trim())""" }  
         default { $results[$name] = $tmp.trim() }  
      if ( $fp ) {
       write-host "Parsing $($reader.BaseStream.Length) bytes"
       # Now go through the file line by line
       $reader.BaseStream.Position=0
       $filesdone = $false
       $linenumber=0
       $FileResults=@{}
       $newest=[datetime]"1/1/1900"
       $linecount++
       $firsttick=$elapsedtime.elapsed.TotalSeconds
       $tick=$firsttick+$refreshrate
       $LastLineLength=1
       try {
        do {
         $line = $reader.ReadLine()
         $linenumber++
         if (($line -eq "-------------------------------------------------------------------------------" -and $linenumber -gt 16)  ) {
          # line is end of job
          $filesdone=$true
         } elseif ($linenumber -gt 16 -and $line -gt "" ) {
          $buckets=$line.split($tab)
          # this test will pass if the line is a file, fail if a directory
          if ( $buckets.count -gt 3 ) {
           $status=$buckets[1].trim()
           $FileResults["$status"]++
           $SizeDateTime=$buckets[3].trim()
           if ($sizedatetime.length -gt 19 ) {
            $DateTime = $sizedatetime.substring($sizedatetime.length -19)
            if ( $DateTime -as [DateTime] ){
             $DateTimeValue=[datetime]$DateTime
             if ( $DateTimeValue -gt $newest ) { $newest = $DateTimeValue }
         if ( $elapsedtime.elapsed.TotalSeconds -gt $tick ) {
          $line=$line.Trim()
          if ( $line.Length -gt 48 ) {
           $line="[...]"+$line.substring($line.Length-48)
          $line="$([char]13)Parsing > $($linenumber) ($(($reader.BaseStream.Position/$reader.BaseStream.length).tostring("P1"))) - $line"
          write-host $line.PadRight($LastLineLength) -NoNewLine
          $LastLineLength = $line.length
          $tick=$tick+$refreshrate      
        } until ($filesdone -or $reader.endofstream)
       finally {
        $reader.Close()
       $line=$($([string][char]13)).padright($lastlinelength)+$([char]13)
       write-host $line -NoNewLine
      $writer.Write("`"$file`"")
      foreach ( $HeaderParam in $HeaderParams.GetEnumerator() | Sort-Object Name ) {
       $name = "$(removekey $HeaderParam.Name)"
       if ( $results[$name] ) {
        $writer.Write(",$($results[$name])")
       } else {
        if ( $ErrorFooter ) {
         #placeholder
        } elseif ( $HeaderParam.Value -eq "counts" ) {
         $writer.Write(",,,,,,")
        } else {
         $writer.Write(",")
      if ( $ErrorFooter ) {
       $tmp = $($ErrorFooter -join "").substring(20)
       $tmp=$tmp.substring(0,$tmp.indexof(")")+1)+","+$tmp
       $writer.write(",,$tmp")
      } elseif ( $fp ) {
       $writer.write(",$LineCount,$($newest.ToString('dd/MM/yyyy hh:mm:ss'))")   
       foreach ( $FileResult in $FileResults.GetEnumerator() ) {
        $writer.write(",$($FileResult.Name): $($FileResult.Value);")
      $writer.WriteLine()
     } else {
      write-host -foregroundcolor darkgray "$($file.name) is not recognised as a RoboCopy log file"
    write-host "$filecount files scanned in $($elapsedtime.elapsed.tostring()), $($ProcessCounts["Processed"]) complete, $($ProcessCounts["Error"]) have errors, $($ProcessCounts["Incomplete"]) incomplete"
    write-host  "Results written to $($writer.basestream.name)"
    $writer.close()
    If you have any other questions, please feel free to let me know.
    If you have any feedback on our support,
    please click here.
    Best Regards,
    Anna Wang
    TechNet Community Support

  • **ERROR: Messages in webi log files

    Hi,
    My log files seem to be littered with messages like:
    ***ERROR:C3_cdbDSLServer:(A) cdbDSLServer_LoadUniverseMetadata : CreateChildItem (false)....
    or
    **ERROR:C3_Dictionary:cdbDictionaryFolder::RegisterFolder failed for: "WE Calendar"....
    or
    **ERROR:C3_tbDimensionalSLDS:BOException: ...
    or
    **ERROR:IECore_kc3CoreEngineImpl:GetUniverseLOV failed.
    etc.
    These show up in the Webi server logs.  We don't see these on the screen and I am having a hard time tracking any information about them down.  Is it a *real* error? Should I be concerned??
    thanks,
    -missy

    SAP Support said not to worry about the messages marked as ***ERROR as they 'soft errors'.
    For the example of :
    ERROR:C3_cdbDSLServer:(A) cdbDSLServer_LoadUniverseMetadata : CreateChildItem (false)...
    This message was due to a report loading the objects and taking longer than the system is designed to wait, so it logs a message and continues on to completion.  I paraphrased that a bit, but that was the primary description.
    In contrast, if it was a hard error, the report would fail and a message of some sort would be displayed on screen.
    If reports start failing or messages are displayed on screen, then I should be concerned and should open a new ticket.
    So pretty much what you said Prabhith.
    I can't say I get a warm and fuzzy feeling about this response, but nothing is failing on our end, so at the same time I am not going to push the issue.  His response was prompt and he had seen it before, so I am less worreid, but I will still continue to keep an eye on it as usual.
    thanks,
    missy

  • Parsing sendmail log file in a Java application

    Are there any good parsing libraries for sendmail log files?
    I already found these libraries but I'm not sure if they do what I need:
    http://www.opennms.org/documentation/java-apidocs-stable/org/opennms/netmgt/syslogd/package-summary.html
    http://code.google.com/p/jsyslogd/

    >
    I've written a simple text editor, and this editor saves files with a particular extension. It can both open and save files. I've put the text editor program in a JAR. What I'd like to do, if possible, is associate the file extension with the text editor program. That is, I'd like to, when I click on a file with the extension, have the text editor come up with the file opened in it.
    Can anyone give me ideas on how to do this, please? >If the editor is launched using webstart, the launch file can suggest a file association.
    Note that an application that accesses the local file system needs to be digitally signed before it can break out of the applet like 'sandbox' in which it runs, unless it uses the JNLP API to access the files. The JNLP API is available to any app. launched using webstart.
    There is an example of both claiming a file extension, and accessing files using the JNLP API, in this [File Service Demo|http://pscode.org/jws/api.html#fs]. The complete source and a build file can be downloaded from the filetest.zip just near the launch buttons.
    I suggest you try the sandboxed version first - if you think that will suit your user, go with that.
    As an aside, for best chance of a solution, I recommend folks add [Duke stars|http://wikis.sun.com/display/SunForums/Duke+Stars+Program+Overview] to match the importance of the task.

  • Curl command to parse web sites

    Hi there,
    I am trying to parse price details on the www.travelbag.co.uk web site. Basically I use a preformatted URL which contains date ranges and hotel information and put that into the Curl command. The idea then is via Applescript to add 1 to the day and have that running in a loop so I can determine which times of the year are the cheapest for a particular hotel etc. I have a working set up for the KenwoodTravel.co.uk web site but the problem with travelbag is that it first returns a holding page (like a please wait whilst we get the details). The Curl returns this http response and finishes. Doing it via a web browser works of course but I was wondering if anyone knew any tricks with the Curl command to wait for the full response?
    Thanks,
    Mike...
    PS hope this all makes sense.

    It is probably working through AJAX, much like the forums here. Your best bet is to debug the site in Safari's developer mode to try to find the AJAX request. You can then feed that into Curl.

  • Parse Security Logs for User Account logon Computer Name

    Greetings,
    I was recently tasked with creating a list of user accounts and the computer in which they logged onto.  Unfortunately, we do not have time to use the logon script method.   I believe we can achieve this goal using software similar to LANSweeper
    however not all computers will be turned on at a given time and I believe this application gathers it's information from the client PC.  One possible solution I see is parsing the data from our domain controllers Security Logs / Successful Logons however
    this is proving to be a challenge. Any suggestions?  
    Thanks,
    Chris

    Hi Chris,
    I was recently tasked with creating a list of user accounts and the computer in which they logged onto.
    I believe we can achieve this goal using software.
    There is no built-in tool to complete this task.
    However, we can configure event log trigger to send email when specific logon events are generated.
    Here are some related articles below for you:
    Getting event log contents by email on an event log trigger
    http://blogs.technet.com/b/jhoward/archive/2010/06/16/getting-event-log-contents-by-email-on-an-event-log-trigger.aspx
    Send an email when an event is logged
    http://blogs.iis.net/rickbarber/archive/2012/10/26/send-an-email-when-an-event-is-logged.aspx
    Best Regards,
    Amy
    Please remember to mark the replies as answers if they help and un-mark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact [email protected]

  • Regular web log (access.log) rotation?

    I am using Weblogic as a web server and I have enabled http logging to
    the access.log file. What is the BEA recommended method for copying and
    purging the access.log file on a regular (daily) basis?
    - Kevin

    Refer to README.TXT of Service Pack 5 for details.
    Cheers - Wei
    Kevin McEntee <[email protected]> wrote in message
    news:[email protected]..
    I am using Weblogic as a web server and I have enabled http logging to
    the access.log file. What is the BEA recommended method for copying and
    purging the access.log file on a regular (daily) basis?
    - Kevin

  • Parse a log file...

    Hi All,
    1. I want to parse the content of log file, but when I open the log file it does not show me field names .
    it starts with row containing the contents directly, where i want to read and process only three fields randomly.
    I have written the code that works on IIS logs, the log i want to parse having field separator ' ' as single white space.
    2. Some of log files are zipped, so I am unable to open and read them. so that I can parse them.
    can any one have any clue or code that help me out.
    thanks!

    bhatnagarudit wrote:
    Hi All,
    1. I want to parse the content of log file, but when I open the log file it does not show me field names .
    it starts with row containing the contents directly, where i want to read and process only three fields randomly.
    I have written the code that works on IIS logs, the log i want to parse having field separator ' ' as single white space.
    2. Some of log files are zipped, so I am unable to open and read them. so that I can parse them.
    can any one have any clue or code that help me out.
    thanks! Here is a suggested algorithm .. (I don't want to write the code for you :-))
    You have the following format.
    314159b66967d86f031c7249d1d9a8024.. mybucket +[04/Aug/2006:22:34:02 +0000]+ 72.21.206.5 314159b66967d86f031c724... 3E57427F33A59F07 REST.PUT.OBJECT* /photos/2006/08/puppy.jpg +"GET /mybucket/photos/2006/08/puppy.jpg?x-foo=bar"+ 200 NoSuchBucket 2662992 3462992 70 10 "http://www.amazon.com/webservices" "curl/7.15.1"
    Read the file in, go thru lines one by one. For each line,
    1. Get the content in the first square brackets. Regular Expression: [&].
    2. From there, get the fourth (4th) word separated by space.
    3. From there, get the content in the first pair of double quotes. Regular Expression: \"&\".

  • Parse XMLFormatter log files back into LogRecords

    My app uses a java.util.logging.FileHandler and XMLFormatter to generate XML log files, which is great. Now I want to display those logs in my app's web admin console.
    I assumed that there would be an easy way to read my XML logs files back into a List of LogRecord objects, but my initial investigation has revealed nothing.
    Anyone have any advice?

    If you remove an "active" log file, then this can cause problems. If you remove an archieved log file, then it is OK.
    If you change the log directory, then you SHOULD inform all your applications that use this directory... Depending on the service, this information is usually stored inside the config files of the services.
    Mihalis.

  • Cisco Prime Infrastructure 1.3 Web Log In Error

    Hi,
    I recently installed Cisco Prime Infrastructure 1.3 per the set up guide. I am able to log into the VM back end but when I try to log into the web interface with the admin account I get the following error message "Invalid Username or Password. Please try again". I verified that the username and password are correct but I still can not get in.

    I think there is some bug (or feature) that makes that the accounts defined during setup are not valid in the web gui.
    Since the users I created using CLI, work when I SSH into the PI 1.3, I know they are correct.
    Clearly the web part uses a different way to authenticate users. I'm told all this worked just fine in PI 1.2, so not sure what cisco is up to.
    To resolve this use the root account to login in the web interface.
    It has the same pw as the admin user.
    There you can create an admin account (and or others)
    Cheers,
    Michel

Maybe you are looking for