DX Infrastructure Manager

Expand all | Collapse all

Need help with RegEx in Logmon...

  • 1.  Need help with RegEx in Logmon...

    Posted 01-16-2018 12:25 PM

    Please help me understand why this regex doesn't work.

    I've tried with, and without format rules.

    With format rules my 'start expression' is ({) and 'end expression' (})

     

    Regex:

    /.*(Application)\W+(.*zip).*(Status)\W+((?!STARTED)\w+).*(Updated)\W+(.*\w+).*/

    or

    /(Application)\W+(.*zip).*(Status)\W+((?!STARTED)\w+).*(Updated)\W+(.*\w+)/

     

    Log-file looks like this:

    [
       {
          "Application": "items-inventory-api-mos-test items_inventory_api-v1.0-external-3.8.x.zip",
          "Status": "FAILED",
          "Updated": "a month ago"
       }
    ]

     

    It seems like the watcher only can manage to do one line at a time, even with the format.



  • 2.  Re: Need help with RegEx in Logmon...

    Posted 01-16-2018 12:52 PM

    You are correct, by default logmon only will look at one line at a time.

    The use of the format is needed look at blocks of information.

     

    However when I put your regex in a test tool with your sample log it does not work.

     

    you might try something like the below:

    /(Application)\W+(.*zip).*\r\n.*(Status)\W+((?!STARTED)\w+).*\r\n.*(Updated)\W+(.*\w+)/

     

    I did not test in logmon but did test in regex budy and this worked over multiple lines you will still need to use the format block.



  • 3.  Re: Need help with RegEx in Logmon...

    Posted 01-17-2018 03:06 AM

    Ok, I will try.
    I thought that \n (newline) wasn't necceassary when using the format block, and that logmon was instead in single line-mode.



  • 4.  Re: Need help with RegEx in Logmon...

    Posted 01-17-2018 04:19 AM

    Hmm dosn't work... tried yours and also tried this one: 

    (Application)\W+(.*zip).*\n\W+(Status)\W+((?!STARTED)\w+).*\n\W+(Updated)\W+(.*)\S

    Jan 17 10:15:16:550 [2660] logmon: check_profiles - found C:\Nimbus\log\Anypoint\test.txt
    Jan 17 10:15:16:550 [2660] logmon: check_profiles - adding C:\Nimbus\log\Anypoint\test.txt
    Jan 17 10:15:16:550 [2660] logmon: check_profiles - profile [Test Profile] allocated
    Jan 17 10:15:16:550 [2660] logmon: check_profiles - filename added: C:\Nimbus\log\Anypoint\test.txt
    Jan 17 10:15:16:550 [2660] logmon: check_profiles - added subject, alarm and qos
    Jan 17 10:15:16:551 [2660] logmon: check_profiles - ident=name
    Jan 17 10:15:16:551 [2660] logmon: check_profiles - added mode and interval
    Jan 17 10:15:16:551 [2660] logmon: check_profiles - added formats
    Jan 17 10:15:16:551 [2660] logmon: check_profiles - added max alarm message
    Jan 17 10:15:16:551 [2660] logmon: check_profiles - added watchers
    Jan 17 10:15:16:551 [2660] logmon: check_profiles - added exclude
    Jan 17 10:15:16:551 [2660] logmon: check_profiles - added expansion
    Jan 17 10:15:16:551 [2660] logmon: check_profiles - ending scan
    Jan 17 10:15:16:551 [2660] logmon: check_profiles - done with file search of C:\Nimbus\log\Anypoint\
    Jan 17 10:15:16:553 [2660] logmon: check_profiles - [Test Profile] running in thread
    Jan 17 10:15:16:553 [3872] logmon: fileInit - martinservera\x-pulsen-scom
    Jan 17 10:15:16:553 [3872] logmon: setting privileges needed
    Jan 17 10:15:16:553 [3872] logmon: impersonating user x-pulsen-scom in domain martinservera
    Jan 17 10:15:16:579 [3872] logmon: fileSetId - <no user>
    Jan 17 10:15:16:579 [3872] logmon: fileFind - getting file lock
    Jan 17 10:15:16:579 [3872] logmon: fileFind - got file lock
    Jan 17 10:15:16:579 [3872] logmon: [Test Profile] start scanning 'C:\Nimbus\log\Anypoint\test.txt'
    Jan 17 10:15:16:579 [3872] logmon: [Test Profile] storing file-stats as 'Test Profile'
    Jan 17 10:15:16:579 [3872] logmon: getFromDB()- In
    Jan 17 10:15:16:580 [3872] logmon: SREQUEST: nametoip ->192.168.81.123/48000
    Jan 17 10:15:16:580 [3872] logmon: RREPLY: status=OK(0) <-192.168.81.123/48000 h=37 d=38
    Jan 17 10:15:16:580 [3872] logmon: SREQUEST: _close ->192.168.81.123/48000
    Jan 17 10:15:16:581 [3872] logmon: SREQUEST: db_open ->192.168.81.123/48009
    Jan 17 10:15:16:581 [3872] logmon: RREPLY: status=OK(0) <-192.168.81.123/48009 h=37 d=0
    Jan 17 10:15:16:581 [3872] logmon: SREQUEST: db_get ->192.168.81.123/48009
    Jan 17 10:15:16:581 [3872] logmon: RREPLY: status=OK(0) <-192.168.81.123/48009 h=37 d=233
    Jan 17 10:15:16:581 [3872] logmon: getFromDB: mode 514
    Jan 17 10:15:16:581 [3872] logmon: getFromDB: eof 161
    Jan 17 10:15:16:581 [3872] logmon: getFromDB: mtime 1516120086
    Jan 17 10:15:16:581 [3872] logmon: getFromDB: checksum 3432
    Jan 17 10:15:16:581 [3872] logmon: getFromDB: refbuf
    Jan 17 10:15:16:581 [3872] logmon: getFromDB: refEOF 161
    Jan 17 10:15:16:581 [3872] logmon: getFromDB: refchar_first 0
    Jan 17 10:15:16:581 [3872] logmon: getFromDB: refchar_last 0
    Jan 17 10:15:16:581 [3872] logmon: getFromDB: refsize 161
    Jan 17 10:15:16:581 [3872] logmon: SREQUEST: db_close ->192.168.81.123/48009
    Jan 17 10:15:16:581 [3872] logmon: SREQUEST: _close ->192.168.81.123/48009
    Jan 17 10:15:16:581 [3872] logmon: getFromDB()- Out
    Jan 17 10:15:16:581 [3872] logmon: Filename = C:\Nimbus\log\Anypoint\test.txt
    Jan 17 10:15:16:581 [3872] logmon: FileOpen: h->curr.mtime 1516120086
    Jan 17 10:15:16:581 [3872] logmon: FileOpen: h->curr.eof 161
    Jan 17 10:15:16:581 [3872] logmon: FileOpen: h->save.mode 0 & h->mode 513
    Jan 17 10:15:16:581 [3872] logmon: (detectFileEncoding) read buffer[161] bytes
    Jan 17 10:15:16:581 [3872] logmon: (detectFileEncoding)Encoding of the file is '0'
    Jan 17 10:15:16:582 [3872] logmon: (detectFileEncoding)Encoding of the file is 'ISO-8859-1'
    Jan 17 10:15:16:582 [3872] logmon: lgm: Read File
    Jan 17 10:15:16:582 [3872] logmon: lgm: read the line: [[]
    Jan 17 10:15:16:582 [3872] logmon: lgm: check format start..[0]
    Jan 17 10:15:16:582 [3872] logmon: lgm: format start
    Jan 17 10:15:16:587 [3872] logmon: lgm: Read File
    Jan 17 10:15:16:587 [3872] logmon: lgm: read the line: [ {]
    Jan 17 10:15:16:587 [3872] logmon: lgm: check format start..[0]
    Jan 17 10:15:16:587 [3872] logmon: lgm: format start
    Jan 17 10:15:16:587 [3872] logmon: lgm: FORMAT END START
    Jan 17 10:15:16:587 [3872] logmon: lgm: read the line: [ "Application": "items-inventory-api-mos-test items_inventory_api-v1.0-external-3.8.x.zip",]
    Jan 17 10:15:16:587 [3872] logmon: lgm: check format start..[1]
    Jan 17 10:15:16:587 [3872] logmon: lgm: pcre_exec returned -1
    Jan 17 10:15:16:587 [3872] logmon: lgm: offset[0]: -1 offset[1]: -1
    Jan 17 10:15:16:587 [3872] logmon: lgm: pcre_exec retCode :[-1]
    Jan 17 10:15:16:587 [3872] logmon: lgm: read the line: [ "Status": "STARTED",]
    Jan 17 10:15:16:587 [3872] logmon: lgm: read the line: [ "Updated": "a month ago"]
    Jan 17 10:15:16:587 [3872] logmon: lgm: read the line: [ }]
    Jan 17 10:15:16:587 [3872] logmon: lgm: pcre_exec returned 2
    Jan 17 10:15:16:587 [3872] logmon: lgm: offset[0]: 2 offset[1]: 3
    Jan 17 10:15:16:587 [3872] logmon: lgm: pcre_exec retCode :[2]
    Jan 17 10:15:16:587 [3872] logmon: [Test Profile] FORMAT END [tewst] - ' }'
    Jan 17 10:15:16:587 [3872] logmon: lgm: excludeLine:[0]
    Jan 17 10:15:16:587 [3872] logmon: (scan) Watcher1 offset 0
    Jan 17 10:15:16:587 [3872] logmon: [Test Profile] In WithI18n section [ {],[ERCPÝ],[ISO-8859-1],[-1]
    Jan 17 10:15:16:587 [3872] logmon: [Test Profile] In WithI18n section [ "Application": "items-inventory-api-mos-test items_inventory_api-v1.0-external-3.8.x.zip",],[ERCPÝ],[ISO-8859-1],[-1]
    Jan 17 10:15:16:587 [3872] logmon: [Test Profile] In WithI18n section [ "Status": "STARTED",],[ERCPÝ],[ISO-8859-1],[-1]
    Jan 17 10:15:16:587 [3872] logmon: [Test Profile] In WithI18n section [ "Updated": "a month ago"],[ERCPÝ],[ISO-8859-1],[-1]
    Jan 17 10:15:16:587 [3872] logmon: [Test Profile] In WithI18n section [ }],[ERCPÝ],[ISO-8859-1],[-1]
    Jan 17 10:15:16:587 [3872] logmon: lgm: read the line: []]
    Jan 17 10:15:16:592 [3872] logmon: lgm: Read File
    Jan 17 10:15:16:592 [3872] logmon: lgm: read returned null
    Jan 17 10:15:16:592 [3872] logmon: lgm: p->format: 0 format: 0 mode 0x201
    Jan 17 10:15:16:592 [3872] logmon: [Test Profile] used 12 ms scanning 154 bytes
    Jan 17 10:15:16:592 [3872] logmon: (scan) - before ptScanPClose...
    Jan 17 10:15:16:592 [3872] logmon: (ptScanPclose) - closing Test Profile modified
    Jan 17 10:15:16:592 [3872] logmon: (ptScanPclose) - before storeInDB
    Jan 17 10:15:16:592 [3872] logmon: (ptScanPclose) - after storeInDB
    Jan 17 10:15:16:592 [3872] logmon: (ptScanPclose) - leaving
    Jan 17 10:15:16:592 [3872] logmon: (scan) - after ptScanPClose...
    Jan 17 10:15:16:592 [3872] logmon: scan - getting csMappingLock lock
    Jan 17 10:15:16:592 [3872] logmon: scan - leaving csMappingLock lock
    Jan 17 10:15:16:592 [3872] logmon: fileCleanup - <no user>



  • 5.  Re: Need help with RegEx in Logmon...

    Posted 01-19-2018 05:14 AM

    So, we managed to get the output to a single line:

    log file:

    [ { Application: items-inventory-api-mos-prod items_inventory_api-v1.0-ext-prod-3.9.x.zip, Status: FAILED, Updated: an hour ago }, { Application: net-tools-api-prod-mos net-tools-api-master.zip, Status: UNDEPLOYED, Updated: 5 days ago } ]

    Problem is that I can't get it to work as a global mode ("Don't return after first match"). Which makes the regex only to match the first application.

    I'm using the CAT-mode with Logmon.

    I have a similar logmon-profile, URL-mode, checking a xml-file. And that profile will keep matching the rest of the XML.

    Are CAT and URL not working the same way?

     

    Regex:

    /(Application)\W\s(\S+)\s\S+\s(Status)\W+((?!STARTED)\w+)\W\s(Updated)\W\s([^.]*)\s}\W+/

     

    So... How do I get the watcher to keep matching after the first match?



  • 6.  Re: Need help with RegEx in Logmon...

    Posted 01-17-2018 06:57 AM

    A single expression in regex won't work with multiline. It only read line by line.

     

    But you can use a block with a Start/End of an expression. Doing this logmon will read only what is inside this block. If I understood what you trying to do, you only need to evaluate if the status is different of STARTED inside a block starting with Application/SOMETHING/zip extension and ending with a period of time.

     

    So you can:

    • Create a profile and point the logfile
    • Create for this profile a New Format Rules (ex.: zipfile)
      • Start Expression: /Application.*\.zip/
      • End Expression: /Updated.*/ or you can use a number of lines.
    • After this you can create a watcher that will only analyze lines inside this block, and now you can use a simple expression: /Status.:.*(?!STARTED)/


  • 7.  Re: Need help with RegEx in Logmon...

    Posted 01-17-2018 07:04 AM

    Hi,

     

    I wrote in my first post that this is inside a format (block)

    With format rules my 'start expression' is ({) and 'end expression' (}

    I'm not looking for only one line, I want to gather more information. That's why I use the block/format.

     

    And I guess that's why Gene are using \n in his exampel. But I can't get that to work. I also think that \n is not usable in single line mode. Which the format/block should maybe be? Not sure.



  • 8.  Re: Need help with RegEx in Logmon...

    Posted 01-17-2018 08:47 AM

    Yeah! You cannot use \n. It always read line by line, even inside the block.

     

    I think that you will need to create small blocks to analyze, start with { and end with } will englobe too many lines/variables. You will need to find a logic using a few lines only inside of the blocks.

     

    What do you need to alarm? 



  • 9.  Re: Need help with RegEx in Logmon...

    Posted 01-17-2018 09:10 AM

    Are you really sure it should see it as line by line when using blocks?

    So, let's say I want to match 10 words in one log. All of them are on different lines. Then I would need 10 format rules, and 10 watcher rules? Also... this would generate 10 different alarms.

     

    What I've heard from before (here on the forum I guess) is that when using block, it's equal as having everything in single line-mode. And that's the way to not use \n .

     

    It's a bit confusing when Gene (CA) gives an example with \n, and later you (CA) say that \n is not possible, and also that everything is only matched line by line within a block.



  • 10.  Re: Need help with RegEx in Logmon...

    Posted 01-17-2018 09:34 AM

    Hi I am very sorry if I caused any confusion.

    As I stated I did not have a chance to test this in longmon. I just reviewed your regex as your asked and provide a working regex statement.

     

    I did do some testing in logmon but as I am unlcear what exactly you need to alarm on I could only guess is that you want to check the status field for failed or some other value.

     

    I created the following log file:

        [
           {
              "Application": "items-inventory-api-mos-test items_inventory_api-v1.0-external-3.8.x.zip",
              "Status": "FAILED",
              "Updated": "a month ago"
           }
        ]
            [
           {
              "Application": "items-inventory-api-mos-test items_inventory_api-v1.0-external-3.8.x.zip",
              "Status": "STARTED",
              "Updated": "a month ago"
           }
        ]
            [
           {
              "Application": "items-inventory-api-mos-test items_inventory_api-v1.0-external-3.8.x.zip",
              "Status": "FAILED",
              "Updated": "a month ago"
           }
        ]
            [
           {
              "Application": "items-inventory-api-mos-test items_inventory_api-v1.0-external-3.8.x.zip",
              "Status": "STARTED",
              "Updated": "a month ago"
           }
        ]

     

    They setup the following profile for logmon and was able to alert on the failed status.

       <testFormat>
          active = yes
          interval = 5 sec
          scanfile = C:\Temp\multilineformat.log
          fileencoding =
          scanmode = cat
          alarm = yes
          qos = yes
          message = no
          subject =
          user =
          resetFile = no
          initialfileptr = 2
          resumefileptr = 4
          command_timeout_active = no
          command_timeout =
          command_severity = 2
          command_timeout_alarm = 0
          alarmFOpenFail = no
          clearFOpenFailRestart = no
          monitor_exit_code = No
          max_alarm_sev = 5
          max_alarms =
          max_alarm_msg =
          password =
          <formats>
             <formatrule>
                active = yes
                start = /(Application)\W+(.*zip).*/
                end = /(Updated)\W+(.*\w+)/
                lines = 0
             </formatrule>
          </formats>
          <watchers>
             <watcher test>
                active = yes
                match = /(Status)\W+((?!STARTED)\w+).*/
                level = warning
                subsystemid =
                message = ${WATCHER} - Messag: ${var}
                i18n_token =
                restrict =
                expect = no
                abort = no
                sendclear = no
                count = no
                separator =
                suppid =
                source =
                target =
                qos =
                runcommandonmatch = no
                alarm_on_first_match = no
                commandexecutable =
                commandarguments =
                pattern_threshold_severity = information
                pattern_threshold_message =
                timeout = 1
                pattern_threshold =
                expect_message =
                expect_level =
                regexfromexternalfile = no
                patternfilepath =
                token =
                <variables>
                   <var>
                      definition = *
                   </var>
                </variables>
             </watcher test>
          </watchers>
       </testFormat>

     

     

    hope this helps



  • 11.  Re: Need help with RegEx in Logmon...

    Posted 01-17-2018 09:42 AM

    Thanks for the answer.

     

    There is no problem for me to match just one line. But that's not what I want

    As my regex says... it tries to match and group more words, on different lines. Which I later want to use as variables.

     

    It looks like this:

     

    Group 1.11-22`Application`
    Group 2.26-98`items-inventory-api-mos-test items_inventory_api-v1.0-external-3.8.x.zip`
    Group 3.106-112`Status`
    Group 4.116-123`STOPPED`
    Group 5.131-138`Updated`
    Group 6.142-153`a month ago`


    These groups will later be my variables in logmon.

    And I want to have this in one watcher, and one alarm.



  • 12.  Re: Need help with RegEx in Logmon...

    Posted 01-17-2018 09:49 AM

    unfortunately logmon does not have a grouping mechanism as your describe at this point that I can think of that would allow for what you are asking.



  • 13.  Re: Need help with RegEx in Logmon...

    Posted 01-17-2018 10:13 AM

    I did a little more testing as was able to get all of your different break out into variables so my alarm message looked like:

    watcher test - Group1:Application - Group2:"items-inventory-api-mos-test items_inventory_api-v1.0-external-3.8.x.zip - Group3:Status -  Group4:FAILED - Group5:Updated - Group6:a month ago

     

    the below config is what allowed this. If this will not meet your needs you will probably need to do some scripting to get what you need in a format that logmon can handle or open an ER, unless some one else has a suggestion for you.

     

       <testFormat>
          active = yes
          interval = 5 sec
          scanfile = C:\Temp\multilineformat.log
          fileencoding =
          scanmode = cat
          alarm = yes
          qos = yes
          message = no
          subject =
          user =
          resetFile = no
          initialfileptr = 2
          resumefileptr = 4
          command_timeout_active = no
          command_timeout =
          command_severity = 2
          command_timeout_alarm = 0
          alarmFOpenFail = no
          clearFOpenFailRestart = no
          monitor_exit_code = No
          max_alarm_sev = 5
          max_alarms =
          max_alarm_msg =
          password =
          <formats>
             <formatrule>
                active = yes
                start = /(Application)\W+(.*zip).*/
                end = /(Updated)\W+(.*\w+)/
                lines = 0
             </formatrule>
          </formats>
          <watchers>
             <watcher test>
                active = yes
                match = /(Status)\W+((?!STARTED)\w+).*/
                level = warning
                subsystemid =
                message = ${WATCHER} - Group1:${var} - Group2:${var2} - Group3:${var3} -  Group4:${var4} - Group5:${var5} - Group6:${var6}
                i18n_token =
                restrict =
                expect = no
                abort = no
                sendclear = no
                count = no
                separator =
                suppid =
                source =
                target =
                qos =
                runcommandonmatch = no
                alarm_on_first_match = no
                commandexecutable =
                commandarguments =
                pattern_threshold_severity = information
                pattern_threshold_message =
                timeout = 1
                pattern_threshold =
                expect_message =
                expect_level =
                regexfromexternalfile = no
                patternfilepath =
                token =
                <variables>
                   <var>
                      definition = 1/12:22
                      operator = eq
                   </var>
                   <var2>
                      definition = 1/26:98
                      operator = eq
                   </var2>
                   <var3>
                      definition = 2/12:17
                      operator = eq
                   </var3>
                   <var4>
                      definition = 2/22:27
                      operator = eq
                   </var4>
                   <var5>
                      definition = 3/12:18
                      operator = eq
                   </var5>
                   <var6>
                      definition = 3/23:33
                      operator = eq
                   </var6>
                </variables>
             </watcher test>
          </watchers>
       </testFormat>



  • 14.  Re: Need help with RegEx in Logmon...

    Posted 01-17-2018 11:52 AM

    Nice! Thanks for that!

    Problem here is when there will be more apps like "items-inventory-api-mos-test items_inventory_api-v1.0-external-3.8.x.zip" with other names. I would need to add new variables and count the positions/lines.

    Also new formats.

     

    Do you have any other recommendation? Something else than Logmon maybe?

     

     



  • 15.  Re: Need help with RegEx in Logmon...

    Posted 01-17-2018 11:50 AM

    rith escreveu:

     

    Are you really sure it should see it as line by line when using blocks?

     

    Yes. There is a idea for this question: Treat multi-line log messages as single block and allow variables to work across the whole block 

     

    Currently (version 3.91) the logmon probe replaces any carriage returns ( \r ) or new lines ( \n ) with spaces when parsing a log file. 



  • 16.  Re: Need help with RegEx in Logmon...

    Posted 01-17-2018 11:54 AM

    Ok. Thanks Leandro!