DX Unified Infrastructure Management

 View Only
  • 1.  REGEX: Part 1 - The Basics

    Posted Aug 28, 2012 04:27 PM

    We get a lot of cases logged with regards to Regex and it's usage, for those who may not have used regex before here is a little guide:

     

    Regex is a powerful pattern matching language used as part of many of the Nimsoft Probes and Tools, the following is a brief introduction to Regex and it’s usage within the Nimsoft NMS software.

     

    The first point to make is Nimsoft implementation regex is based around Perl’s implementation (Other flavors use differing syntax).

     

    When specifying a pattern match using REGEX in NMS we need to tell the NMS software we want to use Regex, we do this by opening and closing the syntax with / [Forward-slash] as in the following example.

     

    /<regex syntax goes here>/

     

    The next most basic usage is the “match all” characters statement:

     

    .*            (Dot Asterisk) – Dot represents any alpha-numeric, character or special character. Asterisk represents “Any number of times”. The two used in conjunction create the expression match anything any number of times (everything!)

     

    |              (pipe Symbol) Is used as an OR operator

     

    \              (Back-slash) Is the escape character operator and is used to escape special characters. As a example if you wanted to match a back-slash in an expression, it has a special meaning in regex and hence has to be escaped like \\

     

    \s            Matches white spaces, ie breaks between words

     

     

     

    Let’s take a look at a very simple REGEX statement to match an alarm with the message of:

     

    Average (5 samples) total cpu is now 82.61%, which is above the warning threshold (75%)

     

     

    /.*total\scpu.*above.*threshold./

     

    The above expression would match an alarm that comes into the NAS wich states total cpu is above its defined threshold. Using the example and syntax definitions try to work out how this works.

     

     

    Now let’s imagine we would like to match a similar alarm but for memory:

     

    Average (5 samples) total cpu is now 82.61%, which is above the warning threshold (75%)

     

     

    We could write a similar expression as to the one previous, however in some cases it might make more sense to have one regex rule match both alarms we do this by using “grouping” syntax and the | (pipe) [OR] operator.

     

    ()             (Open, Close Parenthesis) Wrapping syntax in parenthesis creates a “group” this is useful to us in two ways. It allows us to isolate certain parts of our syntax and hence use operators local to that part of the expression or use references to those “groups” in the NMS software (The logmon probe is a good example of addressing groups). In the following example we use a “group” to isolate part of our syntax so we can use the OR operator on just that section of the expression.

     

    /.*total\s(cpu|memory).*above.*threshold.*/

     

    Notice the group (cpu|memory) section which essentially states if the string matches cpu OR memory then match. Grouping this section is very important without the parenthesis the expression would say match the string:

     

    total cpu

    OR

    Memory above threshold

     

    So that’s the basics covered and you should now be able to create pattern matches based on regular expressions within the Nimsoft NMS Software. You will find these techniques especially useful then using the NAS Auto-Operators and probes such as Logmon.



  • 2.  Re: REGEX: Part 1 - The Basics

    Posted Aug 29, 2012 08:05 AM

    Good stuff, Carl! If you are going to be posting more stuff like this, maybe there should be a new message board for things like tutorials. That might help people find such helpful information in the future as they move down the list of recent posts.

     

    Regarding your example, I would suggest one change (unless using the regex in the UMP--more on that in a minute). There are countless ways to tweak a regex, and I do not want to complicate matters with all kinds of options. But I would recommend not using .* at the start and end of a regex (outside the UMP). Regexes of any variety I have ever seen, especially PCRE (Perl-compatible regular expressions) used by NMS, do not need to match the whole string but any part of it. So using .* at the start and end of the regex is unnecessarily. Even though the regex still works with them, it is incredibly less efficient. In many cases, that should not matter much, but in pre-processing rules in the NAS it could make a big difference.

     

    Unfortunately, the UMP does not use standard regexes. I am not sure what the story is there, but I have found that regexes in the UMP need to start with .* and maybe end with it as well. Other than that, they may work the same as the rest of NMS.

     

    BTW, there is some good information in the Perl regex reference, especially if you want a list of metacharacters, quantifiers, escape sequences, or character classes:

     

    http://perldoc.perl.org/5.14.2/perlre.html

     

    This post from Carl is a much better place to start learning about regexes, but the reference can provide more detailed information when needed.



  • 3.  Re: REGEX: Part 1 - The Basics

    Posted Aug 29, 2012 11:24 AM

    The hint I brought over from Perl regex is to use anchors for the start ^ and the end $ of the line where possible. This can make the regex far more efficient. For instance a regex:

     

    /^Disk/

     

    will know it does not match if the first character of the line is not D

     

    whereas the regex

     

    /Disk/

     

    Will keep moving along the line testing each character against D

     

    Obviously if you know the start of your string this can make thing a lot quicker.

     



  • 4.  Re: REGEX: Part 1 - The Basics

    Posted Aug 29, 2012 11:38 AM
    Carl, I have been looking for something like this, very useful.

    Thank you


  • 5.  Re: REGEX: Part 1 - The Basics

    Posted Aug 29, 2012 12:27 PM

    Thank you gents for the positive and useful feedback.

     

    I have created the a new section for tutorials, not sure how to work this yet and is limited by ACL to who can create new articles however everyone can comment.

     

    I fill intergrate the responses to this post into the tutorial ASAP.

     

    I will also create a seperate post regards this new section when I have chance and we can throw some ideas around.

     

    I will be inviting you all to participate in the creation of articles should you so wish as the advise of the community is invaluable.

     

     



  • 6.  Re: REGEX: Part 1 - The Basics

    Posted Aug 29, 2012 01:03 PM

    Check out www.rubular.com

     



  • 7.  Re: REGEX: Part 1 - The Basics

    Posted Aug 29, 2012 07:22 PM

    One other thing I would like to add:

     

    Setting case insensitivity.  RegEx is (of course) case sensitive so: ErRoR or Error, or ERROR all have different settings.

     

    I saw one example in Nimsoft to use /[Ee][Rr][Rr][Oo][Rr]/ to account for it, but using (?i) will make it ignore case:

     

    /(?i)error/