Monitoring with logmon

[Miller Echagarreta]

Hi everyone.
I need monitoring a port in state "listen" locally.
I can't to use net_connect because isn't permit make query to the network for security reason of customer. So my estrategy focus in to use logmon probe.

This is my configuration:
1) I am capturing the state the port with a profile in the command mode. (netstat -n -a | grep tcp | grep "LISTEN" | grep -v "grep" > /opt/nimsoft/probes/system/logmon/netstat.log). The output of the profile is ---->

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:48000 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:48001 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:48007 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:48008 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:48009 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:48010 0.0.0.0:* LISTEN
tcp6 0 0 :::22 :::* LISTEN
tcp6 0 0 ::1:25 :::* LISTEN

2) I am reading the log (/opt/nimsoft/probes/system/logmon/netstat.log), for example with this regex: /127.0.0.1:25\s+0.0.0.0:\W\s+LISTEN/
This is regex working god, and generate an alarm ---->

But although the above is fine, I need the opposite, that the alert is generated when (127.0.0.1:25 0.0.0.0:* LISTEN) is not found, which would indicate that the port is no longer connecting up.

Is possible this with logmon? Or maybe I need to change my regex? I don't know how to meet this requirement

Any idea?

[Gregg Stillwell]

HI Miller, You could use negative RegX and you could use command mode or you could use exit code monitoring in command mode. If you run your program in command load(don't read file) netstat -n -a | grep (your pattern you want to see). If it finds the exit code will be 0 if not 1. See example. You could also | to wc -l and use a variable for operend (<,>,=), No need to create file.

[root@REDHATTest ~]# ls -l
total 56
-rw-------. 1 root root 6921 Jan 28 2019 anaconda-ks.cfg
-rw-------. 1 root root 6577 Jan 28 2019 original-ks.cfg
-r-xr-xr-x. 1 root root 2102 Jul 5 2019 rc.local
-rw-r--r--. 1 root root 3 Jun 17 13:18 sepap-install.log
-rw-r--r--. 1 root root 267 Feb 29 19:20 sepfl-install.log
-rw-r--r--. 1 root root 11010 Mar 18 2019 sepfl-install.log.13372
-rw-r--r--. 1 root root 267 Jul 2 2019 sepfl-install.log.15773
-rw-r--r--. 1 root root 267 May 22 2019 sepfl-install.log.16277
-rw-r--r--. 1 root root 243 Mar 18 2019 sepfl-kbuild.log
-rw-r--r--. 1 root root 629 Mar 18 2019 sep-install.log
[root@REDHATTest ~]# echo $?
0
[root@REDHATTest ~]# ls -l |grep GG
[root@REDHATTest ~]# echo $?
1
[root@REDHATTest ~]# ls -l |grep log
-rw-r--r--. 1 root root 3 Jun 17 13:18 sepap-install.log
-rw-r--r--. 1 root root 267 Feb 29 19:20 sepfl-install.log
-rw-r--r--. 1 root root 11010 Mar 18 2019 sepfl-install.log.13372
-rw-r--r--. 1 root root 267 Jul 2 2019 sepfl-install.log.15773
-rw-r--r--. 1 root root 267 May 22 2019 sepfl-install.log.16277
-rw-r--r--. 1 root root 243 Mar 18 2019 sepfl-kbuild.log
-rw-r--r--. 1 root root 629 Mar 18 2019 sep-install.log
[root@REDHATTest ~]# echo $?
0

[Miller Echagarreta]

Oh ... Gregg, I didn't know that logmon could work this way.
I configured the command monitoring like this ---->

Exist some way of clear the alarm when the condition change?

[Gregg Stillwell]

You can get rid of the grep -v grep. The exit code only applies to the last grep/command. 

Clearing is a little tricky. I think I have successfully used this doc to do it.

https://knowledge.broadcom.com/external/article?articleId=135865

[Gregg Stillwell]

Or to clear you can just add another setting in your threshold window of the exit code settings, to send a clear on a 0....I think so anyway. Try it out and post if that worked for the clear.

[Miller Echagarreta]

Gregg, I thought on the solution that you explain, but the severity "clear" not exist. Maybe this is an improvement.

The solution that I thought is to create a new AO in NAS that find the alarm of logmon with expression filter and "Close" action, on overdue age = 4s. In my LAB it's working.

[Gregg Stillwell]

Yes that would be too easy, maybe an enhancement. You could get a clear if the command you used was netstat -an | grep (pattern to watch) | wc -l. The output would be 1 from wc -l if it finds 1 match and 0 if no match. Then you could create 2 watchers which would be test your output in the variable section on alarm according to value.

The "Ping section" in this doc will help. In your case the "Match Expression" would be just a "*" in your 2 watchers but your varible section would use operend to check for your command value.
https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/ca-unified-infrastructure-management-probes/GA/alphabetical-probe-articles/logmon-log-monitoring/logmon-use-case-examples.html

[Garin Walsh]

You can post the output through "wc -l" to count the lines returned - then you can have watchers for "/^0$/" to get the error and "^[^0].*" to get everything else to clear.

[Miller Echagarreta]

Excellent Gregg, this solutions it's much better. This is my command.

netstat -an | grep "LISTEN" | grep 127.0.0.1:25 | grep 0.0.0.0:* | wc -l

My whatchers are ----> /0/ = port down and /1/ = port establlished

Thanks so much for teach me.

[Gregg Stillwell]

No problem Miller, and you are using Garin's suggestion for evaluation, I do it in the variables tab....good luck.

[James Christensen]

One important point when it comes to the sending of a "clear" message from logmon (actually important when using logmon in general) is that you should set the suppression key for each watcher rule and not just let it be the logmon default. If you have the two watcher rules (one that sends an alarm and one to clear) the suppression key for both rules should be identical. Even when not sending clear messages, the suppression key should be set for each watcher rule to ensure each alarm is separate and distinct from other alarms coming from other profiles and watchers or one profile with multiple watchers configured in the same logmon probe instance. You can use built-in variables so that the key contains the profile and watcher names.

------------------------------
[Designation]
[City]
------------------------------

[Miller Echagarreta]

Thanks James, your information it's true. In fact Gregg published this link https://knowledge.broadcom.com/external/article?articleId=135865