Capture date with regular expression at file.log with logmon probe

[Miller Echagarreta]

Hi Everyone !

I have a request of monitoring, where we need to monitoring the date of the day on a file...

This data comes from a file that i read with logmon probe:

Example:

Listado de /DATALAKE/LANDINGS/BODEGA_DE_DATOS
Found 35 items
-rw-r--r--   3 nififlow supergroup      38893 2024-07-04 00:03 /DATALAKE/LANDINGS/BODEGA_DE_DATOS/ACTIVIDAD_ECONOMICA.csv
-rw-r--r--   3 nififlow supergroup      12234 2024-07-04 00:03 /DATALAKE/LANDINGS/BODEGA_DE_DATOS/APL_DIMEN.csv
-rw-r--r--   3 nififlow supergroup  218719671 2024-07-03 00:16 /DATALAKE/LANDINGS/BODEGA_DE_DATOS/APL_TRADUCCION_INSTAN_DIMEN.csv
-rw-r--r--   3 nififlow supergroup        580 2024-07-04 00:03 /DATALAKE/LANDINGS/BODEGA_DE_DATOS/CARACT_COMERCIO_ELECT.csv

If today is 2024-07-04 and a line of the file indicate 2024-07-03 i need generate the alarm that the indicate the route (/DATALAKE/LANDINGS/BODEGA_DE_DATOS/APL_TRADUCCION_INSTAN_DIMEN.csv) wasn't updated...

 

That is possible doing this with logmon probe or do you have another idea... ?

[Luc Christiaens]

You can try the monitoring of the "file age" with the dirscan probe, tab "age of file".

Via a schedule you can define when to check/run so you can decide on when you decide a file is too old.

The alarm will give you the file name + how old in minutes.

[Miller Echagarreta]

Hi Luc, i can't to use dirscan because the files is only possible obtaing with a command... "hdfs dfs -ls /DATALAKE/LANDINGS", this is files from Hadoop Distributed File System (HDFS). They isn't conventional linux command "ls -lah".

So I managed to do this configuration through logmon, generating a file from a shell script that exports the data of the command performed to a file.log. I read this file and generate the alarm when it matches the search pattern

[Rajesh Banuka]

Hi Miller,

I don't see a way to compare parsed value with current date (dynamic value). We may have to use a script.

Rgds,

Rajesh B

[Luc Christiaens]

Hello, I just posted an older probe: probe_cmd that could (perhaps) do what you want/need.

In the probe add the configuration:

-------

      <12>
         active = yes
         description = check if a file last update contains a todays date
         resources = bgbulab58
         cmd = type "c:\temp\rabbitmq_heartbeat_intern.bat.log"
         qos_on_exit_code = no
         print_all_error_messages = no
         start_time = 0700
         end_time = 1700
         percentage = 0
         qos_prefixed = no
         interval = 180
         qos_source = remote
         create_current_date_var = yes
         <watchers>
            <0>
               regexp = date:.(\w+),.(\d+).(\w+).(\d+).(\d\d:\d\d:\d\d).GMT
               <matches>
                  <1>
                     source = $resource
                     target = $resource
                     alarm_eval = INFO:($v ne $a_dd.$a_mon.$a_yyyy) 
                     alarm_description = "date_check server: $resource invalid update date: $v ne $a_dd.$a_mon.$a_yyyy"
                     suppression_key = "date_check"
                     samplevalue = $2.$3.$4
                  </1>
               </matches>
            </0>
         </watchers>
      </12>

----

Where you change the "cmd" line with your command so that the regexp expression can search in the output for a match and will execute the "alarm_eval" with the date verification rule and send an alarm when the dates are not equal.