Logmon - Not seeing EOF

[Sam Green]

Hi,

Came across a strange issue today with logmon v3.92 (out of date I know, but so is the OS it's running on, old RHEL5).

I have a CSV file (quite big, about 17000 rows), browsed to it via logmon with no problem.  When I click view to read the final portion of the file, I don't see the end of the file.  It appears to stop earlier in the CSV file..

Never had this issue before.

I'm trying to post all values in a column to QoS so I can total them in SQL, then alert via jdbc_response.

Never seen this issue with logmon before, the release notes for the newer versions don't mention any fixes for this strange problem either.

Anyone experienced similar?

------------------------------
CA - UIM administrator
------------------------------

[Stephen Danseglio]

Hi Sam,

In logmon pre-4.0 versions when running on Linux, there have been some reported cases where a given logmon profile is watching /var/log/messages (file was not being rotated, so it grew huge to ~+8GB) and interval was set to 5 minutes. What is the actual size of the file containing 17k rows?

In that scenario, the logmon probe most likely never reaches the end of the file within the given interval.

Upon each interval, the file was being read from the beginning, since it could not find the last read position.

The latest logmon GA version is v4.11 so first, it would be best to use that version if possible and re-test.

But yes, note that the latest GA version is supported for RHEL 6.x and 7.x, but not v5.x.

As per the older Release Notes at support.nimsoft.com, on a 32-bit OS, if that is the case, the file size which the logmon probe can support is up to 2 GB.

Also, if the file(s) selected for monitoring are very large and using a wildcard, (~500+ depending on the file size), the probe could be crashing/creating a core dump so you might want to check for a recent core in the nimsoft installation directory.

Also check consumption of CPU and memory during reproduction of the issue if possible.

You can also refer to the following KB Article.

What is the largest logfile that can be monitored by the logmon probe?
https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=34285

Steve

------------------------------
Support Engineer
Broadcom
US
------------------------------

[Sam Green]

Hi Steve,

Thanks for getting back to me on this one.  The file is no where near those limits, so I think I can safely rule that out.  It's only 2.4mb but has 12700 rows within it.

I've exported the QoS data to a csv and sorted the columns by low-high and manually compared some of the numbers. 

It appears logmon isn't always scanning column 14, as some of the numbers in the csv logmon is reading are not appearing in the exported QoS data.  Yet I noticed that instead of the correct value, a value from column 12 replaces it.  So it appears that logmon thinks it's reading column, but it appears to be reading some from 14, and others from 12.

This is despite the fact that my watcher specifies column 14 and to 'ignore to'.  I previously tried this as: column 14 to column 14.

I then suspected by comma's maybe incorrect in the csv, which would create such behaviour.  But when I open the csv in Excel and convert text to columns using comma's, it tracks the correct columns.

So all in all, I'm not much better off at the moment, but I think I found out what it's doing, just need to figure out why!

I'll check if I can upgrade the probe to a slightly later version and retest.

Cheers,
Sam

------------------------------
CA - UIM administrator
------------------------------

[Stephen Danseglio]

Hi Sam,

Also check for non-ASCII characters in the log using Notepad++ with option View->Show Symbol-> "Show All Characters"

Steve

------------------------------
Support Engineer
Broadcom
US
------------------------------

[Sam Green]

Hi Steve,

Thank you, I've managed to get to the root of the problem.  The issue is the file itself, as it contains addresses and the addresses contain commas!

I should have thought about this before I raised it so I apologize.

Logmon was correct all along!  As far as it was concerned anyway!  After using an awk command on the file it reported the same value as logmon did, then I spotted the problematic lines!

Many thanks,
Sam

------------------------------
CA - UIM administrator
------------------------------

[Stephen Danseglio]

Ok glad you identified the problem. What did you have to do about handling the commas?

------------------------------
Support Engineer
Broadcom
US
------------------------------

[Sam Green]

Hi Steve,

I'm looking into an awk command which can handle the escaped commas within the file.

Will report back if I get something working.

Cheers,
Sam

------------------------------
CA - UIM administrator
------------------------------