Idea Details

Log file analysis in Portal

Last activity 06-03-2019 08:37 PM
Rick.Brown's profile image
04-22-2016 01:58 PM

DevTest can log a vast amount of information. I would appreciate some kind of in-depth log file analysis in Portal to investigate what each server and Workstation application is reporting, so I don't need to have direct file system access to every machine on which a piece of DevTest is installed. I envisage sets of graphs and charts as portlets on the portal home page as overviews, with a Portal section for additional functionality, such as the ability to filter and query on application, hostname, timestamp, class, level, process, etc.


Comments

12-11-2018 03:58 PM

Rick.Brown Please see if you still have the configurations mentioned. May be I can work a bit and extend in upcoming holidays.

 

Thanks

10-13-2017 03:33 PM

Can we start with whatever is available and used by Support?

I like Rick's point of including this in the Portal. When we install on a Server, most users do not have access to the log files so it makes research difficult. I created a workaround for this issue using a virtual service, but it would be nicer to productize log analysis.

 

As Rick did, I also tried Elastic Stack and got it partially working. If we were to go in the direction of tools like this, I'd like to see some canned example filters and dashboards for use. Creating the parsing across the various logs is equally confusing and prone to error.

01-02-2017 09:46 AM

I'm on vacation over Christmas, so what did I do? I looked at the configuration needed for this

 

This is what I've come up with, so far. I have no immediate need to add agent logging, so I haven't looked at the log file format of those files.

 

There are 4 products involved to make a log monitoring solution that I would be happy with:

FileBeat - this is a lightweight file reader and sender.

LogStash - this takes input from FileBeat processes and translates the log format into fields that can be queried, sending output to:

ElasticSearch - this is the monitoring storage, It also provides REST APIs for data manipulation & query.

Kibana - this is the visualisation web interface.

 

I installed v5.1.1 of each of the above products onto my Mac.

 

There were some configuration changes needed.

FileBeat:

I created two files. "vse_matches.yml" configuration, and "startFileBeat.sh" to run an instance.

 

LogStash:

I created two files. "vse_matches.conf" configuration, and "startLogStash.sh" to run an instance.

 

ElasticSearch:

No configuration needed.

 

Kibana:

No configuration needed.

 

I don't have the option to upload files here. If you want my configurations, let me know and I'll send them by email.

08-05-2016 04:25 PM

Rick.Brown,

 

We have a customer, asking how they can provide access to log files to DevTest developers/engineers without give them accesss(RDP) to the DevTest Server. I suggested change it to a folder where he could share it with read permissions where their developers could map it on their laptops to have access.

 

This sounds a much better solution, so I am asking if you could share some more about how to setup this. I would like to learn how to do it in my local environment, so I can suggest to other DevTest users.

 

Regards,  

06-24-2016 10:57 AM

In case it helps to move this idea from "under review" to something more immediate, I installed three packages this morning, whilst clearing my mind of the Brexit

 

FileBeat (to read DevTest log files and send extracted data to a web server via REST)

ElasticSearch (to store & filter log events in the web server via REST)

Kibana (to visualise the logs in a webpage)

 

I edited the filebeat.yml file to point to my DevTest log directory, but I didn't add IP address or hostname of the ElasticSearch server (which, I presume, would allow me to report log messages across a network).

I haven't looked at parsing our custom timestamps, but this would either be done in JSON config in FileBeat or as an index template in ElasticSearch

I created a visualisation in Kibana with some filters in a pie chart, to show counts of "INFO", "ERROR", DEBUG", "WARN". I added a dashboard for this visualisation.

 

I can see the overview of events in dashboard. I can also click on the Discover tab and see the first 1024 characters of every log message (filebeat.yml has a setting to change this to increase the max size of captured messages).

06-08-2016 08:48 AM

Since we are talking about graphs and charts, can we plan some sort of integration with tools like Kibana for log data visualization.

05-26-2016 04:21 PM

We currently have a tool used by support to collect log files across a DevTest environment.  Perhaps a first step would be to enable that tool for all customers.  As a second pass, it might make sense to display those log files in the portal and let them be searched and filtered, something similar to find and replace for VSI editing, without the replace capability.

 

Additional thoughts and comments would be great.