Idea Details

Collect logs filtered by time stamp

Last activity 03-02-2016 08:29 PM
reiju05's profile image
07-28-2014 12:02 PM

Every time a customer opens a case, we ask to provide all logs.

The logs usually collected from agent/s, NES/s and NAC.

Sometimes it may result in huge files that takes time to collect, upload, download, unzip and of course - analyse.

I would like to have an option to filter prior to collecting.

For example, if customer report the issue started on 28.08.2014 @ 14:00, I would like the customer to collect the logs starting at 28.08.2014 @13:00 and ends @ 14:30.

 

Thanks,

Julia


Comments

03-02-2016 08:29 PM

Hi,

I like this idea. Why dont as well we can see the logs in ROC? Because it's hard to see details from actions and executions of deployments during their execution or already finished.

 

Other good idea is to have the ability to see the reports of deployments finished with all details of execution (inputs and outputs). I say because sometimes is good to see all details of finished deployments, not just for deployments which are running.

 

Thank you.

 

GA

07-07-2015 03:49 AM

Thank you all for your great ideas. I'm currently working on a specification for the logs collector enhancements.

I will keep you posted.

 

Regards,

Nofar

03-27-2015 05:46 AM

Agree, the logic is pretty easy to implement but it’s quite significant .

 

vijay

03-26-2015 03:36 PM

That's a good and critical point you raise. Perhaps it wouldn't take too much effort to incorporate basic location awareness / time zone conversion in the log collection functionality to get around that?

03-26-2015 02:01 AM

Woops, meant to reply to orig idea    

 

 

 

Best route, IMO, without more research into the logcollector functionality in place(will post more necessary if need be) - currently:

 

-- Use and improve upon the existing 'Collect Logs' functionality found in the logs.

 

When I say improve - I mean:

               a. Add the functionality Julia originally posted, ability to filter by date(s)/time(s), leaving the rest behind reducing archive size.

               b. Add additional functionality, eg; Ability to specifically select the component(s)(including agents) to archive logs from, archive type, preset collection profiles, etc, etc(I have a l

                              1a. Ability to select components to collect logs from.

                              2a. Provide alternate methods for collecting the logs outside a .zip file the default installation directory of your web browser(eg; alternate location, eg; /tmp/logs.tar.gz on the NAC) This would eliminate the issue of log archives not making it to the local machine attempting the collection

                              3a. Ability to save commonly used scenarios that apply for specific log collection with your current issue, as 'profiles' or something similar.

                              4a. Ability to switch log4j log levels for components on the fly before log collection, have these reset to previous values upon completion(no restart should be required)

                              5a. To extend upon what kogantivigay just said, address this by allowing the user to specify TZ according to UTC offset for each component that is different from NAC, do the math, and grab the right log sections.

                              6a. I have a few more if anyone is interested, but I could go on here forever...filters are our friends

 

2. Correct a lot of the bugs we encounter with this so it is more reliable(about 60-70% of the time it works currently) - a couple things I run into constantly-

                              1a. Sometimes the logs make it to the NAC, but the archive is not transferred to the local machine for whatever reason(quite a few things, a couple out of our control);

                              2a. Just doesn't work at all, usually due to the configuration used being far from default, which is expected(eg; NES(s)/NAC configured to use 8443/https, etc)

                              3a. Sometimes the log archive will be missing logs due to 2a, or other reasons I can list later, but this would coincide with 2a. in the first section.

                              4a. I have more I wrote down somewhere in the event this day ever came.

 

So to sum it up...without even looking, I am sure this would require a near overhaul of the logcollector operation, but if not even better

 

 

 

Jeremy

03-26-2015 02:00 AM

Best route, IMO, without more research into the logcollector functionality in place(will post more necessary if need be) - currently:

 

-- Use and improve upon the existing 'Collect Logs' functionality found in the logs.

 

When I say improve - I mean:

               a. Add the functionality Julia originally posted, ability to filter by date(s)/time(s), leaving the rest behind reducing archive size.

               b. Add additional functionality, eg; Ability to specifically select the component(s)(including agents) to archive logs from, archive type, preset collection profiles, etc, etc(I have a l

                              1a. Ability to select components to collect logs from.

                              2a. Provide alternate methods for collecting the logs outside a .zip file the default installation directory of your web browser(eg; alternate location, eg; /tmp/logs.tar.gz on the NAC) This would eliminate the issue of log archives not making it to the local machine attempting the collection

                              3a. Ability to save commonly used scenarios that apply for specific log collection with your current issue, as 'profiles' or something similar.

                              4a. Ability to switch log4j log levels for components on the fly before log collection, have these reset to previous values upon completion(no restart should be required)

                              5a. To extend upon what kogantivigay just said, address this by allowing the user to specify TZ according to UTC offset for each component that is different from NAC, do the math, and grab the right log sections.

                              6a. I have a few more if anyone is interested, but I could go on here forever...filters are our friends

 

2. Correct a lot of the bugs we encounter with this so it is more reliable(about 60-70% of the time it works currently) - a couple things I run into constantly-

                              1a. Sometimes the logs make it to the NAC, but the archive is not transferred to the local machine for whatever reason(quite a few things, a couple out of our control);

                              2a. Just doesn't work at all, usually due to the configuration used being far from default, which is expected(eg; NES(s)/NAC configured to use 8443/https, etc)

                              3a. Sometimes the log archive will be missing logs due to 2a, or other reasons I can list later, but this would coincide with 2a. in the first section.

                              4a. I have more I wrote down somewhere in the event this day ever came.

 

So to sum it up...without even looking, I am sure this would require a near overhaul of the logcollector operation, but if not even better

 

 

 

Jeremy 

03-24-2015 11:44 AM

You need to also consider the fact that NAG/NES/NAC are spread across different regions so different timezones otherwise you end up collecting wrong logs

03-04-2015 04:28 AM

Yes, I see this as a must have. I work with another piece of comercial software that has exactly this feature. There is a GUI on which you select the log "types" (one type of log can be spread into several distinct files), then press a button and it generates a tarball containing all of the required files one can then easily send to support. So the process is:

1. set loglevel to debug

2. reproduce the behaviour producing the error

3. grab the tarball

4. reset loglevel to defaut.

I find this very convenient.

My 2 cents worth.

Bernard

02-13-2015 03:35 PM

Definitely a great feature/idea on paper - It may be difficult to implement from my understanding of the product, but would be well worth it... I can't really complain too much on the current logging facility as it typically will catch most everything needed with default log4j settings, but granularity would be a godsend here   Awesome idea Julia..as always.

12-02-2014 12:40 PM

I totally agree that Ideas are the best approach for ER's due to the increased visibility, and generally I would argue should be the single approach but in this case I had a casual conversation on a related subject and shared my thoughts but didn't follow up with a formal ER. My bad.

12-01-2014 05:37 PM

That works too - Ideas are not the only channel to get product enhancements through, although they do benefit from being visible to everyone.


Thanks, Kyle_R.

12-01-2014 04:48 PM

I discussed my idea with the other product managers that are responsible for that aspect of the product, but I haven't created an Idea.

11-26-2014 01:40 AM

Hello Mark,

 

If the suggestion went through as an Idea here on Communities, provide the link so that we can go and vote on it, as it is a different - good - idea from what is posted here about date restriction.

 

Thanks, Kyle_R.

11-20-2014 10:57 AM

I've previously suggested a feature that would automate the collection of diagnostics and usage. Many products have an opt-in to provide anonymized usage information, so that we can better understand how the product is being used, and to provide diagnostics. This would not only greatly improve the supportability, but also provide metrics to help drive prioritization of feature development. The logs are only one aspect.

07-30-2014 10:16 AM

I also very much support this idea. We often get tons of logs uploaded that we have to dig through to find the relevant time period, especially given that many of the logs are irrelevant given they are from other days/months entirely.

07-30-2014 03:11 AM

I like the idea, and not just for Release Automation but across products. "CA Remote Engineer" for example is an interface which collects "log sets" for multiple products.

 

My first thought was to suggest gathering along these lines:

     * From date/time - could map to the "created date/time" of the log file OR to an offset relative to the "End time."

     * End date/time - could map to the "last modified date/time" of the log file.

 

As it is far easier to build a script to check these two meta-tags than to code opening each one of a large number of disparate files and searching for genuine first/last date/timestamps in all of their different formats.

 

My second thought is that some files can be created well before an event takes place, and that the last modified date can be well after the event. So the user has to be either aware if their log sets are likely to do that and leave a bigger date range, or there does have to be some type of coding nightmare in order to open the file contents and look for strings.

 

So my final thought is that it would be good if all log files were either created in a format that can be opened with an Editor (like Microsoft Event Viewer for Windows) with an Export function (which would allow easy date ranges and filtering), or we need a third tool in the middle to query the logs first (as they're not all "CA" logs), compare against templates and then export as desired. Best would be both - but at the cost of us maintaining this viewer to account for third party logging changes. Cross company working group for all products, anyone?

 

Still, I like the original statement of the issue, as log sets can be needlessly big and may need manual gathering to narrow them.

 

Kyle_R.

07-28-2014 03:44 PM

And off-course if we can have something like mentioned in original idea its best then anything else.

07-28-2014 03:41 PM

I like the idea, but IMHO it will be tricky as our logs are scattered in mulitple files and to trace and yield result between specific time frame will be slightly tricky. But I will like to have a feature by which at least the old logs later then issue seen date can be filtered out.