Apache Tomcat Log Analysis in v14.1

[Georgy]

Hi Friends,

 

We completed a clarity upgrade from v13.0 to v14.1. So glad to see all those smart sections working like expand/collapse switches. We were also excited about the new performance analyze feature added for Tomcat users in v14.1. CA has added 5 new tables and 2 jobs for this new feature, as shown below:

 

■ LOG_DETAILS

■ LOG_FILES

■ LOG_REPORTDEFS

■ LOG_SESSIONS

■ LOG_SUMMARY

and the 2 jobs being:

1) Tomcat Access Log Import/Analyze Job

2) Delete Log Analysis Data Job

 

Two days after the upgrade (once we were sure that we got some access logs to analyze the performance), we ran the first job to import logs and fill the above log tables. But to our surprise, only the LOG_SESSIONS and LOG_REPORTDEFS tables are being populated. No data in the other 3 tables

 

Did we miss anything here? Any other configuration to be done to activate this performance analysis feature? Please let us know if you were able to populate the other log tables as well.

 

P.S. The upgrade to v14.1 was so smooth. Thanks CA

 

Regards,

Georgy

[Anon Anon]

You are way too ahead buddy.......

[Suman Pramanik]

Hi Georgy,

 

Yes we added few tables for detailed log analysis and we will be soon shipping a portlet package which you can install using content addin and use. So watch out the community space

Regards

Suman Pramanik

[Georgy]

Hi Suman,

 

Thanks a lot for letting us know about the upcoming portlet package. I guess this future portlet also will be looking in the above newly added log tables for displaying the log analysis data, rt?

 

Our issue here is that even after runing Tomcat Access Log Import/Analyze job, these 3 tables (LOG_DETAILS, LOG_FILES, LOG_SUMMARY) are empty. Is there anything else to be done other than running this job?

 

Regards,

Georgy

[Suman Pramanik]

Hi Georgy,

 

Yes those portlet will pull the data from the tables mentioned,

 

Tomcat Access Log Import/Analyze

• §Job Function

–This job imports the tomcat access logs into the Clarity db schema.

• §Job Specifications

–Takes an optional date argument specifying which date to retrieve log files for

–Date defaults to previous day if not specified.

–Dynamically discovers the app services within the Clarity cluster and fetches/imports tomcat access logs for those services.

–Inserts detailed information and summary information into the Log Analysis db schema tables

• §Recommended Configuration

–Schedule after midnight in the same timezone as the app servers.

–This will automatically retrieve/import the previous days logs just after Tomcat stops writing to them at the end of the day.

Hope this will help

Regards

Suman Pramanik

[Georgy]

Hi Suman,

 

Yes, I also thought the same. As you said, this job should be inserting detailed and summary information into the log analysis db schema tables which its not doing now

 

Regards,

Georgy

[Suman Pramanik]

Hi Georgy,

 

 

This job design is to externally bulk import logs from other Clarity environments , so i believe it will populate when you import the logs from other environment.

 

 

However i will again test myself and will let you know.

 

 

Regards

Suman Pramanik

[Georgy]

Hi Suman,

 

Great! thanks a lot for testing this. Will be waiting for your results.

 

Regards,

Georgy

[Suman Pramanik]

Hey Georgy,

 

The problem i had with my system was my beacon was not working properly, so i stopped the beacon, added again and started and it works like charm.

 

select * from LOG_FILES

app-access-2014-10-28.log

app

111

28-OCT-14

local

10.130.111.202

28-OCT-14

 

I ran only for today and it populated. Try the workaround and let me know.

 

Regards
Suman Pramanik

[Georgy]

Hi Suman,

 

wow, that worked  Just restarting the beacon didn't fix it. Also when we go to the new Process Monitor page in CSA, we had an error there suggesting us to put a proper multicast address and a bind address.

As all our DEV, UAT, TRAIN, STG systems were using the same default multicast address, among which only the DEV is in v14.1, I guess that caused this issue. We gave different values for the DEV multicast address and multicast port. Also gave the DEV server IP as the Bind Address in CSA as our DEV env only has one server, no cluster.

 

After all this, removing the beacon, adding it back again, starting it  fixed both the issues. The error in the Process Monitor page in CSA is gone and all the 3 log tables are now geting populated with data We ran the populate job two times giving yesterday and today as the Log Date parameter.

Thanks a lot for your time in fixing this issue...really appreciated.

 

Regards,

Georgy

[Suman Pramanik]

Hi Georgy,

 

You are welcome , Yes the log analysis & Process Engine depends on the correct configuration of beacon.

 

Regards
Suman Pramanik

[Damon Logiudice]

Wow Mr. Joseph,

Very impressive to see such a smooth upgrade to 14.1 just 3 weeks after the GA date for this new release of CA Clarity PPM. Says a lot about CA, the upgrade process, and the maturity/agility of both CA and your organization. Are you working on CA Clarity on-premise or on-demand (SaaS)? Also, are you supporting one large customer or several customer instances of CA PPM? Just want to get a sense of what your challenges are day to day.

 

Kind Regards,
Damon

CA Technologies

[Georgy]

Hi Damon,

 

Yea, the upgrade went smoothly. Happy to get this done so fast in our DEV environment . We are yet to plan our Production upgrade as we guess v14.2 will be out before the end of this year. Our customer has CA clarity installed on-premise and they only have one large instance of CA PPM.

 

Regards,

Georgy

[varve03]

Hi

I hope some one else already faced this issue and have some resolution in place.

We have a audit requirement to maintain all the logs of client level event / activities.

Apart from the Date, time, URL Actioned we need following information also.

1. User of Application / System

2. Source IP Address

3. Success or Failure of events

 

For item 3 we can use browser status, but we cant find Source IP address information. Is this possible ?

Also for User of Application / System Account, we thought we would use session_cookie info from LOG_DETAILS  and link with LOG_SESSIONS table to find the actual user id, but this field (sesion_cookie) is always blank.

Note: I am working on SaaS environment. CA PPM 14.4.0.234.

I did check with  another environment on 14.3.0.29, i don't have the same issue. Is this an issue in 14.4 ?

 

What are all the scenarios this session cookie is updated as "-"  ?

 

Any suggestion / advice please..

 

Regards

 

Venky