Upgrade to 9.6 -- Slow after Upgrade

[Legacy User]

Currently, we have 4 collectors and a Mom. We upgraded from 9.1.7 to 9.6.

 

After the upgrade, using both workstation and the new webview, everything is consistently slow, when changing the Time periods.  ie....going from live to 7 days, or live to 30 days. I did a timing, going down the Time Range,  up to 24 hours (3 seconds), at 48 hours you start to see the delay (15 seconds), 7 days (3 minutes), 30 days is longer either crashes or can take over 5 minutes. fyi....no

 

Prior to the upgrade, there was no slowness with changing to longer time periods. I checked and my memory settings are the same as the pervious version on everything. Also, I'm the only one logged in currently.

 

Has anyone experience this?

 

While I was writing this, I did a 30 day Time Range, and Workstation just died. ie.....went back to login.

 

JKW

[Legacy User]

So, a little more info.  I have two customers that I upgraded to 9.6. (one from 9.1.7, the other from 9.5.3), both are experiencing the same issue. The response back can be up to 5 minutes or hang the console (and the Investigator or Management Modules, associated with the Workstation that's opened up)

 

I tested on the 9.5.3 installation that I have not installed yet, and it takes about 8 to 10 seconds to show the 30 days....normal

 

So, I now tried webview, put it on 30 days....webview then says "The server is requiring you to restart your session", and requires you to log back in.

 

Something is definitely wrong with a 9.6 upgrade from a previous version. I may try a fresh installation, import the smarstor data, and then try again. I'm wondering if the upgrade is having problems with the 9.6 upgrade. Will also try to put the Collectors and EM in debug mode for logging.

[Haruhiko Davis]

Did you an in-place upgrade or did you do a parallel installation and move configs over? If you haven't tried the parallel installation, please try that and let us know the behavior.

[Legacy User]

Thank you for posting this information; we've been planning a test of the new version ourselves (upgrading from 9.5.1.0). Please keep us apprised of your experience - it's extremely helpful.

[Legacy User]

Morning,

 

I wanted to update this thread. I've opened a support call with CA and provided debug and install logs on the Mom and Collectors. Still waiting on a solution. During that timeframe, I found out some additional information:

 

1) If I try to do a 30 day Time Range with 9.6 workstation, it closes all my windows and goes back to login.

 

2) Interesting enough, I tried the 9.5.3 workstation, and it does not hang or close all the windows. It allows me to keep working and bring up other windows. It also does sometimes show the 30 days of data. It's interminnent. Seemed like after a restart of the Mom and Collectors. 30 days showed up in 10 seconds, then I logged in and out and it started slowing down again. (But no crashing of workstation with the 9.5.3 workstation)

 

3) Looked at the logs in debug mode. One thing that I do see is that the problem may be communication with the EM's and pulling 30 days of data. The MoM does say waiting on response from Collectors on the request. However, no limit appears to be hit in the log files. What's interesting is that the logs finally indicate that the metrics have been received, but the poor 9.6 Workstation just sits there in a hung state, until it finally closes and goes back to the login.

 

So in SUMMARY:

 

• Workstation 9.5.3 with APM Introscope Version 9.6, is more stable than Workstation 9.6
• The issue may be related to communication of data between the Workstation > MoM > Collectors
  

I did verify that my old tuning parameters were still on the system. (See 9.6 EM Performance Tips).

 

I will update this as we get further with CA.

 

JKW

[Anon Anon]

Hi,

It's definitely not a generic problem as we test different upgrade scenarios internally, seems like something specific to your installation and as suggested above it would be important to know how upgrade was performed and secondly if correct OSGI jars were deployed.I would recommend opening a support issue while original issue is reproducible with following details

-Debug MOM\Collector logs

-Thread dumps from all EM's while slowness\hang is observed

-Thread dumps from Workstation client if possible

-Installation log files (<APM_Home>/install folder)

so we can help you further.

 

Regards,

Kulbir.

[Legacy User]

It's not specific to an installation. Two different customers are seeing this, each one with different installations. The only commonality is that I did an in-place upgrade. One customer has been upgraded from 9.1.4 > 9.1.7  > 9.6, and I see the same problem. The other customer 9.5.3 > 9.6, see the same problem.  Yes, I used the 9.6 osgi files. (osgiPackages.v9.6.0.0.unix.tar).

 

I will open up a service desk ticket as well.

 

JKW

[Legacy User]

Status - 8/25/2014

 

I wanted to give an update status to anyone following this thread. I opened up a Service Ticket with CA and provided debug logs, but I'm still waiting.

 

Over the Weekend, I tried a new installation, but used the same DB files. I installed 9.6, new installation. I then modified 4 config files, domains.xml, users.xml, realms.xml, server.xml. The rest are mostly out of the box. I did modify the "lax" files to increase jvm size. I did not deploy any of the customized modules from my previous installation. I used out-of-the-box CA Dashboards. I concentrated on the SOA Performance Overview Dashboard for my testing.

 

The new installation still used the same db's.....ie.....SmartStor, APM DB, Baseline, Trace....

 

Problem still the same. 7 days, sometimes I get the metrics, sometimes I don't, 30 days can crash Workstation back to the login prompt. So, a new installation, with same db files did not work.

 

I also did some testing basing a comparison of our 9.1.7 installation in Production, which has far more metrics than the development environment. I tried using both Workstation 9.6 and Workstation 9.5.3, with the 9.6 installation. The results are below.

 

------------------------Base Comparison -- Production using 9.1.7 Workstation ------------------------

 

 

Production -- Version 9.1.7 -- SOA Performance Overview Dashboard
      Workstation...9.1.7  (Logged into MoM)

      Time Range 7 Days......3 seconds
      Time Range 30 Days......10 seconds

          ----- Has more Data than DEV. Same Physical Configuration as DEV. (memory,cpu,disk)

---------------------Using 9.6 Workstation ---------------------------------------

DEV -- Version 9.6 --- SOA Performance Overview Dashboard
         Workstation 9.6  (Logged into Collector 01, fewer agents)

         Time Range 7 Days.......1.5 minutes

          Time Range 30 Days..... 4 minutes

          Workstation 9.6 (Logged into MoM)
           Time Range 7 Days...... Crashes Workstation back to Login Prompt (Sometimes it doesn't. but doesn't display all the data)

           Time Range 30 Days...... Crashes Workstation back to Login prompt

------------------------- Using 9.5.3 Workstation --------------------------------

 

          DEV -- Version 9.6 -- SOA Performance Overview Dashboard

          Workstation 9.5.3 (Logged into Collector 01, fewer agents)

             Time Range 7 Days......1 minute...all data displayed

             Time Range 30 Days....3 minutes ... all data displayed

          Workstation 9.5.3 (Logged into MoM)

             Time Range 7 Days.....2 minutes, but partial data display, two of the graphs never updated after 10 mins.

             Time Range 30 Days....After 10 minutes....nothing...didn't wait longer....but no crashes

 

SUMMARY: Whether using a new or upgraded 9.6 installation, looking at historical metrics is extremely slow, past 7 days, it hangs or does not display in the Workstation, at 30 days, the Workstation Crashes.

[Legacy User]

Status Update -- 8/27

 

CA is working on the problem. We provided our SmartStor Database and our Home Introscope directory. Even going to one collector with Workstation is slow. We are using the SOA Performance -- Overview Dashboard for testing. the Out-of-the-box Dashboard.

 

JKW

[Legacy User]

CA has replicated the problem with our installation and smarstor data. They are still researching a resolution. We have halted upgrades to our two Prod Environments, until resolved.

The problem exhibits itself on viewing longer term data in the Console. Viewing Data from the MoM over 7 days (tried 30 days), either doesn't display all the data or crashes workstation. The problem also exhibits itself if you login to just a Collector, view the Data in the Console over 7 or 30 days...It does eventually appear (logged into a collector only) but takes 2 to 4 minutes to display.  (In comparison, version 9.1.7 took 3 to 6 seconds connected to a Collector and on a 9.1.7 Mom....12 to 30 seconds for 7 or 30 days)

 

I will let the group know, once this is resolved. If anyone else is experiencing slowness with 9.6 upgrades, please comment on this thread. This would help to understand if this is a larger problem.

 

Thanks,

JKW

[Legacy User]

Thank you for keeping us apprised!

[Billy Cole]

JKW,

 

We have upgraded our test environment to 9.6 and are seeing similar behavior with webstart and webview workstations.  Our 9.1.1.1 production environment returns in 10 to 20 seconds for 7 to 30 day worth of data.  With our production environment, I am using an installed Introscope workstation with 1 gig of memory.

 

Our test environment isn't resourced (cpu/memory) as our non-production or production environments.

 

If we use webview/webstart 9.6 with a lower historic window than 7 days, the data will return within 20 seconds and populate all five base graphs.  With 7 and 30 day windows, one or two of the five base graphs will populate or none at all.

 

Could you share the CA support case number so I can reference it?

 

Thank you,

 

Billy

[Legacy User]

Billy,

 

The case number is:

 

CA Support Request 21907947 01 - SLOW AFTER 9.6 UPGRADE

 

Thanks, JKW

[Legacy User]

CA is providing a fix to the problem. They provided some files to resolve the problem, and they are now putting together a fixpack for this. Another interesting item, Workstation needs additional memory. After installing Workstation 9.6 on my desktop, it occasionally crashed, out of memory. With this new release, we had to increase the JVM to 1GB.  Also, one other item, if your system does not have enough free memory, workstation will not come up, and you see no errors.

 

These are the files that were updated:

EM's -- product/enterprisemanager/plugins/com.wily.introscope.em_9.6.0.jar

            product/enterprisemanager/plugins/com.wily.introscope.em.client_9.6.0.jar

 

 

 

Workstation -- product/workstation/plugins/com.wily.introscope.ui_9.6.0.jar

                        product/workstation/plugins/com.wily.introscope.em.client_9.6.0.jar

 

 

 

Modify the Introscope_Workstation.lax --  Increase JVM to 1gb  (-Xms1024m -Xmx1024m)

 

We are still waiting on the Fixpack to resolve a couple other issues. CA is working on packaging the fixpack now.  Webview still has the same problem, slow with historical data over 7 and 30 days.  Also, the Workstation WebStart  (link on the 8081 URL), needs to be updated with the fix as well. The fix for these will be in the Fixpack.

 

CA recommended that each customer that experiences this problem, open a ticket, and request the fixpack.

 

I will do a final update, once we receive the final fixpack.

 

JKW

[Steve_Oak]

We had simialar issues that started in 9.5.2 upgrade...

CA Support Request 21858940 01 - PROBLEMS CONNECTING.

 

We are seeing the MoM complaining of slowness of the Collectors and disconneting them while the Collectors seem to be oblivous to the situation and seem to be just working taking agents except reporting that the MoM disconnected them

 

9.6 actually helped from what we were experiencing in 9.5.2.

 

I have sent my support people your ticket number since they have updated different files...our updates have centered around:

 

<MOMhome>/ws-plugins/com.wily.introscope.soa.dependencymap.ui.9.6.0.jar

 

Currently I can get 3-4 good days before having to restart the cluster...

 

We use WebStart and some WebView, I have my WebStart heap set at 512 min and 800 max (I have start up issues if I go much beyond the 800 with some users)

 

I just restarted the cluster (MoM and 3 Collectors) and I get almost instant results when going from live to 7 Days and to 30 days...but in a couple of days it will slow down and finally not even open the agent tree...restart the cluster and we are back flying again...

 

Steve

[Anon Anon]

I´ve seen a similar behaviour in version 9.1 two years ago, I guess.I did a in-place upgrade and after that the search was very slow. I enabled debug logging in the Introscope client and saw that that it was not possible for the client to connect to the "Custom Business Application Agent (Virtual)" and so the client suffers connecting after a timeout (60 (?) seconds for each of these Custom Business Application Agents). After installing a fresh MOM and EMs the problem disappeared.

My conclusion was to reinstall the MOM and the EMs completely new in an update case and to avoid in-place upgrade.

[Legacy User]

Jankei,

Unfortunately, In this case, a completely new install would not work to resolve this issue. I actually did a complete new install to see if that would resolve the problem, but it did not. Of course the problem is related to the amount of Data in the Smartstor, so I used our Smartstor. So, a new install might have worked, until the SmartStor(s) accumulated over 7 to 30 days of data, and then the problem would be seen. The fixpack that CA is working on will resolve the problem.

 

I'll update the thread again, when I get the complete fix pack. We still have issues in Webview and Workstation WebStart (Data Slow over 7 days), but the fixpack they are working on will fix all the EM's, Workstation, and Webstart.

 

JKW

[Anon Anon]

Good information. I'm planning to do an in-place upgrade from 9.1.6 to 9.6 shortly...

[Legacy User]

The problem has been resolved. We received Hotfix "323483_916Hotfix3.zip" from CA, which resolved the problem. We are now able to view 7 days and 30 days, with no slowness. (7 days took about 3 seconds, 30 days took about 10 seconds).

 

Workstation webstart works great as does webview. Listed below are the steps provided to us by CA.

 

**** One thing to remember is that if you install workstation 9.6 on your desktop, you will need to patch it as well.

 

**** According to the CA Support staff that worked with us, you will need to open a ticket with CA to get the patch.

 

The following are the instructions from CA that we received:

-----------------------------------------------------------------------------------------------------------

Download hotfix:  323483_916Hotfix3.zip

Unzip the file and deploy the files.

Here is the deployment instructions.

- Copy the following jars to EM (product\enterprisemanager\plugins):

com.wily.introscope.em_9.6.0.jar

com.wily.introscope.em.client_9.6.0.jar

 

- Copy the following jars to WS (product\product\workstation\plugins), for

both webstart and standalone WS:

com.wily.introscope.em.client_9.6.0.jar

com.wily.introscope.ui_9.6.0.jar

com.wily.introscope.workstation_9.6.0.jar

 

- Copy the following jars to WV (product\webview\plugins):

com.wily.introscope.em.client_9.6.0.jar

com.wily.introscope.ui_9.6.0.jar

com.ca.apm.introscope.workstation_9.6.0.jar

 

Note: com.wily.introscope.workstation_9.6.0.jar and

com.ca.apm.introscope.workstation_9.6.0.jar are included for versioning

purpose only.

[Chris_Kline]

This has now been documented and posted as a technical advisory, with the hotfix available for download without opening a support ticket.

 

http://www.ca.com/us/support/ca-support-online/product-content/status/release-notes/product-technical-advisories.aspx

 

Cheers,

 

Chris Kline

APM Product Management