Important: Problems to Certificates on your CA Spectrum One Click Installs – Please Read

[kiran_diwakar]

Hello All,

 

I just wanted to reach out and ensure I give you heads up for a key issue that our engineers discovered yesterday!

The CA Spectrum OneClick Java JRE certificate will expire on Monday March 01 2015. This will impact all the customers who are currently on 9.2.x, 9.3.x, and 9.4.x using the supported versions of java.

Because of this, while all other components/servers of Spectrum will work appropriately, OneClick Client will not launch for all the users whose security setting on the OC Client box is set to HIGH or VERY HIGH in the Java Control Panel.

So you would need to apply the hot fix we are providing (plans to follow later in the day) or use the workaround to lower Java Control Panel security setting to medium on OneClick Client Box ONLY (other servers like OC Server, SpectroServer, Reporting etc can still remain on HIGH/VERYHIGH security)

I acknowledge the fact that given that the certificate will expire and the problem will manifest itself on March 1, we could have given you a bit more advanced notice – it was a late catch on our side

We are already working on ways to not let this happen in future at such short notice for us as well as for our users.

 

Having said that, the good news is that we have a workaround for this that the team has put together and our support team will publish by end of day today EST.

Additionally, we also have plans in place that will be published for fixes/solutions for the problem that we will be making available – and that you will need to move to. Please look out for that note/information.

 

As the Spectrum group, we have done a good job in the last couple years with our releases, quality and our general things and I know the expectations from us are high. Hence, I personally apologize for the problem and inconvenience that you will be going through – please rest assured that we are doing our best to give you the best and the easiest way out from the myriad of possibilities we went through. The team has been up since last night getting a solution and our support teams are trying to get as many scenarios covered in their docs and ensuring as much heads up as possible internally/externally – we have all hands on deck for this.

 

Please do not hesitate to reach out me or to your support contacts if you need more information or have any suggestions.

 

Regards,

Kiran Diwakar

[Anon Anon]

Thanks Kirwan for the update.

 

I have sent a note regarding using Java 6 and from my searching the setting is not apparent. It may be useful to the community if the situation with Java 6 can be clarified.

[kiran_diwakar]

Hi Ian,

 

Yep, I saw that and I see you are talking to the right people.

You will hear back from us very soon

 

Regards,

Kiran Diwakar

[Frank Daeter]

What about the other CA applications? (eHealth / SOI / ...)

[natma01]

A note with regard to all products: The workaround -- lowering the Java JRE security setting from High or Very High to Medium and using the exceptions list -- should work around this issue for most clients.

 

-Margaret, eHealth PM

[kiran_diwakar]

Hi Frank,

 

I am not sure about the other products. I don't believe there is an issue on their side, coz we would have heard. Anyway, I will check it and let this group know.

 

One more thing - the workaround is to lessen the Java security (through Java control panel) only on the box/system that has the OC client.

I am just trying to get a feel of how problematic would this be in your scenarios. I know there would be sensitive deployments - but given this is a client system (maybe outside of the NOC or Data Center), would this workaround only for client system be too difficult?

Again, as I said, we will be having solutions for all platforms/releases by end of this week, but just wanted to make sure I have a feel for the workaround - so comments from people would be much appreciated.

 

Regards,

Kiran Diwakar

[Legacy User]

All

 

One of my customers is getting this.

 

We removed cert checking and set security to lowest

 

Still no good

 

Windows 7

 

 

    <jar href="lib/contrib/clientcivpn

[baude02]

Hi Dan,

 

Have you tried adding the Spectrum OneClick server address to the exception list? Then there is  no need to change the security levels

 

Regards,

Dencel

[kiran_diwakar]

Hi Dan,

The certificate will expire on March 1st - so not sure how you are running into the problem now. Did you do something else?

If not, this could be a completely different problem and not related. Would request you to check with local support

 

Regards,

Kiran Diwakar

[Kristian Schuster]

Hi Kiran,

 

The problem with the workaround will be that most users - for security reasons - don't have the rights/ permission on their computers to change Java related settings (or other settings).

The Java settings on the Client PCs are configured via Group Policies and these Group Policies are company wide and have to match the company's Security Policy. I think it will be difficult to lower the Java security settings only for the group of Spectrum OneClick users and even more difficult to push through that the Java settings are changed for all users in the company.

 

Especially with the latest news in mind that big companies are more and more attacked by hackers, lowering security settings is a no-go for most of our customers.

 

Are there any news about the patches?

While we are running out of time, the patches need to be easy to install. Will it be possible to update the current servers easily? Any information is very welcome to prepare the customer's server for the upgrade procedure.

 

BTW: Best would be if the patch does not contain any further fixes, because there is no time left to test the patch in the customer's environment before installing. And negative impacts on the server due to fixes is not what we need now

 

Best regards

Kristian

[Stuart Weenig]

Agreed. We are in the same situation. Users don't have permission to perform the workaround so the patch is our only option.

[kiran_diwakar]

Hi Stuart,

 

Right – we absolutely could think of atleast a couple dozen customers we work with closely where we know the workaround will not fly.

As I said, the solution will be made available, this was a heads up for the team here to have a plan in place for this to be tried out.

Over the day today, Rene will send out our plan that we are fine tuning and it went through multiple iterations.

 

Please stay assured that we are pushing ourselves very hard to get this done in time for all of the customers. Thanks for the note and confirmation as well as the patience.

 

Regards,

Kiran Diwakar

Director, Product Management

Office: +78964 | Mobile: [phone] | [e-mail]

<mailto:[e-mail]>[CA][cid:image003.jpg@01CF96F2.15E42900]    <http://www.ca.com/us/default.aspx>

<http://twitter.com/CAInc>[LinkedIn]<http://www.linkedin.com/company/1372?goback=.cps_1244823420724_1>[Facebook]<https://www.facebook.com/CATechnologies>[YouTube]<http://www.youtube.com/user/catechnologies>[Google]<https://plus.google.com/CATechnologies>[Slideshare]<http://www.slideshare.net/cainc>

[Stuart Weenig]

No problem. I'm sure you're all working diligently to fix this recently

discovered problem. I work a 9/80 schedule, so I won't be here Friday and

I'm taking a personal day (planned since last week) tomorrow. Hopefully

something comes out today so I can get it patched.

 

On Wed, Feb 25, 2015 at 12:18 PM, kiran_diwakar <

[kiran_diwakar]

If you are on 9.4, we possibly could have something today.

 

Regards,

Kiran Diwakar

[Stuart Weenig]

We are on 9.4.0. 

[kiran_diwakar]

Hi Kristian,

 

Thanks so much for your response and we understand that it would be the case for many of our users.

So the solution/patches will be available by end of the week for most versions.

We have kept it absolutely minimal so that the changes in them are isolated to this certificate problem only.

 

The notes that we will release with the solution will have all the details. In the meantime, the idea was to have the workaround wherever applicable.

Please stay tuned and really appreciate all your patience with this.

 

Regards,

Kiran Diwakar

Director, Product Management

Office: +78964 | Mobile: [phone] | [e-mail]

<mailto:[e-mail]>[CA][cid:image003.jpg@01CF96F2.15E42900]    <http://www.ca.com/us/default.aspx>

<http://twitter.com/CAInc>[LinkedIn]<http://www.linkedin.com/company/1372?goback=.cps_1244823420724_1>[Facebook]<https://www.facebook.com/CATechnologies>[YouTube]<http://www.youtube.com/user/catechnologies>[Google]<https://plus.google.com/CATechnologies>[Slideshare]<http://www.slideshare.net/cainc>

[Andre Wawrzyniak]

I have checked the certificates in SOI.

SOI 3,1

[From: Thu Feb 28 01:00:00 CET 2013  To: Mon Mar 02 00:59:59 CET 2015]

 

SOI 3.3

[From: Thu Feb 28 01:00:00 CET 2013,  To: Mon Mar 02 00:59:59 CET 2015] &

[From: Thu Feb 28 01:00:00 CET 2013, To: Mon Mar 02 00:59:59 CET 2015]

 

Am I correct that, SOI will be affected by this problem if the Java security settings are higher than medium on Mar 02 00:59:59 CET 2015?

 

regards

 

Andre

[kiran_diwakar]

Hi Andre,

 

The SOI team is also checking and let the community know.

I am getting the SOI key product folks on this update

 

Regards,

Kiran Diwakar

[Anon Anon]

Kiran, thank you for notifying the community!

[sivajp]

Hi Kiran,

 

Thanks a lot for the notice in the community.

[Anon Anon]

Is a mass-mail planned (similar to how the heartbleed email was delivered for eHealth) for when the hotfix(es) becomes available? Or is best to keep an eye on this thread for the release announcement?

 

Thanks for the heads-up.

[Rene Cantwell]

We are in the process of using a new notification system that will use your email addresses if you have opened a support issue with use within the last 12 months.

 

In addition the same advisory will be posted to the Spectrum product page and this community.

 

--

Rene’

 

Rene’ Cantwell

CA Technologies

Support Delivery Manager

603-334-2497

[e-mail]<mailto:[e-mail]>

[Anon Anon]

Thank you for the insight!

[Justin_Kulikowski]

Is there a way for us to reproduce this issue ahead of March 1?

 

I would like to see it break when set to High/Very High, and then work when set to Medium -- just to validate the workaround.

[Rene Cantwell]

Yes, you can change the system clock on the client box to a date after March 1st.

 

The recommended workaround will be to use the exception list over changing the security setting.  You may want to try them both.

 

 

 

--

Rene’

 

Rene’ Cantwell

CA Technologies

Support Delivery Manager

603-334-2497

[e-mail]<mailto:[e-mail]>

[Anon Anon]

Anxiously awaiting for what fix will be.

Don't want to speculate here, but I'm hoping fix is in the form PTfs with updated JAR files, or simply plain jar files.

The new jars would simply have updated signature certs (btw, not clear to me if those are ones affected).

Changing security settings or updating Exceptions list will not work for many users .

This is not just due PC admin rights but also to new JRE security restrictions, that will simply ignore those settings.

[Rene Cantwell]

The details are available in the Advisory which is now available from our Product Page News area or this link:

https://support.ca.com/phpdocs/7/7832/7832_certificate-expiry_product-advisory.pdf

 

One additional note to the workarounds recommended.  If the Exception Site List (workaround #1) is going to be used, you will have to add your BOXI server if on a different box than your OneClick Sever.  The entry will need the port number (at least on Linux) i.e. http://hostname:8088

[Anon Anon]

Rene,

 

Thanks for the Advisory. After seeing the workaround, I think this issue is being caused by JRE Security changes which were introduced by Oracle with release of JRE 1.7_60ish, which changed the security from Medium to High. Users running OneClick server which has a lower JRE requirement and client machines running JRE release lower than 1.7_60ish might get away by just accepting a warning message, till the point they are able to apply new hotfixes. In our company we were aware of this JRE changes from couple of months and have already put our OneClick and CABI servers URL's in the exception list.

 

Our Spectrum environment is 9.3 H01 on RHEL which has JRE requirement of 1.7_u51. Client computers run JRE 1.7u76 with Security set to High. I tested the certificate expiry scenario by changing the date on a client PC to March 2nd and with exception list in place there was no impact on launching OneClick. I just had to acknowledge that the certificate is Expired.

 

Hope this helps some of the customers!!!

 

[Anon Anon]

An additional problem that you may run into in trying to use the Exception List is if your site uses Deployment Rule sets. This is a JAR file, typically in C:\Windows\Sun|Java\Deployment\DeploymentRuleSet.jar which is distributed within an organization. If your Java control panel has "View the active Deployment Rule Set" as shown, you're one of the affected ones. The problem is (as Oracle states), that anything added to the Exception Site List is ignored.

 

https://www.java.com/en/download/faq/exception_sitelist.xml

 

Deployment Rule Set

If an active deployment rule set is installed on the system, the deployment rules take
precedence over the exception site list. The exception site list is
considered only when the default rule applies. See Deployment
Rule Set for more information about deployment rules.

[kiran_diwakar]

Hello Al,

 

Thanks – that is very good information.

I have asked the engineers to review it and check if we can add it to our advisory/document.

Thanks again!

 

Regards,

Kiran Diwakar

[RobertHermann]

Anyone able to reproduce the situation after 1st march today by changing the system time?

 

I tried on a win7 client box:

- shut down windows time service

- set date to 5th of march

- set java sec to high

- start oneclick > oneclick starts .. no problems about expired certitficate

 

Spectrum 9.3

[kiran_diwakar]

Hi Saurabh,

Thanks for this information, it would be definitely beneficial for atleast some of the customers. Thanks.

 

Regards,

Kiran Diwakar

[kiran_diwakar]

Hello All,

 

Hopefully everyone is able to review the product advisory that was posted yesterday.

As you see we will have solutions (HFs) in place and the one for 9.4.x is already available.

I know many of you wanted to solution to be isolated to this problem and we tried our best.

But given the solution is a cumulative (hotfix), we wanted to make sure that the critical patches that people have applied to their current environments, they do not miss those or lose those.

So after heavy deliberation, we decided to merge a few critical fixes as part of this cumulative hotfix as well. The details are in the product advisory.

As we mentioned earlier, we tried to keep as minimum as possible to not only get you over the hump of this security issue, but also keep you current with respect to the key patches that you might have already applied.

 

I request you to please review the product advisory in detail and if you are on 9.4.x to take advantage of the available solution as soon as possible. Additionally, we would request you to review the potential workarounds that might be applicable and allowed in your organizations so that you could potentially take care of the problem before it happens.

 

Another thing that aggravates this is the fact that March 1 is a Sunday and not everyone might NOT be in office. This could lead to calls/pages for you and/or your teams over the weekend - would request you to consider that scenario and plan accordingly as well.

On our end we are having our on call engineers be aware of it, and putting contacts in place for specific scenarios.

 

As always, comments/suggestions very welcome.

I would definitely want to call out our appreciation for this group for your support, patience and understanding in this case - it has helped us stay focused on the problem at hand a lot better!! Thanks once again.

Regards,

Kiran Diwakar

[Torsten Syring]

Hello all,

 

maybe a stupid question.

We already upgraded an environment with 14 SpectroServers and two OC servers to 9.4.2

I know that in Fixed_Issues.09.04.02.01.htm there is the statement:"The 09.04.02.01 OneClick is only compatible with the 09.04.02.01 SpectroSERVER"

but in our opinion it is sufficient enough to upgrade only the OC servers to 9.4.2.1 because according to the release notes nothing changed besides the signed jars.

We also not able to upgrade the 14 servers in time.

So any comments about that?

 

regards

Torsten

[Rene Cantwell]

Torsten,  I'll check into this for you.

 

--

Rene'

[Skip.Taylor]

Any update on this?  We also upgraded to 9.4.2 yesterday.  Can we apply this to our One Click Servers only or will we have to apply to all Spectrum servers?

[kiran_diwakar]

Hi Skip,

 

If you have moved to 9.4.2, then it is easier for you.

9.4.1 would be having only the security changes.

Yes, it would need to applied to all servers – as it is a hot fix

 

Please let me know if you need any clarifications or help

 

Regards,

Kiran Diwakar

[Skip.Taylor]

Kiran,

How is it easier if I am already at 9.4.2?   It appears to me this is an upgrade as opposed to a hotfix which means I will have to follow the same steps as an upgrade.  I have 8 servers to update and it took nearly 5 hours to accomplish

[kiran_diwakar]

Hi Skip,

 

I am sorry, what I was intending to say is that given you are on 9.4.2, there are no additional changes in 9.4.2.1 – so the risk and tests on your part would be minimal.

But yes, you would need to update and run the installer on all the systems and I am extremely sorry for the additional work you would need to do here.

This would be the case for all the customers who would want to use the patch and not the workaround.

The reason is that the certificate needs to be updated for over 200 jar files and there is no way the patch mechanism allows us to do that.

We were trying VERY HARD to try and get a PTF instead of a hot fix, but given the above we could not create a patch and hotfix was the only way.

We understand it is extra work – but be assured we really tried all other possible solutions.

 

For 9.3 & 9.2 as well we are trying to get as minimum changes in the HF as possible.

While 9.3 solutions will be available by today or tomorrow, please refer the documentation. Our goal was not to get you to lose the critical patches you would have already applied that we would have given.

We intend to get the 9.2 solution out by end of the week – over the weekend. I will send a detailed note for 9.2 solution.

Please stay tuned

 

Once again – I would want to earnestly remind and request you to create awareness within your users/operators about this impending problem so that there is no panic and additional frustration in your teams.

Please do not hesitate to reach out to us if you need anything else

 

Regards,

Kiran Diwakar

[Skip.Taylor]

Thanks for the explanation.  Looks like I will be busy tomorrow morning. 

[baude02]

Hi All,

 

Please be aware that only Spectrum versions 9.2.3 H12 and onwards (Spectrum 9.3.x and Spectrum 9.4.x)  are affected by this certificate expiring.

All previous versions are running on an expired certificate that had already expired in 2011. Java 1.6 also does not have the same security settings as Java 1.7 and will not lock you out, but only brings up a pop-up message.

 

Please contact CA Support if you are not sure about the version you are using. The details are in the $SPECROOT/Install-Tools/.history file

[Anon Anon]

We are on 9.2 Dencel and confirmed this today that we only get the pop up message.

[kiran_diwakar]

So you don't have an impact, right Ian?

Regards,

Kiran Diwakar

[Anon Anon]

Just completing some testing but looks like we get the pop up but can log on , so just a cosmetic rather than operational impact.

 

We have to upgrade to 9.4 soon anyway so if just that we can live with it for a few weeks.

[yonghoe]

hi if we are using the SOI conenctor with the SSL cert from oneClick console will this cert expiry cause any issue.

 

 

YH

[kiran_diwakar]

Hi Folks,

 

Our SOI teams are evaluating the impact of this and whether there is a problem. You should be hearing from that team soon, if there is a problem.

One update, we do have an issue for eHealth as well The eHealth dev team is working diligently to get back to the customers with the plan around it. Stay tuned.

Regards,

Kiran Diwakar

[Anon Anon]

Hi Kirwan, any more information on the issue with ehealth because it is coming up to month end reporting for us which is serious for the customer.

[kiran_diwakar]

Hi Ian,

 

The eHealth Dev team is working on a plan and hope to get out some advise hopefully today.

Regards,

Kiran Diwakar

[natma01]

WITH REGARD TO EHEALTH & LIVE HEALTH:

 

Yes the eHealth impact is smaller -- it appears to effect only the Live Health applications -- but we are testing similar workarounds and fixes. I will post a full statement with regard to impact, workarounds and fixes tomorrow, Friday, February 27, 2015.

 

-Margaret, eHealth PM

[kiran_diwakar]

Hi Ian,

 

Completely understood – I have notified the eHealth team.

You will hear an update hopefully today

 

Regards,

Kiran Diwakar

[kiran_diwakar]

Hi Robert,

Are you on 9.3 H01 or 9.3 H02..

 

Regards,

Kiran Diwakar

[RobertHermann]

Install-Tools\ # cat .history

9.2.0.0.327 11/09/2012 11:08

9.2.2.8.201 11/09/2012 11:47

Spectrum_09.02.02.PTF_9.2.211 installed on 11/06/2013 17:01:24.

9.3.0.0.304 06/11/2014 09:34

9.3.0.1.82 06/11/2014 09:52

Spectrum_09.03.00.PTF_H0112 installed on 10/28/2014 07:44:38.

Spectrum_09.03.00.PTF_H0113 installed on 10/28/2014 07:48:38.

 

Btw, I cannot use java control panel to config security settings. Because I am not using a system wide installed JRE on the win7 client. Instead I am using an absolute call to a javaws JRE folder(specific java version) which has in jre/lib a deployment.config and deployment.properties file.

 

content of deployment.config

deployment.system.config=file:///C:/path-to-jre/jre1.7.0_51/lib/deployment.properties

deployment.system.config.mandatory=true

 

content of deployment.properties

deployment.security.level=HIGH

 

I know these settings are applied because if i remove deployment.properties file, the oneclick startup complains about it is missing (this is because deployment.system.config.mandatory is set to true, so it has to be there).

 

Oneclick was in java cache before i made above settings.

[Legacy User]

Kirwan, I assume that since the patches are cumulative and effectively an upgrade they will need to be applied to all SpectroServers and the OneClick box?  Is that correct?  I did not see anything in the pdf referenced further up in this post that explained that.

[kiran_diwakar]

Hello bsiebert,

 

Right, it would need to e applied to the servers as well. The release notes will have the information. We will clarify it further

 

Regards,

Kiran Diwakar

[Rene Cantwell]

Update:  We have released 9.4.2.1.  It is available for download from both the support.ca.com Download Center and the Spectrum Solutions & Patches Index at CA Spectrum Infrastructure Manager r9.4 Solutions & Patches - CA Technologies. The ftp link in the advisory is also available.

[Anon Anon]

It looks like 9.3 H03 is up. Is this the final build? Haven't seen any official statement about it yet.

 

ftp://ftp.ca.com/pub/CA-SPECTRUM/Updates/GA/Spectrum_09.03.00.H03/

[Rene Cantwell]

Please do not grab this yet, we still have some validation steps to complete.  I’ll announce when we are done and it’s officially released.

 

Thanks,

Rene’

 

Rene’ Cantwell

CA Technologies

Support Delivery Manager

603-334-2497

[e-mail]<mailto:[e-mail]>

[Rene Cantwell]

Update:  We have released 9.3.0.H03.  It is available for download from the Spectrum Solutions & Patches Index at CA Spectrum Infrastructure Manager r9.3 Solutions & Patches - CA Technologies and the ftp link in the advisory is also available.

[kiran_diwakar]

Hello Team,

 

We have 9.3 and 9.4 Hot Fixes out there - I would request you to kindly share your feedback if you have tried any of those.

Your success with it will instill confidence in other users (and ofcourse in us as well) - so that people can be a bit more aggressive with their updates - using the real power of the community.

We have made very good progress with 9.2 H13 and I will have updates around that later in the day. Thanks once again all for the support and understanding with this one.

Regards,

Kiran Diwakar

[Skip.Taylor]

Installed 9.4.2.1 on 8 servers with no issues.

[kiran_diwakar]

Thanks Skip - any rough idea on how much time it took you? Thanks again!

Regards,

Kiran Diwakar

[Skip.Taylor]

Kiran,

it took me around 4 hour to complete the install.  The time includes backing up the databases on 3 Spectro servers.

[kiran_diwakar]

Hi Skip,

 

Thanks, appreciate your response.

The day was quiet for us - we were geared to run into situations and put in processes in place.

Very happy it was uneventful I would want to believe that for most the workaround and/or the patches worked well.

 

I will post a summary of this in the coming week.

Thanks everyone for your support and understanding.

 

Regards,

Kiran Diwakar

[kiran_diwakar]

                                            Important Announcement Related to 9.2.x

 

Hello All,

 

So here is my update on 9.2 H13 for this security problem.

First the critical (good) news – we will be releasing 9.2 H13 by tomorrow (28^th Feb) morning Eastern Time.
Again, while we acknowledge it would have been nicer to give more time, we have pulled this in by another couple days and most of you will have atleast some time to try it and consume it BEFORE the problem hits your environment.

 

An important point for all nonetheless – there were over 15 PTFs that were issued on 9.2 H12 – some of them critical PTFs. We consciously did not get all of these in H12 – reasons complexity of merging changes and the additional time thereof to test these. So it might happen that one of the PTFs you had for 9.2.x is not there in H13, but do not panic – we will be issuing PTFs on top of H13 - it is a planned drop, not a miss. Starting next week, through the week, we will be releasing PTFs ported onto the H13 stream.

 

To help you get clarity – please see the attached spreadsheet. The first tab “PTFs included in H13” is simple – all these are included in H13.

The second tab “PTFs Currently Not in H13” – are the ones that we will issue PTFs for starting next week.

We believe we would be able to cover all the PTFs in the second tab before the end of next week. I have retained the ticket numbers in the spreadsheet so that whoever opened that ticket is able to easily mark and assess the document.

 

Once again, the team is working 24x7 and trying to come up with creative ways to minimize the impact and effort to you all.

Please be in touch with your local support teams or your CA Account Teams. While I am around as well – given the nature of this problem our engineering, support, sustaining leaders are all available over the weekend – we are geared up and have structure in place with phone numbers of the right people with our guys front ending this. We do not anticipate any problems in your updates, given the HFs are as isolated as possible and the extensive tests we are trying.

 

Questions/concerns/critique very welcome.

 

Regards,

Kiran Diwakar

[e-mail]

[phone]

Attachment(s)

Spectrum_9-2_H13-PatchList_And_Details.xlsx   14 KB 1 version

[kiran_diwakar]

Folks, a small typo that was pointed out by my colleague:

 

"An important point for all nonetheless – there were over 15 PTFs that were issued on 9.2 H12 – some of them critical PTFs. We consciously did not get all of these in H12 – reasons complexity of merging changes and the additional time thereof to test these."

 

I meant to say we consciously did not get all of these in H13..

Pretty sure most of you would have got it, but still wanted to throw out the clarification.

 

Regards,

Kiran Diwakar

[Anon Anon]

9.3 Hotfix installed on 4 Spectroservers and 2 One Clicks. This took about 45 minutes running concurrently.

[Anon Anon]

9.3 H03 installed over 9.3H01 on Solaris with no problems. Running concurrently, we had 4 SS and 2 OC patched in about 45 minutes.

[Veronique Lefebure]

One question: If a OneClick session is already open now, will there be any problem on the 1st of march ? will it get closed or something ?

[kiran_diwakar]

Hi VLefebure,

Great point. Yes, we believe that would work.

Just to double check a few of our engineers are checking out as well.

Will confirm soon

Regards,

Kiran Diwakar

[kiran_diwakar]

                        Great Possible Workaround - Keep OC Client Open from Today....

 

Hello Team,

 

So our engineers confirmed - tried some quick tests and also verified some code segments.

As mentioned by VLefebure, you can keep OC Client open today and you can continue using it, as long as you don't close it

The certificate validation happens during OC Client launch only.

 

While not a permanent solution, this would be very valuable for folks to buy some time while the workaround and/or the HF is being applied.

Please let us know if you have any questions/comments.

 

Regards,

Kiran Diwakar

[kiran_diwakar]

Amazing, thanks Al.

Regards,

Kiran Diwakar

[Legacy User]

I've got to be honest here, I'm amazed and shocked at how big of a mess this is.  Less than 4 days notice (2 business days) for something that will break Spectrum for users?  I know things slip through, but this is going to cause major impact for us.

 

We have a large Spectrum environment.  We're an enterprise customer (as are most Spectrum users).  We can't implement an upgrade to our Spectrum infrastructure with 2 business days of notice.  Our change control requires more than that.  Our testing requires more than that.  Simple sanity and logic requires more than that.

 

I'm also very frustrated and upset that the "fix" is a full software upgrade.  I'd like to see this released as a minimal hotfix that just updates the certificates and could be applied to a OneClick server.  I've got 20 dedicated SpectroSERVER installs and a pair of OneClick servers, all with Fault-Tolerant backups.  Installing a minimal patch to the OneClick servers on such short notice would be a big enough pain in the ***, but doing an upgrade against our entire Spectrum infrastructure that requires going to a release that only went GA barely a week ago?  Seriously?  CA is not a mom-and-pop shop supporting off the shelf software; don't behave like you are.

 

The last two Spectrum upgrades we've performed have both hit bugs that required assistance form CA support to resolve and resulted in downtime and cleanup work.  The last upgraded took 9.5 hours, 7 of it with Support trying to resolve issues resulting from bugs.  We've still got 3 open cases (all at least 3-4+ weeks old) for unresolved issues from our last upgrade (to 9.4.1).  You'll have to forgive me if the prospect of doing another upgrade with no time to plan or test it leaves me a little pissed off.

 

As for your "work-around", our security policies don't allow Java security settings to be reduced below "High", and I'm not sure if non-admin users will even be able to add URLs to the exception list.  We may end up with a completely broken Spectrum infrastructure on Monday thanks to this debacle.

 

I feel like you guys dropped the ball on this big time, and your "solution" is just as big of a failure.

[kiran_diwakar]

Hello cpcashell,

 

First, I completely understand your frustration and trust me we are not very happy ourselves, but trying to find the best available path of the ones in sight.

I acknowledge the sentiment and perception of "you guys dropped the ball", I think from a end user and consumer perspective, and your expectations thereof your assessment of us as the product team - I accept that, and once again apologize the problem it is going to cause you over the weekend. There were many things that just came together to break this, including Java, but as I said, I take that critique and we assure you we have some introspection to be done.

 

Now coming to the problem - let me make our support teams aware of your situation. Can I please request you do the same. We will work with you through this.

Also, did you check the latest "workaround" of keep OC Client open and not closing it? That might help you buy sometime while you get through some of the hiccups.

 

Regards,

Kiran Diwakar

[Anon Anon]

We are on Spectrum 9.2.2.008 (seriously old, I know) and were planning to upgrade to 9.4.1 in a couple of phases (first H12, then to 9.4.1). We are actually fortunate to be on a release prior to H12, because our Spectrum clients are using Java v1.6 and the certificate is not the one expiring on March 1, 2015. They are using the cert that already expired on Nov 11, 2013. So our users have already clicked through warnings for that older cert and will not be affected on March 1st. We'll be installing H13 in the 2nd week of March.

 

However, our SOI v3.2 will be affected, as we confirmed it is using the cert expiring on March 1st. I don't use SOI much, but it appears only when the Console is loaded that Java is running. The web Dashboard/Administration runs without Java.

 

Hope this might be helpful to some folks.

[kiran_diwakar]

Hello Valjean,

 

Thanks so much for your time for the post and sharing the information

Regards,

Kiran Diwakar

[Anon Anon]

Our team completed our Spectrum server environment using the distributed installer for a total of 38 servers without issue. RHEL moving from 9.3 H01 to H03, in case anyway was curious. Maybe 2 hours on the actual install time.

[kiran_diwakar]

Awesome jpcordero - these kind of responses are very encouraging.

 

Regards,

Kiran Diwakar

[kiran_diwakar]

Hello Team,

 

Please stay tuned - we are almost done with 9.2 H13 - we are trying to complete the formalities and other admin things like checksum validation et al.

I will confirm the details in another hour or so. In the meanwhile, I would request people to please review the below techdocs that the team has put together and published - this could help is better preparation and better understanding:

 

TEC1651784 Which CA Spectrum 9.2.3.H12 PTF patches are included in 9.2.3.H13?

TEC1885985 Which CA Spectrum 9.3.0.H02 PTF patches are included in CA Spectrum 9.3.0.H03?

*this one includes a note about a post H13 PTF

 

We also published this KB that not related to the JRE Cert problem but may be seen as a 9.4 problem:

TEC1212723 orb_args parameter value in SS/.vnmrc may be truncated after upgrading Spectrum to 9.4.2.x on Linux platform.

 

Questions/comments welcome. Stay tuned for the 9.2 H13 GA notice very soon..

Regards,

Kiran Diwakar

[Rene Cantwell]

There’s one other “gotcha” you all should be aware of.  If you have made changes (i.e. https port) to your server.xml file on your OneClick box, you will have to reapply the change after installing 9.4.2.1, H03 or H13.  During the install we put down a new server.xml file with the default settings.

 

--

Rene’

 

Rene’ Cantwell

CA Technologies

Support Delivery Manager

603-334-2497

[e-mail]<mailto:[e-mail]>

[Anon Anon]

Hi I can see that media has been published to ftp://ftp.ca.com/CAproducts/CA-SPECTRUM/Updates/GA/Spectrum_09.02.03.H13/

is this H13 release ready to go ?

we are desperate to get out hands on it.

 

Thanks

[Rene Cantwell]

Yes H13 is ready to go, you can download now.

 

 

 

Rene’ Cantwell

CA Technologies

Support Delivery Manager

603-334-2497

[e-mail]<mailto:[e-mail]>

[kiran_diwakar]

                              Spectrum 9.2 H13 for Windows and Linux Now Available

 

Hello Team,

Here I am with the important update related to the remaining hot-fix (H13) for Spectrum 9.2 code stream.

You can get the image from:

ftp://ftp.ca.com/pub/CA-SPECTRUM/Updates/GA/Spectrum_09.02.03.H13/

 

Please note that we have completed the validation of Windows and Linux platforms for H13.

We are still continuing to verify  the Solaris image, I would want to request you to wait getting the Solaris image for 9.2 H13 until I or someone from the team explicitly confirms that it is ready to be downloaded. If you use the Solaris image before we confirm, you might pick the image that is not right or still being uploaded. This could create problems that would be very difficult to triage.

 

In another hour or so, we will confirm Solaris as well.

 

We are trying our best to be as fast as we possibly can and hope you are able to work through this problem.

Please let us know if you need anything.

Regards,

Kiran Diwakar

[Issac08]

Kiran,

  

I am running 9.3 baseversion can hotfix H03 can be diretly applied ?Since it is my first time to apply hotfix ,any document is there ?

 

And which version of java is afftected here ?

[kiran_diwakar]

Hello Issac08,

 

Right, H03 can be applied on top of the base version 9.3

Hot Fixes are typically a lot simpler than releases.

If you review the readme that you get with the image of the hotfix, it will guide through the key things.

 

Our teams are monitoring this forum religiously so if you need anything please call out here. Do not hesitate to reach out to your local support as well, in case you get blocked.

 

By the way, you will find that on this forum as well as through some direct emails, we have quite a few customers who went to H03 without any major hiccups

Hope this helps

 

Regards,

Kiran Diwakar

[kiran_diwakar]

                                   Spectrum 9.2 H13 for ALL PLATFORMS Now Available

 

Hello All,

We confirmed Solaris as well, so ALL the supported OS images are now available for download.

ftp://ftp.ca.com/pub/CA-SPECTRUM/Updates/GA/Spectrum_09.02.03.H13/

 

This concludes the final remaining set of HFs that we were working on to be uploaded.Thanks and all the best!

As mentioned earlier - we have our support, L2 and most other product teams on call over the weekend, in case anyone needs us.

Feel free to reach out to your local teams for any issue/clarification.

 

Regards,

Kiran Diwakar

[kiran_diwakar]

Hello Folks,

 

We got a confirmation that one of our larger customers completed their update to 9.2 H13 successfully

Any other updates or other comments?

Regards,

Kiran Diwakar

[Issac08]

Thanks Kiran,

In which version of Java it is affected ,some of my users can accesss the console.Kindly confirm.

[kiran_diwakar]

                                   Spectrum 9.2 H13 for ALL PLATFORMS Now Available

 

Hello All,

We confirmed Solaris as well, so ALL the supported OS images are now available for download.

ftp://ftp.ca.com/pub/CA-SPECTRUM/Updates/GA/Spectrum_09.02.03.H13/

 

This concludes the final remaining set of HFs that we were working on to be uploaded.Thanks and all the best!

As mentioned earlier - we have our support, L2 and most other product teams on call over the weekend, in case anyone needs us.

Feel free to reach out to your local teams for any issue/clarification.

 

Regards,

Kiran Diwakar

[kiran_diwakar]

I apologize Issac08 - I missed commenting on that part. JRE 1.7 and above are impacted.

If you have 9.3 - then you are impacted as we ship with 1.7

In any case, you can confirm the Java version.

 

Did your operators change the date to March 1st or later and still don't see a problem?

Regards,

Kiran Diwakar

[Marcio Moreno]

I decided not applying H03 in Spectrum 9.3, and so far, there is nothing wrong. Someone who did not install available Hot-Fixes had problems with JRE Certificate?

[kiran_diwakar]

Hi Marcio,

 

That is strange - 9.3 has 1.7 JRE - so you should have run into problem

Are your security settings "medium"?

Regards,

Kiran Diwakar

[Veronique Lefebure]

Yes, a colleague of mine who did not change the Java security level to Medium, got the problem this morning.

We did not install the patch, but things went well for those of us who either let the session open or changed the java security level.

We are running 9.3 (9.3.0.2.91)

[kiran_diwakar]

Right - that could be another reason - you wouldn't have closed OC as well.

 

Regards,

Kiran Diwakar

[kiran_diwakar]

I meant OC would have been kept running from before 1st March - that was the last workaround.

Just wanted to clarify. You might want to try opening new OC Client (WITHOUT closing the current client) and see if that runs into issue.

If it does (and i think it would will) - then you should try and get to H03 as soon as possible.

 

Regards,

Kiran Diwakar

[Marcio Moreno]

Hi Kiran,

 

No, my security level is set to "High", in portuguese "Alta" and OC client has to be closed frequently by a huge number of operators.

 

Tks,

Marcio

[Issac08]

Marcio,

What is the version?

[Marcio Moreno]

Hi Issac,

 

java version "1.7.0_67"

Java(TM) SE Runtime Environment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

 

 

Regards,

Marcio

[Veronique Lefebure]

Hi,

I am running 9.3.H02. I was planning to upgrade to 9.4.2 soon. Do you advice me to first upgrade to 9.3.H03 or may I skip it ?

thanks,

Veronique

[kiran_diwakar]

Hi Veronique,

 

I would strongly recommend 9.4 to everyone

9.4 would have longer life than 9.3

 

Regards,

Kiran Diwakar

[Issac08]

Hi Kiran,

 

From 9.3 it can directly updated to 9.4?

If means any document is there and what are backup must needed..thanks in advance !!

[kiran_diwakar]

Hi Issac08

 

9.3 to 9.4 is a very standard upgrade path. It is a lot easier than what you would have gone through for 9.2->9.3

The reason being with 9.3 there was change in data models to accommodate multi byte characters.

 

If you just look at 9.4 readme, you will get all that is needed.

 

Also, hope you are considering 9.4.2.1 - right? That is the latest and greatest with all the fixes for OC JRE Certificate expiration issues.

 

Regards,

Kiran Diwakar