Automic Workload Automation

 View Only
Expand all | Collapse all

Security Notice for CA Automic Automation

  • 1.  Security Notice for CA Automic Automation

    Posted Jun 10, 2022 03:05 AM
    CA Technologies, A Broadcom Company, is alerting customers to multiple vulnerabilities in CA Automic Automation. Multiple vulnerabilities exist that can allow a remote attacker to execute arbitrary code or commands, access sensitive data, enumerate users, or elevate privileges. CA has published solutions to address this vulnerability and recommends that all affected customers implement these solutions.

    More info can be found here:
    Support Content Notification - Support Portal - Broadcom support portal


    /Keld.


  • 2.  RE: Security Notice for CA Automic Automation

    Posted Jun 13, 2022 10:28 AM
    Edited by Carsten Schmitz Jun 13, 2022 10:29 AM
    @Kaj Wierda I appreciate the heads-up here. But as a company, we are displeased​ with the way Broadcom (still) reports Automic security bulletins. Points I kindly ask to take into consideration:

    1. as an Administrator, it's my task to make specific recommendations on just how critical updating is. This announcement is rather light on details. The links are dead (there are no "CVE2022-pending1"-style things in the CVE database). It would be very much required to have some more in-depth information to advise decision makers on short notice whether to approve an emergency update or not.

    2. Related: Does this not even specify WHICH agent is affected? You are selling a multitude of Automic agents. Sorry, merely saying "CA Automic Automation Agent" does not cut it. Windows? Linux? Maybe even Java? We have multiple engines and potentially up to 900 agents. I'd rather you identify your products than me doing overtime to do emergency updates on some obscure RA agent potentially due to incomplete vendor information.

    Thanks, regards,
    Carsten


  • 3.  RE: Security Notice for CA Automic Automation

    Posted Jun 13, 2022 11:14 AM
    I agree 100%  This announcement should contained detailed information. 
    That said I believe it is related to unix only agents based upon the link referenced.  "The original Automic Automation security advisory can be found here."

     



  • 4.  RE: Security Notice for CA Automic Automation

    Posted Jun 14, 2022 02:54 AM
    Thanks for the link, Timothy.

    I am just confused about the date of this notification (Id 18598): 06 August 2021 !?
    Did we all miss out on this alert?


    ------------------------------
    Regards, Nicole
    ------------------------------



  • 5.  RE: Security Notice for CA Automic Automation

    Posted Jun 14, 2022 03:24 AM
    what @Carsten Schmitz says.​


  • 6.  RE: Security Notice for CA Automic Automation

    Broadcom Employee
    Posted Jun 14, 2022 11:48 AM
    Thanks for the feedback. Several updates were made to this follow up notification based on your feedback, including a more precise solution indicated which agents are effected (UNIX).

    ------------------------------
    Kaj Wierda
    Sr. Product Line Manager | Automation

    Broadcom Software
    ------------------------------



  • 7.  RE: Security Notice for CA Automic Automation

    Posted Jun 20, 2022 05:27 AM
    Edited by Carsten Schmitz Jun 20, 2022 05:28 AM
    All: Broadcom has published an UPDATE to the security advisory.

    I don't know about you, but I have NOT received email notification about this by Broadcom. Either way, here it is:

    https://support.broadcom.com/web/ecx/support-content-notification/-/external/content/SecurityAdvisories/0/20629

    12.3.7+HF2 is NOT sufficient anymore, apparently one now needs 12.3.8 instead. Or maybe 21? Not sure. Apparently, writing unambiguous advisories is hard :( 

    For all vulnerabilities:

    • Upgrade UNIX Agents and Automation Engine to V21

    The current and recommended releases are 21.0.3 or 12.3.8.

     It also adds that for V12.3.x, the agent authentication method needs to be changed to LOCAL_REMOTE, which entails downloading, from AWI, a key file for every agent, and deploying it as a zip file to every agent (that has a key store).

    Side note: For us, this means we'd need to download about 800 files manually, by hand, from AWI, and deploy them to computers that are managed by a third party. Which is NOT a workable process.




  • 8.  RE: Security Notice for CA Automic Automation

    Broadcom Employee
    Posted Jun 20, 2022 05:38 PM
    Edited by Shannon Hebert Jun 20, 2022 05:47 PM
    Hi Carsten and All,

    Our notifications are sent from support-noreply@broadcom.com and contain a hyperlink that your corporate email security may have flagged. I'll look to ensure all is working as it should with our notifications, though please ensure you're signed up for all related Automic products by searching and enabling the notification from here: https://support.broadcom.com/user/notifications.html

    Our goal with the recent security advisory is to ensure customers don't get compromised. We provided fixes and published a security advisory last year to give customers adequate time to upgrade. We chose not to disclose details publicly to protect customers and allow you time to upgrade/apply fixes. The security researcher requested public recognition, which is the reason for filing the CVEs, now updated here.

    As stated in the announcement, the recommendation and solution for the Unix agent vulnerabilities are to update to at least 12.3.7. We recommend you use the latest version of the major release you're on, which is currently 12.3.8 HF3, and 21.0.3 HF1 was published today.
    The recommendation for the Automation Engine vulnerability is to either turn on the local_remote authentication method or update it to 21.0. Both can be used. Or, if our customers use local_remote for authentication in 12.3 as a short to medium-term mitigation, it can be turned off in 21.0 using the method described in the documentation. 

    • It is also possible in version 21.0 to retrieve the authentication package through a REST method.

    When considering what action to take, the risk is reduced substantially if your agents are protected within internal networks and have strict security guidelines for users. If agents are located on the DMZ/accessible to the internet, the AE engine, components, and agents should be upgraded to 12.3.7+ with local_remote or v21 with TLS in place as they're at risk. Each customer will have to assess the risk for themselves.

    Broadcom PM, Engineering, and Support want to ensure customers take appropriate action to mitigate potential risk. We'll look over your upgrade plan and give feedback before your maintenance. If you're upgrading to v21.0, we have July 16th and October 15th open for 2022 Designated Upgrade Weekends.

    Please create a Support case if you have any questions or would like us to review your plans.

    Thanks,

    -Shannon

     

    Shannon Hebert

    Manager of Automation Support

    Broadcom Software 




  • 9.  RE: Security Notice for CA Automic Automation

    Posted Jun 21, 2022 09:57 AM
    Hi @Shannon Hebert

    I just want to ask this. Why is the final fix (not workaround or mitigation) for CVE-2022-33756 only available for v21 but not for versions with 12.3.x? 





  • 10.  RE: Security Notice for CA Automic Automation

    Posted Jun 21, 2022 10:52 AM
    Hi,

    CVE-2022-33756 is a weakness in Automics authentication algorithm, which was replaced by TLS in version 21.

    So the fix would be to backport TLS support to 12.x, which was one of the, if not the, major change in V21 and can't be done easily or in a reasonable amount of time.

    Regards, Matthias


  • 11.  RE: Security Notice for CA Automic Automation

    Broadcom Employee
    Posted Jun 23, 2022 10:28 AM
    Hi Olgun, Matthias, and All,

    Over the past several days, we've been discussing how we could enable this process to be more straightforward for 12.3 and a large number of agents. Matthias is correct that porting TLS to 12.3 is a significant architectural change we designed for in v21. By versioning standards, this is one of the several reasons we can identify v21 as a new major release.

    We're brainstorming other possibilities to update the encryption keys, but this is not a straightforward task with the safeguards in place in 12.3.

    Please continue your remediation of the vulnerabilities by following our current documented process. We'll let you know if we can identify a time-saving approach to update these keys while maintaining the stabilization of the 12.3 ecosystems.

    Thank you,

    -Shannon

     

    Shannon Hebert

    Manager of Automation Support

    Broadcom Software | Agile Operations Division




  • 12.  RE: Security Notice for CA Automic Automation

    Posted Jun 23, 2022 01:07 PM
    Hello Shannon

    We began the process of updating from version 12.2 to version 21 at the company. This will be a complex and long procedure that will take many months to complete, with the entire process ending up in October.
    My concern is, can I update the agents now to the version 12.3 or higher while I am still using the 12.2 system version to mitigate this vulnerability ?

    Thank you in advance.


  • 13.  RE: Security Notice for CA Automic Automation

    Broadcom Employee
    Posted Jun 23, 2022 02:06 PM
    Hi @Gilberto Pacheco

    agents can connect to a higher AE version but not vice versa.

    Agent-version <= AE-version

    You must start upgrading your AE-system (and Analytics and AWI) before you can upgrade your existing Agents.

    br Michael


  • 14.  RE: Security Notice for CA Automic Automation

    Posted Jun 23, 2022 01:07 PM

    Hello Shannon

     

    We began the process of updating from version 12.2 to version 21 at the company. This will be a complex and long procedure that will take many months to complete, with the entire process ending up in October.

    My concern is, can I update the agents now to the version 12.3 or higher while I am still using the 12.2 system version to mitigate this vulnerability ?

     

    Thank you in advance.




  • 15.  RE: Security Notice for CA Automic Automation

    Posted Jun 28, 2022 06:55 AM
    Edited by Carsten Schmitz Jun 28, 2022 06:55 AM

    Hi Shannon.

    Thanks for the additional infos (I have only seen the updates to this thread yesterday, my bad).

    As for notifications: I visited the mail notification interface you linked. I have never visited this one before (I feel like there were a bunch of predecessors or parallel interfaces where I made similar selections in the past). THIS incarnation of an interface had checked "Critical Notification" and a bunch of other things, but "Security Notification" was unchecked. I made sure it's all checked now.

    As for the required security fix: I'll toy today with our dev installation to implement the procedure you folks recommend for 12.3.8, and determine whether we need to download and distribute a key file for our ~900 agents individually, or whether that can somehow be streamlined by us. Again I will also need to give a management recommendation soon whether to perform this expensive action at all, or accept a (largely unspecified) security risk.

    Let's just say all of this is not exactly a great situation for anyone involved.

    Not to beat a dead horse too much, but: On another layer, the entire situation is, let's say, resurfacing more questions. For many years (back to pre-CA days), our company and others have asked the regents of Automic (a middle-ware handling both extremely sensitive data and credentials) for it to be independently audited for security. CA has told us that the software is audited by a reputable third party, but refused to divulge who that is, and any of the findings/reports. It would appear that security researchers unaffiliated with Broadcom have now uncovered a number of flaws. It would appear from the available information that this would pose major questions on the thoroughness of these independent audits.

    About two years ago, I reported a security concern myself (one that is very high impact, albeit possibly also entirely or near-unavoidably ingrained in product design). I proposed a possible way to escalate this concern even further into a vulnerability (which, granted, I didn't test for because that required certain prep work, but would have appreciated an answer to). I deliberately reported this not to product management, but at that time to a security function published on the Broadcom website. It later turned out that this security function was merely inherited from Symantec. Yet I was told that Broadcom takes such reports seriously and will investigate, however, I never noted any further correspondence from Broadcom on the matter ever again.

    Maybe it is time to rethink what Broadcom/Automic should do to rebuild trust in the security assurances for the product?

    Thanks,

    Carsten



    Original Message:
    Sent: Jun 20, 2022 05:38 PM
    From: Shannon Hebert
    Subject: Security Notice for CA Automic Automation

    Hi Carsten and All,

    Our notifications are sent from support-noreply@broadcom.com and contain a hyperlink that your corporate email security may have flagged. I'll look to ensure all is working as it should with our notifications, though please ensure you're signed up for all related Automic products by searching and enabling the notification from here: https://support.broadcom.com/user/notifications.html

    Our goal with the recent security advisory is to ensure customers don't get compromised. We provided fixes and published a security advisory last year to give customers adequate time to upgrade. We chose not to disclose details publicly to protect customers and allow you time to upgrade/apply fixes. The security researcher requested public recognition, which is the reason for filing the CVEs, now updated here.

    As stated in the announcement, the recommendation and solution for the Unix agent vulnerabilities are to update to at least 12.3.7. We recommend you use the latest version of the major release you're on, which is currently 12.3.8 HF3, and 21.0.3 HF1 was published today.
    The recommendation for the Automation Engine vulnerability is to either turn on the local_remote authentication method or update it to 21.0. Both can be used. Or, if our customers use local_remote for authentication in 12.3 as a short to medium-term mitigation, it can be turned off in 21.0 using the method described in the documentation. 

    • It is also possible in version 21.0 to retrieve the authentication package through a REST method.

    When considering what action to take, the risk is reduced substantially if your agents are protected within internal networks and have strict security guidelines for users. If agents are located on the DMZ/accessible to the internet, the AE engine, components, and agents should be upgraded to 12.3.7+ with local_remote or v21 with TLS in place as they're at risk. Each customer will have to assess the risk for themselves.

    Broadcom PM, Engineering, and Support want to ensure customers take appropriate action to mitigate potential risk. We'll look over your upgrade plan and give feedback before your maintenance. If you're upgrading to v21.0, we have July 16th and October 15th open for 2022 Designated Upgrade Weekends.

    Please create a Support case if you have any questions or would like us to review your plans.

    Thanks,

    -Shannon

     

    Shannon Hebert

    Manager of Automation Support

    Broadcom Software 


    Original Message:
    Sent: Jun 20, 2022 05:26 AM
    From: Carsten Schmitz
    Subject: Security Notice for CA Automic Automation

    All: Broadcom has published an UPDATE to the security advisory.

    I don't know about you, but I have NOT received email notification about this by Broadcom. Either way, here it is:

    https://support.broadcom.com/web/ecx/support-content-notification/-/external/content/SecurityAdvisories/0/20629

    12.3.7+HF2 is NOT sufficient anymore, apparently one now needs 12.3.8 instead. Or maybe 21? Not sure. Apparently, writing unambiguous advisories is hard :(

    For all vulnerabilities:

    • Upgrade UNIX Agents and Automation Engine to V21

    The current and recommended releases are 21.0.3 or 12.3.8.

     It also adds that for V12.3.x, the agent authentication method needs to be changed to LOCAL_REMOTE, which entails downloading, from AWI, a key file for every agent, and deploying it as a zip file to every agent (that has a key store).

    Side note: For us, this means we'd need to download about 800 files manually, by hand, from AWI, and deploy them to computers that are managed by a third party. Which is NOT a workable process.


    Original Message:
    Sent: Jun 10, 2022 03:04 AM
    From: Keld Mollnitz
    Subject: Security Notice for CA Automic Automation

    CA Technologies, A Broadcom Company, is alerting customers to multiple vulnerabilities in CA Automic Automation. Multiple vulnerabilities exist that can allow a remote attacker to execute arbitrary code or commands, access sensitive data, enumerate users, or elevate privileges. CA has published solutions to address this vulnerability and recommends that all affected customers implement these solutions.

    More info can be found here:
    Support Content Notification - Support Portal - Broadcom support portal


    /Keld.



  • 16.  RE: Security Notice for CA Automic Automation

    Posted Jun 29, 2022 02:06 AM
    Edited by John Birck Jun 29, 2022 02:12 AM
    Hello Carsten,

    Thanks a lot for naming it all!
    I had subscribed for security notices too but never got an info about this bug known since August 2021. 
    August Twenty-ONE.
    Currently I have a  hard stand in our company with the security guys. They are already pestering me since years for AWA security and now this product and its company do definitely look a bit lite-hearted in security issues. In worst case we will lose access to servers with sensitive information. Which means some departments finally can substitute AWA.
    With the decrease of support, loss of current German documentation and the software quality in general I am continously losing ground to another scheduling product. What a pity!
    Kind regards & thanks a lot four your clear words

    John

    ------------------------------
    Technical Project Manager
    Aveniq IT
    Switzerland
    ------------------------------



  • 17.  RE: Security Notice for CA Automic Automation

    Posted Jun 29, 2022 03:28 AM
    Edited by Carsten Schmitz Jun 29, 2022 03:28 AM
    Hey John,

    Thank you for the open words, and the encouragement too!

    I feel you, and judging by exchanges with others, several other clients are, too. We've also already looked into product alternatives and the situation is similar here. I also need to answer to various internal security functions (and I'm part of one, too), and they are mightily unimpressed with the "Gesamtsituaton" (i..e overall state of affairs) ;)

    Best,
    Carsten


  • 18.  RE: Security Notice for CA Automic Automation

    Posted Jun 29, 2022 07:58 AM
    Edited by Carsten Schmitz Jun 29, 2022 08:02 AM
    So ... I don't get it. Has anyone figured this out?

    Premise 1: To fix CVE-2022-33756 without upgrading to V21, I'm supposed to have 12.3.7 and configure "local_remote" auth method

    Premise 2: For the secure way, I need to shut down all agents, download an INDIVIDUAL (per agent) package, and distribute it (How do I do this with 900 agents when I'm not allowed more than at most a few hours of downtime?)

    ... but THAT question isn't new. This one is:

    Premise 3: The documentation says that to set "local_remote", I am to use the "AE.DB load utility" *sic*. HOWEVER:

    Premise 4: The documentation for the AE Load Utility says, for the required option "-T":

    -TAuthentication method and

    -KCompany key string Authentication details, see Agent Authentication

    Important! Only use the utility with these start parameters if you newly install without using the Automic Web Interface.

    OTHER documentation says that:

    "Warning! Changing the authentication method is possible, but involves considerable effort. For more information, see Changing the Authentication Method."

    ... but that links BACK to the above documentation, which says these parameters are ONLY EVER to be used "if you newly install without using the Automic Web Interface."

    But regardless of the ambiguity of that sentence alone: Either way, I am NOT newly installing, and I am using the Automic Web Interface (not for installing, of course, but in general).

    So, apart from the total impossibility to change this on ~900 agents in realistic time: Could I even, with an existing installation, or could I not? Which one is it?

    Thanks.



  • 19.  RE: Security Notice for CA Automic Automation

    Posted Jun 29, 2022 10:48 AM
    Hi @Carsten Schmitz

    we used  Changing the Authentication Method
    header: To Change the Authentication Method from NO to LOCAL_REMOTE (Server and Agent)​  . 
    it was in a healthy way. We have successfully completed. If you follow the "oha" table while doing the process, you will see that you have inserted 0000 records there.

    note: The only thing you need to pay attention to,  If you do not select all agents while "withdrawing and downloading" , you need to logout and login each time because awi doesn't understand that you chose a different agent. I think this may be a bug for awi. If you select all agents at once, it's fine. I suggest select all agents for this. 

    If the number of agents is high, I suggest you:
    1. Complete your agent upgrades first.
    2. Perform agent upgrade and local_remote steps at separate times.
    3. Make copies of the new .ini files in advance. Prepare these ini files before for you to rename the original when agents are down. (for keystore and package section). 
    4. Gather all your downloaded packages in one common file that agents can access. --> example path: /automic/Agents/keystorePackage/package.agent_name ... 

    It will speed up the work for 900 agents.





  • 20.  RE: Security Notice for CA Automic Automation

    Broadcom Employee
    Posted Jul 25, 2022 05:38 AM
    Edited by Kaj Wierda Jul 25, 2022 05:38 AM

    I have posted following update regarding CVE-2022-33756 just before the weekend:

    https://community.broadcom.com/enterprisesoftware/discussion/security-notification-update-cve-2022-33756-1#bm920432ee-50ee-4138-a823-8cc2d3df177d



    ------------------------------
    Kaj Wierda
    Sr. Product Line Manager | Automation

    Broadcom Software
    ------------------------------