Automic Workload Automation

 View Only
Expand all | Collapse all

WARNING: Serious ZDU bug impacting AE 21.0.5

  • 1.  WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Feb 24, 2023 09:37 AM
    Edited by Michael Lowry 28 days ago

    Summary: Do not upgrade to v21.0.5 using Zero Downtime Upgrade without first contacting Broadcom Support.

    Update 2023.03.02 18:20 CET: Broadcom has acknowledged that this problem is due to a bug in AE v21.0.5.

    I have not seen an announcement about this so I thought I would share our experience in the hopes others may be spared needless trouble.

    Eleven days ago, Broadcom released AWA 21.0.5. We had been waiting for bug fixes delivered in this specific release, so we were understandably eager to install the new version. Unfortunately, we ran into a problem when using the Zero Downtime Upgrade feature to upgrade one of our systems from 12.3.9 HF2 to 21.0.5.

    Everything went OK until step 4. Connections in the ZDU Wizard. During that step, we clicked Disconnect All Agents and then Disconnect All Users. This closed connections to the old-version Communication Processes, and triggered the switch of the PWP to the new version. As soon as the switch happened, the problem manifested. Normal (non-Java) Work Processes began to die with segmentation faults, writing UC4Dump files and forced traces.

    Some WPs would stay up for a few minutes, but others would die almost immediately after starting. Restarting the WPs worked fine, but they would soon die again. Eventually, more WPs were down than up. Also, the WP traces were beginning to fill up the file system, so we chose to stop the system and call Broadcom Support for help.

    Broadcom Support is currently investigating the problem. Their preliminary guess at a root cause is a known bug in 21.0.5. The bug is apparently related to ZDU, and a hotfix is expected soon.

    If you intend to upgrade to v21.0.5 using Zero Downtime Upgrade, I strongly recommend that you contact Broadcom Support for advice first.

    One more thing: I believe Broadcom should take a more proactive stance in notifying customers of known serious problems affecting releases that are publicly available. The Release Notes tool on the Automic Downloads site lists no known problems for Automation Engine 21.0.5. I searched for relevant KB articles and found only one partial hit: KB article 258694. (That bug is supposedly PostgreSQL-specific though, and we use Oracle; also, the symptoms described in the article are somewhat different from the ones we experienced.) We had been waiting for many months for bug fixes included in v21.0.5, and had reported several of these bugs; so Broadcom Support knew or should have known that we were intending to upgrade to this version. AE v12.0.5 was released almost two weeks ago, but Broadcom informed us that they knew about this problem only after we had run into the problem and opened a support case.



  • 2.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Feb 24, 2023 11:36 AM
    Edited by Matthias Schelp Feb 24, 2023 11:38 AM

    Hey Michael,

    thanks for the warning. I have once been a fan of the ZDU, but since we experienced multiple issues we stopped using it. Looks like things didn't change much.

    Regards,

    Matthias



  • 3.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Feb 25, 2023 05:59 PM

    Hi ,

    I haven't seen automic team write one word in the known issue tab even once. usually a clean page greets us every time :D interesting. thanks for warning.



    ------------------------------
    Olgun Onur Ozmen
    https://www.linkedin.com/in/olgunonurozmen/
    ------------------------------



  • 4.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Feb 26, 2023 06:07 AM
    Edited by Michael Lowry Feb 26, 2023 06:25 AM

    The Automic Downloads site recommends version 21.0.5 as an upgrade suggestion. It lists no known issues.



  • 5.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Feb 27, 2023 01:53 AM

    Hi Michael,

    Thanks for the information. We planned to upgrade from version 21.0.4 to 21.0.5 via ZDU. I opened a case by Broadcom to have more information.

    Regards




  • 6.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted 30 days ago

    Please provide results of your case going from 21.0.4 to 21.0.5.  Are your AE on UNIX or Windows?  I am at the same level.  I know that going from 12.3.6 to 21.0.3, I was unable to use ZDU.  Broadcom had verified this was an issue and I was required to do a cold start.  No point in doing ZDU if I have to do a cold start.