Automic Workload Automation

 View Only
Expand all | Collapse all

WARNING: Serious ZDU bug impacting AE 21.0.5

  • 1.  WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Feb 24, 2023 09:37 AM
    Edited by Michael A. Lowry Mar 02, 2023 12:21 PM

    Summary: Do not upgrade to v21.0.5 using Zero Downtime Upgrade without first contacting Broadcom Support.

    Update 2023.03.02 18:20 CET: Broadcom has acknowledged that this problem is due to a bug in AE v21.0.5.

    I have not seen an announcement about this so I thought I would share our experience in the hopes others may be spared needless trouble.

    Eleven days ago, Broadcom released AWA 21.0.5. We had been waiting for bug fixes delivered in this specific release, so we were understandably eager to install the new version. Unfortunately, we ran into a problem when using the Zero Downtime Upgrade feature to upgrade one of our systems from 12.3.9 HF2 to 21.0.5.

    Everything went OK until step 4. Connections in the ZDU Wizard. During that step, we clicked Disconnect All Agents and then Disconnect All Users. This closed connections to the old-version Communication Processes, and triggered the switch of the PWP to the new version. As soon as the switch happened, the problem manifested. Normal (non-Java) Work Processes began to die with segmentation faults, writing UC4Dump files and forced traces.

    Some WPs would stay up for a few minutes, but others would die almost immediately after starting. Restarting the WPs worked fine, but they would soon die again. Eventually, more WPs were down than up. Also, the WP traces were beginning to fill up the file system, so we chose to stop the system and call Broadcom Support for help.

    Broadcom Support is currently investigating the problem. Their preliminary guess at a root cause is a known bug in 21.0.5. The bug is apparently related to ZDU, and a hotfix is expected soon.

    If you intend to upgrade to v21.0.5 using Zero Downtime Upgrade, I strongly recommend that you contact Broadcom Support for advice first.

    One more thing: I believe Broadcom should take a more proactive stance in notifying customers of known serious problems affecting releases that are publicly available. The Release Notes tool on the Automic Downloads site lists no known problems for Automation Engine 21.0.5. I searched for relevant KB articles and found only one partial hit: KB article 258694. (That bug is supposedly PostgreSQL-specific though, and we use Oracle; also, the symptoms described in the article are somewhat different from the ones we experienced.) We had been waiting for many months for bug fixes included in v21.0.5, and had reported several of these bugs; so Broadcom Support knew or should have known that we were intending to upgrade to this version. AE v12.0.5 was released almost two weeks ago, but Broadcom informed us that they knew about this problem only after we had run into the problem and opened a support case.



  • 2.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Feb 24, 2023 11:36 AM
    Edited by Matthias Schelp Feb 24, 2023 11:38 AM

    Hey Michael,

    thanks for the warning. I have once been a fan of the ZDU, but since we experienced multiple issues we stopped using it. Looks like things didn't change much.

    Regards,

    Matthias



  • 3.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Feb 25, 2023 05:59 PM

    Hi ,

    I haven't seen automic team write one word in the known issue tab even once. usually a clean page greets us every time :D interesting. thanks for warning.



    ------------------------------
    Olgun Onur Ozmen
    https://www.linkedin.com/in/olgunonurozmen/
    ------------------------------



  • 4.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Feb 26, 2023 06:07 AM
    Edited by Michael A. Lowry Feb 26, 2023 06:25 AM

    The Automic Downloads site recommends version 21.0.5 as an upgrade suggestion. It lists no known issues.



  • 5.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Feb 27, 2023 01:53 AM

    Hi Michael,

    Thanks for the information. We planned to upgrade from version 21.0.4 to 21.0.5 via ZDU. I opened a case by Broadcom to have more information.

    Regards




  • 6.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Feb 28, 2023 06:49 PM

    Please provide results of your case going from 21.0.4 to 21.0.5.  Are your AE on UNIX or Windows?  I am at the same level.  I know that going from 12.3.6 to 21.0.3, I was unable to use ZDU.  Broadcom had verified this was an issue and I was required to do a cold start.  No point in doing ZDU if I have to do a cold start.  




  • 7.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Mar 01, 2023 01:23 AM

    Hi Lester,

    AE was on Windows with MS SQL DB. We upgraded from 12.3.9 HF2 to 21.0.4 without ZDU. We use only ZDU for minor upgrade. Actually we have to upgrade from 21.0.4 to 21.0.5 (minor upgrade) and we want to use it. I opened a case by Broadcom to know if we are impacted by the same problem as Michael.

    Regards



    ------------------------------
    Donato Faggella
    DevOps Engineer III
    Swisscom (Suisse) SA
    ------------------------------



  • 8.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Broadcom Employee
    Posted Feb 28, 2023 04:29 PM

    Hi Michael and All,

    During our initial testing of 21.0.5, the ZDU was functional. With the details you supplied through the support ticket, we could reproduce the issue internally and will incorporate these differences in our testing methods. The ZDU issues have been corrected and are scheduled to be included in 21.0.5 HF2, due out within the next two weeks.

    I agree that we can improve by raising awareness in our customer base as soon as we confirm an issue that can impact the efforts underway.

    Best,

    -Shannon 



    ------------------------------
    _______________________________________
    Shannon Hebert
    Head of AOD Automation Support
    Broadcom Software | Agile Operations Division
    _______________________________________
    ------------------------------



  • 9.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Mar 01, 2023 03:02 AM
    Edited by Michael A. Lowry Mar 01, 2023 03:02 AM

    Thanks for the reply, @Shannon Hebert. We look forward to the fix and to greater transparency in the future.



  • 10.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Broadcom Employee
    Posted Mar 07, 2023 12:05 PM

    Hi All,

    21.0.5 HF1 was planned to be released last week and has been delayed due to architectural changes within our internal environment. We will notify our customers as soon as it is released. Our current plan is to release the ZDU fix in HF2, which is due out soon after HF1.

    I appreciate your patience.

    -Shannon



    ------------------------------
    _______________________________________
    Shannon Hebert
    Head of AOD Automation Support
    Broadcom Software | Agile Operations Division
    _______________________________________
    ------------------------------



  • 11.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Mar 08, 2023 07:55 AM

    Hi @Shannon Hebert 

    we are waiting for AE 21.0.5 HF1 to fix some issues related to AAI. What will HF2 fix other than the ZDU issue ?

    /Keld.




  • 12.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Broadcom Employee
    Posted Mar 09, 2023 10:25 PM

    Hi Keld,

    The primary fix in HF2 is for ZDU, as you've mentioned. I'll update this thread once HF1 is released and if there are additional fixes in HF2.

    Thank you,

    -Shannon



    ------------------------------
    _______________________________________
    Shannon Hebert
    Head of AOD Automation Support
    Broadcom Software | Agile Operations Division
    _______________________________________
    ------------------------------



  • 13.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Broadcom Employee
    Posted Mar 14, 2023 10:48 AM

    Hi All,

    21.0.5 HF1 was released today: https://support.broadcom.com/external/content/ReleaseAnnouncements/0/21894

    The link above links to the download and fixes within.

    Thank you,

    -Shannon



    ------------------------------
    _______________________________________
    Shannon Hebert
    Head of AOD Automation Support
    Broadcom Software | Agile Operations Division
    _______________________________________
    ------------------------------



  • 14.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Mar 14, 2023 11:26 AM
    Edited by Michael A. Lowry Mar 14, 2023 11:25 AM

    The AWA v21.0.5 HF1 release notes mention seven bug fixes. The release notes also list two known problems:

    • Issues with ZDU
    • :RESOLVE function does not work in v21

    For both of these known problems, the fixed version is the same version, v21.0.5 HF2.

    If these are bugs fixed in this release, why are they listed under known problems instead of bug fixes?



  • 15.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Broadcom Employee
    Posted Mar 14, 2023 11:56 AM

    Hi Michael,

    Those items are listed as known problems, as they are not yet fixed and plan to be included in HF2, targeted to be released within the next week or so.

    Best,

    -Shannon



    ------------------------------
    _______________________________________
    Shannon Hebert
    Head of AOD Automation Support
    Broadcom Software | Agile Operations Division
    _______________________________________
    ------------------------------



  • 16.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Mar 14, 2023 12:21 PM
    Edited by Michael A. Lowry Mar 14, 2023 12:21 PM

    Ok, thanks. In this case, the fixed version should probably be changed to 21.0.5 HF2



  • 17.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Mar 14, 2023 01:05 PM

    Just to make sure I understand...  If we're running a version of 21 prior to 21.0.5 HF2, then we won't be able to use the ZDU to upgrade -- e.g., we'll have to do the old-fashioned manual upgrade from 21.04 to 21.0.5 HF2 -- but we should be able to use the ZDU for newer versions beyond that?




  • 18.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Mar 15, 2023 03:39 AM
    Edited by Michael A. Lowry Mar 15, 2023 03:38 AM

    Not quite correct.

    The bug that will be fixed in v21.0.5 HF2 fixes a problem when using ZDU to upgrade to versions prior to v21.0.5 HF2.

    If you are planning to upgrade to v21.0.5 using ZDU, you should wait for v21.0.5 HF2.

    If you can use the non-ZDU approach, then v21.0.5 HF1 should be fine.



  • 19.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Mar 16, 2023 05:30 AM
    Edited by Michael A. Lowry Mar 16, 2023 05:30 AM

    The text has been updated. The label Fixed Versions has been replaced with Affected Versions.

    Known Issues

    Issues with ZDU; due to issues with ZDU we recommend manual upgrades.
    Components:
    • AutomationEngine
    Affected Versions:
    • Automation.Engine 21.0.5 HF1
    • Automation.Engine 21.0.5
    :RESOLVE function does not work in v21.
    Will be fixed with ticket AE-31468
    Components:
    • AutomationEngine
    Affected Versions:
    • Automation.Engine 21.0.5 HF1
    • Automation.Engine 21.0.5

    @Shannon Hebert: It would be helpful if Broadcom consistently included Fixed Versions for Known Issues, even for those versions that have not been released yet.

    Ideally, Broadcom should also include a link to a KB article for each bug fix and known issue. (Right now, KB coverage is spotty, and there is no straightforward way to find KB articles based on the descriptions on the Downloads site.)



  • 20.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Mar 20, 2023 06:45 AM

    Hi @Shannon Hebert 

    The issue mentioned in this KB article is not mentioned in the Bug Fixes section of the v21.0.5 HF1 release notes. Do you think that, in the future, all bug fixes will be mentioned in the Release Notes...?




  • 21.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Mar 20, 2023 07:02 AM
    Edited by Michael A. Lowry Mar 20, 2023 07:01 AM

    If the KB articles, fix lists, and release notes all shared the same back-end database, it would be more straightforward to link from one to the other.



  • 22.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Broadcom Employee
    Posted Mar 21, 2023 03:28 PM

    Hi Keld,

    This issue was a special case, fixed for 21.0.6 and then backported to 21.0.5 HF1. I did confirm the fix is included, and we intend to publish all released fixes in the release notes. I'll follow up on this process.

    Thank you,

    -Shannon



    ------------------------------
    _______________________________________
    Shannon Hebert
    Head of AOD Automation Support
    Broadcom Software | Agile Operations Division
    _______________________________________
    ------------------------------



  • 23.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Broadcom Partner
    Posted Mar 23, 2023 09:28 AM

    Hi Shannon


    Your lines do confuse me. Is the fix for ZDU included in (the released) HF1 (software download) and just the "release documentation" was not updated or do we have to wait for another HF(2?) that will include the fix for the ZDU problem?

    Kind Regards

    Stefan



    ------------------------------
    Stefan Lerch
    Senior Consultant
    at www.systempartners.ch​

    For AUTOMIC trainings please check https://www.systempartners.ch/en/homee/services/training/trainingautomic/
    ------------------------------



  • 24.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Broadcom Employee
    Posted Mar 23, 2023 10:56 AM

    Hi Stefan,

    Keld asked a question on a non-ZDU topic, which is what my last reply addressed. 

    The ZDU issues will be addressed in 21.0.5 HF2.

    Thank you,

    -Shannon



    ------------------------------
    _______________________________________
    Shannon Hebert
    Head of AOD Automation Support
    Broadcom Software | Agile Operations Division
    _______________________________________
    ------------------------------



  • 25.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Broadcom Employee
    Posted Mar 28, 2023 01:15 PM

    Hi All,

    21.0.5 HF2 is now available for download. This announcement is on its way out to our customers: https://support.broadcom.com/web/ecx/support-content-notification/-/external/content/ReleaseAnnouncements/0/21991

    Thank you,

    -Shannon



    ------------------------------
    _______________________________________
    Shannon Hebert
    Head of AOD Automation Support
    Broadcom Software | Agile Operations Division
    _______________________________________
    ------------------------------



  • 26.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Broadcom Employee
    Posted Mar 29, 2023 04:53 PM

    Hi All,

    An issue has been identified with the Unix Agent in 21.0.5 HF2, so HF3 (including the Unix Agent only) is planned for release tomorrow.

    This will be posted in the HF2 Known Issues: UNIX Agent may not reconnect automatically after an unintended disconnect (e.g., due to a network failure).

    Workaround: Kill the listener process of the respective Unix agent and restart the agent with Service Manager or manually. The Unix agent will then reconnect.

    Additional tests have been added to our QA test plan, including network disconnect testing.

    Best,

    -Shannon



    ------------------------------
    _______________________________________
    Shannon Hebert
    Head of AOD Automation Support
    Broadcom Software | Agile Operations Division
    _______________________________________
    ------------------------------



  • 27.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Mar 01, 2023 03:25 AM
    Edited by Michael A. Lowry Mar 01, 2023 03:25 AM

    Our system is now back up and running thanks to excellent support from Marco S. at Broadcom.

    A cold start failed to resolve the problem, but tracing (tcp/ip=2 & db=4) revealed that an EVNT task was likely causing the WP segmentation faults. We canceled this task and the problem went away. The EVNT object was last edited in 2020, and the activation time of the task was in 2021, likely prior to the previous upgrade. It seems possible that Broadcom's ZDU testing did not include scenarios involving tasks from versions more than a couple of years old.

    I expect we will learn more in the coming days. I will provide a link to the relevant KB article when it is published.

    Thanks also to @Pascal Osthus-Bugat for coordinating support efforts and helping to ensure a quick resolution.



  • 28.  RE: WARNING: Serious ZDU bug impacting AE 21.0.5

    Posted Mar 02, 2023 12:18 PM
    Edited by Michael A. Lowry Mar 02, 2023 12:18 PM

    Broadcom has published KB260912, advising customers not to upgrade to v21.0.5 using ZDU until a fix is released.

    This will be fixed in a future release of 21.0 on a high priority. It is important to not use ZDU to upgrade to 21.0.5 (or any 21.0.5.x hotfixes) until a fix is released. Manual upgrades can still be carried out without issue.