Automation Engine v12.3.9 HF1 removed from downloads site

[Michael A. Lowry]

This morning I noticed that Automation Engine v12.3.9 HF1 was is not longer available from downloads.automic.com.

There are presently five (5) KB articles describing problems supposedly fixed by 12.3.9 HF1 .

249702 Adding a comment to a JOBF runid not possible on 12.3.9 with HTTP 500

Adding a comment to a JOBF runid is not possible via AWI or REST API after having upgraded the AE to 12.3.9 The error displayed in AWI is the following: In the JCP, we can find the related errors: U00045014 Exception 'java.lang.NullPointerException: ....,
Fix version: Component(s): Automation Engine Automation.Engine 12.3.9 HF1 - Available

248312 Solaris Agent : Filetransfers stuck in "Connecting" status

Solaris Agent 12.2.10 is generating cores when is it submitting File transfer. The FT jobs remains in status "Connecting" During the test period where traces where collected the agent generated 81 cores. Job trace: AIN-THREAD 20211223/082607.848 se....,
This bug is fixed in the following releases: Hotfix: 12.3.9 HF1 of the Solaris Agent - available.

244934 Solaris Agent 12.2.10 throws core dumps during File Transfer jobs

Solaris Agent 12.2.10 is generating cores when is it submitting File transfer. During the test period the agent generated 81 can throw up to 80 cores. Job trace: AIN-THREAD 20211223/082607.848 send_IPC_internal(type=CHANNEL_CLOSE,msg(100a05e50,msgI....,
Fix version: Component(s): Agent Unix Automation.Engine 12.3.9 HF1 - Available Automation.Engine 21.0.4 - Available

237697 U00045014 - Unable to find the fromVertex for WorkflowStructureItem

The JCP on each of our AE nodes stopped working and in the JCP logs we are able to find many of the following error messages: U00045014 Exception 'java.lang.IllegalArgumentException: "Unable to find the fromVertex for WorkflowStructureItem{runId=1207....,
Fix version: Component(s): Automation Engine Automation.Engine 12.3.9 HF1 - Available Automation.Engine 21.0.3 HF3 - Available Will be fixed in Automation.Engine 21.0.4 - Planned release....

237242 JCP Rest OutOfMemoryError with Arrays.copyOf causes login impossible due to ErtEstimationResource

Under some circumstances, while displaying a Workflow Monitor the JCP Rest process hangs and eventually dies as it reaches out the maximum memory limit. This could also be seen while doing a Search in AWI. Symptoms on versions prior to 12.3.9/21.0.3....,
Fix version: Component(s): Automation Engine and Automic Web Interface
Fix Version/s: Automation.Engine 12.3.9 HF1 - Available Automation.Engine 21.0.3 HF3 - Available Automation.Engine...

This raises several questions:

1. Why was v12.3.9 HF1 removed?
2. Is there a serious problem with this hotfix?
3. When will the hotfix be made available again?
4. Alternatively, when will HF2 be made available?
5. Also, why was an announcement not made explaining the withdrawal of this hotfix?

Thanks in advance for prompt answers.

[Michael A. Lowry]

In a support ticket, I received a partial answer.

AE 12.3.9 HF1 was temporarily withdrawn due to a problem with the Windows Agent. An updated version will be made available soon.

[Donato Faggella]

Hello Michael.
Thanks for the inputs. Did you have any specific information about the problem with the Windows Agent ? 
Thanks
Bye

[Matthias Schelp]

Hi @Michael A. Lowry,

thanks for the warning. I've installed this version some weeks ago in our test environment. So far we didn't have issues with the windows agent, but I wouldn't go to production with this release after you warning. Would be glad if someone could give details on the error.

​Regards, Matthias

[NorbertZ]

Hi Michael,

thank you for the information!

We already deployed that version on some of our systems.
So far, we did not observe any issues with the Windows agent 12.3.9 HF1. 

@Broadcom Support: 
Do you have more details about the problem with the Windows agent? Is the Windows agent affected or the AE?
We need that information in order to keep an eye on potentially affected agents.
Also, we probably have to decide whether to rollback the Windows agent to 12.3.9.

Kind regards
Norbert

[Michael A. Lowry]

Automation Engine v12.3.9 HF1 is back on the downloads site.

As far as I can tell, the build numbers are the same as last week. I was not able to find an announcement of what if anything was changed.

[Kaj Wierda]

We pulled 12.3.9 HF1 from our download center. as we discovered an escaped defect in 3 of our agent components. We have re-published 12.3.9 HF1 without following components: Windows Agent, AS/400 Agent, z/OS agent.

Once we have addressed the issues in the affected agents, we will publish 12.3.9 HF1 once again. At this time I cannot provide a date as to when we will re-publish 12.3.9 HF1. 

------------------------------
Kaj Wierda
Sr. Product Line Manager | Automation

Broadcom Software
------------------------------

[NorbertZ]

Hello Kaj,

could you please provide more information about the "escaped defect"?
We still need to decide whether to asap rollback our Windows agents to 12.3.9 or to wait for a 12.3.9 hf2 fix. 

- What is the impact of the issue? Job aborts? Agent crash? Security leak?
- In which cases does the issue occur? What is needed to reprocude the issue?
- What do you suggest to users with already installed Windows agents 12.3.9 hf1?

As said, the Windows agent 12.3.9 hf1 seems to be stable so far. 
Even more surprising, why Broadcom decided to do this unexpected step and revoke the agent. 

Kind regards
Norbert

[Donato Faggella]

Hello
I opened a case to receive more information about Broadcom about this problem. There is a KB article with all information

JOBF in Connecting with error U02000063 and Server routine UCFTXEX not found causes Force Traces (broadcom.com)
Regards
Donato

[Kaj Wierda]

Some additional details about escaped defect:

1)

we discovered that changes meant for 12.3.9 HF2, the hotfix that will address CVE-2022-33756, somehow made it into 12.3.9 HF1. So an escaped fix, rather than an escaped defect. These changes lead to AS/400 agents and z/OS failing to connect to AE at startup.

2) 

As already pointed out by Donata Fagella, there is a KB article that explains the issue for Windows agents related to file transfer.

If you are not experiencing 1) or 2) then no action is required until we re-issue 12.3.9 HF1

------------------------------
Kaj Wierda
Sr. Product Line Manager | Automation

Broadcom Software
------------------------------

[Michael A. Lowry]

Thanks for this information.

In future, we would appreciate a more timely announcement when download files are withdrawn, so that we can better judge the risk of installing or continuing to use the affected components.

[Michael A. Lowry]

Automation Engine v12.3.9 HF1 was briefly unavailable again this morning, but now it appears in the list again.

The Windows agent is available. The AS/400 agent is not available despite being listed in the CSV file.

[Michael A. Lowry]

Automation Engine v12.3.9 HF1 is unavailable again today.

[Kaj Wierda]

This is an effect of our automated CI/CD pipeline. Every time we re-build a release that is publicly available on the download center, it is rolled back (meaning removed). This does not indicate an explicit removal due to new issues we found, nor can this be anticipated, as its part of our ongoing software development lifecycle.

We will provide an official announcement once 12.3.9 HF1 is officially re-published.

------------------------------
Kaj Wierda
Sr. Product Line Manager | Automation

Broadcom Software
------------------------------