Task Server Unavailable RE MS22-01-W10-5009543

[Dylan Thomas]

Good day all,

We've been trying to fix the problems/mess caused by :

Bulletin: MS22-01-W10-5009543, Update: windows10.0-kb5009543-x64.msu prevents endpoints from registering to task server (broadcom.com)

(thanks Microsoft)

Our SMP is working no problem, the primary site server is also working without a problem, but we have 2 other site servers that exhibit the symptoms described in this bulletin even after the KB's have been uninstalled, server restarted.  Clients are getting the exact same message and are unable to register to these Task Servers.

Anyone have any other fixes, do we need to modify anything else? 

Kind Regards,
Dylan

[Martin Sapsed]

It's all very well Symantec/Broadcom blaming Microsoft but it sounds like the issue is actually that Altiris persists in relying on NTLM authentication when it's obsolete, insecure and being phased out by Microsoft. As I understand it, this isn't news - we've been talking about switching off NTLM in our security reviews.
As for the workarounds, I have Persistent connections enabled and the test machines on which I've installed the update are still connecting to the task server, although I haven't updated that yet.
Regards
Martin

[Vladimir Zelenjak]

Hi Martin,

Your statement is incorrect - Altiris does not relies on NTLM authentication and does support Kerberos.

If you were impacted by January 11st updates that means that Kerberos wasn't properly configured in your environment including additional steps required for Altiris - registering 3-parts SPNs on the DC. And authentication from Altiris Agent to the Task Server was automatically always downgraded to NTLM.

What Microsoft changed in January 11st update is that automatic fallback from Kerberos to insecure NTLM is not happening anymore. And because Kerberos is not configured properly, authentication just fails now. We can confirm now that according to Microsoft the change was made intentionally.

Besides using Persistent connections, the recommended option going further is to configure Kerberos as required, please refer to the corresponding documentation. If that is not possible for some reasons though and for some customers less secure NTLM is still acceptable, order of Kerberos vs. NTLM type needs to be adjusted manually on the Task Server IIS.

We will be updating shortly https://knowledge.broadcom.com/external/article?articleId=232242

Hope this help,
Vlad.

[Hagai Nachmani]

And if persist connection and/or connection of target to domain is not available or possible, why can't we find a permanent fix that will resolve this as was?

I find that every couple of days, without rebooting or changing anything else the NTLM hierarchy has changed once again and set yet again below the Negotiate bullet. à which means something is keeping (Actively) to change it from the TS or SMP.

 

When is this expected to be permanently resolved? Will Microsoft's Feb22 KB fix this? Is there a fixlet expected by Broadcom? What is the road map?

 

Tnx,

Hagai

---------------------------------------------------------------------
A member of the Intel Corporation group of companies

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[Dylan Thomas]

Good day all,

For some reason the IIS NTLM authentication promotion wasn't working on these site servers, we've now managed to get this working.

Kind Regards,
Dylan

[Hagai Nachmani]

So now previously non connected Windows 2012 targets to any TS are OK?

 

Was the IIS procedure impactful on any other aspect? Is there any risk touching the NS IIS for this action?

 

Tnx,

Hagai

---------------------------------------------------------------------
A member of the Intel Corporation group of companies

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[Chris Farrell]

The KB article is now updated with the recommended resolution steps.
https://knowledge.broadcom.com/external/article/232242/

[Hagai Nachmani]

Thank you Chris,

Yes, seeing all the commotion and the buzz around this KB in the past week.

Still hoping to hear about a permanent solution from MS / Broadcom that won't require a daily check to see if the NTLM configuration remains as was set.

 

Tnx,

Hagai

---------------------------------------------------------------------
A member of the Intel Corporation group of companies

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[Chris Farrell]

Hi Hagai,

We have removed the workarounds involving changes to IIS - if you reload the article you will see the current recommendations for resolving the issues seen. Please see the Microsoft article (which is now linked in the Broadcom article) that explains why various applications are failing. The article explains that in such environments, "... it is likely that Kerberos authentication for 3-part SPNs has not worked for some time." 

https://support.microsoft.com/en-us/topic/kb5011233-protections-in-cve-2022-21920-may-block-ntlm-authentication-if-kerberos-authentication-is-not-successful-dd415f99-a30c-4664-ba37-83d33fb071f4

[JOE.PS1]

We have applied the recommended Kerberos configuration and it has resolved the issues for internal or VPN connected clients.  However, my external clients are still failing to register on task servers over CEM.  I assume this is because the clients can't talk to a DC for Kerberos to work.  What is the proper auth configuration for CEM connected task agents?

------------------------------
Joe
------------------------------

[Hagai Nachmani]

Tnx Chris,
Issue was properly resolved via this KB and the changing of the SPN in active directory Domain Controller:
January Cumulative Security updates prevent endpoints from registering to task server

Broadcom		remove preview
January Cumulative Security updates prevent endpoints from registering to task server After installing Bulletin MS22-01-W10-5009543 and Update windows10.0-kb5009543-x64.msu (from the January 11th MS patches and later), endpoints are not able to r View this on Broadcom >

tnx,
Hagai

[Hagai Nachmani]

hi Chris, noticed that some VMs servers still have the issue appear on them (end targets), while they , the TS and the SMP all have the latest Microsoft KB's and the NTLM on the SMP and TS already set properly (back to it's original settings), and the SPN were properly initiated and implemented.

have you (or any others) encountered such behavior, and know what needs to be done? 
(noticed it on VM win2016 and Win10 - but not all - so i have no idea what is the difference between the working to the none working targets, and have already tried to reinstall the ITMS agent , and still can't find the proper root cause.

tnx,
Hagai

[Chris Farrell]

I would look for what is different network wise with those systems - for example, do they have line-of-sight to a domain controller? Are they hitting the task server and getting denied? If so, they likely are unable to secure a Kerberos ticket. You can test by switching the provider to NTLM on the task server it is attempting to connect to and see if it then succeeds - then you know there is likely a problem connecting to a DC.

[Hagai Nachmani]

Hi Chris,

Already viewed multiple other targets (same subnets, some network configuration) VM and Physical machines that yet again got disconnected from the TS. (yesterday I 1^st only noticed VM's but later saw others as well).

 

Indeed when reverted the NTLM , all targets have successfully return online, but this is a weak workaround, since every restart of the SMP or TS, the NTLM settings are reset back to the original state (since the Jan22 Patches are implemented).

 

Is there any expected additional solution for this issue in Microsoft's Feb22 patches? Or Broadcom's fix let of some sort?

I know you are working on a chance of the entire SMP upgrade to 8.6 RU2 (+) to have the 3-part authenticate to get changed to 2-part, but that will require a full system (ITMS) upgrade, which if not necessary at this point, I wouldn't be so thread to perform just for the NTLM issue without knowing 100% resolves all targets.

 

The problem is the sporadic behavior of this issue, since I can pinpoint and find the root cause that some targets lose connectivity and others don't, if they are on the same network, same image, same OS, physical or VM, and yet , 1 act different than the other (if the NTLM isn't set as priority)

 

Any ideas?

 

Tnx,

Hagai

---------------------------------------------------------------------
A member of the Intel Corporation group of companies

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[Vladimir Zelenjak]

Hi Hagai, All,

Please monitor the official KB (https://knowledge.broadcom.com/external/article?articleId=232242) where we're sharing the latest updates and recommended solutions. If content of the page doesn't answer all your questions and concerns - please engage with the Broadcom Support.

Earlier today we updated the KB with the up to date information.

Kindest regards,
Vladimir.