Issue about the SHA2 Windows Update Situation

[zanda]

https://support.symantec.com/us/en/article.tech255857.html

do i understand this right? we cannot patch Windows 7 and Windows 2008 R2 machines until Symantec releases a new version of endpoint protection?

Most news sites are currently making fun about symantec because it seems that 6 months was not enough for them to test and fix this situation.

Well its not so funny and i wonder why this issue happens and they really didnt test and fix this before when they had multiple months time.

this is a security disaster.

[CCWW]

I would think you could disable SEP and try patching that way. Worst case, you'd have to remove SEP, patch and re-install. I'll be testing it today to see what happens.

[zanda]

well thats only a solution if you have just a few clients. but in a company with 100+ pcs this is impossible.

[Seth G]

Hi, does anyone know if this applies differently to environments where Microsoft SCCM is being used for patch deployment? Thanks.

[CQ]

Symantec issued an update to TECH255857 .  Concerning the SHA2 issue, it now clearly says "This currently affects all versions of SEP." 

[VivaldiR@di0]

I am hearing from our rep that a new SEP Client will be released 8/22. It will be for 14.2 MP1 and 14.2 RU1 MP1 only. There will be no update for 12.x. In other words, if you don't upgrade your 12.x clients to 14.2, they will not recieve Windows Updates. I haven't tried the disabling of SEP and then updating - that sounds like it might be a workaround. No official link or communication to reference yet...

[Torb]

IMHO this must be fixed by an out of band LiveUpdate patch that is pushed out through SEPM/LU. Most systems I know that are still running Windows 7 or 2008R2 are business line computers or OT servers that are close to impossible to touch.

Can someone explain the real issue? Why can't Symantec read SHA2 signed MS updates? Shouldn't that be easy to fix by updating the AV engine through a content update?

[John Owens]

ETA is 8/22. 3rd party applications deploying MS Updates will still get applied though I wouldn't do that.  This only affects Windows 7 and 2008 R2.

If running 12.1 you will need to upgrade to SEP 14.2 RU1 MP1 hotfix version once available.

[John Owens]

@Torb --

 

From my understanding that is not possible.

[Deactivated User]

CQ,

Thanks for pointing that out.  I appreciate it.

I have subscibed to that article, this article, as well as others over the years but only rarely receive notifications of updated information.

-Regards

 

[JJV]

What is SEP actually doing?  I was able to Patch with Sep 14.2 RU1 (14.2.335.1000) without issue, but we use Altiris to Patch, so I'm not sure what protections are in place since we don't use windows update

[Deactivated User]

JJV,

It is not that SEP is doing anything to prevent patch installation.  It is the metadata within the patch bundle from Microsoft that tells the Windows Update client that the patch is not appicable if SEP is installed (or perhaps only if it running at the time, not sure).

Using Altiris bypasses the logic associated with the metadata and effectively force installs the patch.  This is basically equivalent to downloading the patch binary from the MS catalog and performing a (force) install.

Microsoft has explicitly stated that they advise against doing this,  Systems may not start up after a reboot, or more likely a power off - on which is different than a warm reboot.  Loss of data or complete inability to use systems may result.

See the MS KB articles for patches KB4512506 OS Monthly Rollup, KB4512486 OS Security Only and KB4511872 IE Cumulative.

e.g  Look at the last row in the "Known Issues" table at:

https://support.microsoft.com/en-us/help/4512486/windows-7-update-kb4512486

Symptom:

Microsoft and Symantec have identified an issue that occurs when a device is running any Symantec or Norton antivirus program and installs updates for Windows that are signed with SHA-2 certificates only. The Windows updates are blocked or deleted by the antivirus program during installation, which may then cause Windows to stop working or fail to start.

Workaround:

Microsoft has temporarily placed a safeguard hold on devices with an affected version of Symantec Antivirus or Norton Antivirus installed to prevent them from receiving this type of Windows update until a solution is available. We recommend that you do not manually install affected updates until a solution is available.

Here "Symptom and Workaround" are really more like "Notice and Advice / Warning".

If you have seen no problems, consider yourself lucky.  If you want to forge ahead be aware.

-Regards

[ThaveshinP]

So, does this mean that if WSUS is used , similar to Altris - it should still work? Instead of running the update manually?

[mwit]

This affects WSUS and Windows Updates @ThaveshinP, other 3th party programs might work, but is not advised IAMJD mentioned.

[ThaveshinP]

Thanks @mwit

[Torb]

@John. There is starting to be a lot of speculation of what the problem actually is.

Can you describe in detail what the problem is?

is it:

A) SEP is unable to whitelist Windows Updates because they are SHA2 signed something that cause FP detection on the patches

B) SEP interfere with the WSUS update process. Installating patches manually works.

C) Something else

 

We need a detailed technical explanation asap.

[CCWW]

B) was the answer for me. Not sure what's happening behind the scenes but my understanding is that SEP doesn't trust the patches and it just blocks them when they come from WSUS.

When you try and get the updates from WSUS they don't show as being needed for the system even though they are. Manually installing the updates works fine but if you have a lot of systems that would be a lot of work. You could also use PowerShell or something similar to push the patches out and install them.

[Torb]

I would like an offical answers. It might just be that the «compliance» check of wether the patch should be installed or not is only done by Wsus. Doing it manually ignores the check but the risk of errors might still be there. Symantec must answer before people start doing unsupported workarounds.

[Deactivated User]

Withdrawn-

Symantec updated the tech article and addressed my complaint.

[Deactivated User]

TORB,

 

Technically the complaince or applicability check is done by the Windows Update client on each and every endpoint.

WSUS is merely a repository of patches that the WinUpdate client connects to for a source of possible patches to examine and install if applicable.  From the WinUpdate client point of view it functions as an alternative for the Microsoft / Windows Update site.

For our Win 7 / 2008R2 systems reporting into WSUS and that are also running SEP the following is observed:

a) They report back to the WSUS server that the July security patches are NOT applicable.  In WSUS terminology they do not show up as "Needed".  They are not offered for installation.

b) The SHA-2 update, KB4474419,  does show as applicable / needed. 

c) Going to one of the endpoints showing NA in (a) above and directing the WinUpdate client to check with the Microsoft Update site results in the July security patches NOT being offered.  So this mirrors what we see with WSUS

d) For the very few endpoints we have not running SEP, the July security updates are shown as applicable / needed and are offered for installation both via WSUS and Microsoft Update.

e) Downloading the patch from Microsoft Catalog and running the executable on a system with SEP will install the patch.  THIS IS BAD.  The complaince / applicability checking to prevent installation when SEP is detected is not inclued in the executable.  It must therefore be in the metadata that the WinUpdate client uses for checking when obtaining via either WSUS or Microsoft Update.

-Regards

[Brian Broad]

Looks to see how many 2008 R2 servers he has...

 

cries...

[John Owens]

There has been no observed issue in relation to this update. Out of an abundance of caution we worked with MSFT to have the update hidden so that the potential for a False Positive could be prevented. The reason for this is that the version of SymVT that's in use with legacy Operating Systems (Win7/Win2K8R2) does not have the ability to see SHA-2 signatures.

By removing the signature from the evaluation process, there is the potential that the final reputation score is impacted which may result in Conviction/Exoneration variance. For this update, we observed no such False Positives.

However, it's possible a future update may have different behavior, so it's in everyone's best interest to pick up one of the fixed releases as soon as they're available so that this concern can be avoided.

That's why, for customers that have already taken the update (update isn't hidden for 3rd party deployment solutions) they can safely stay on it until we have the updated releases available.

[Deactivated User]

John,

Does this mean that Symantec is working with Microsoft to remove the blocking / hiding of the July updates as delivered by WSUS and Windows Update?

If no, why not? 

The July updates contain fixes for issus that have a very high potential for serious expoit - the RDP vulnerabilities.  We need to patch our systems at the earliest reasonable time that we are able.

If yes, do you have any information from Microsoft when the new revisions will be made available?

-Regards

[John Owens]

Once the machines have the SEP hotfix installed they will be update the MS updates and it will not be blocked. We are not working with MS to remove the block/hiding for updates being delivered by Windows Update.  The hotfix must be installed on these systems once available.

[thedman27]

John,  Are there any Norton products affected by this?  Thanks.  

[CMW]

IAMJD,

I assume that in your post, it should read "August" in place of "July"?  Just want to make sure that I fully understand the situation...

[Deactivated User]

DOH!

 

Yes, you are correct.  August not July.

 

-Regards

 

 

[Scott K]

According to Microsoft Norton is affected:

Microsoft has temporarily placed a safeguard hold on devices with an affected version of Symantec Antivirus or Norton Antivirus installed to prevent them from receiving this type of Windows update until a solution is available. We recommend that you do not manually install affected updates until a solution is available.

Source: https://support.microsoft.com/en-hk/help/4512486/windows-7-update-kb4512486

 

Microsoft announced this SHA-2 change 6+ months ago. It is totally unacceptable that Symantec didn't address this major problem earlier this year.

[John Owens]

Yes. Norton is affected though I am not sure what versions. I would reach out to Norton support for that.

[John Owens]

@ScottK

We acknowledge that the lack of prepared support for legacy Operating Systems (Win7/Win2K8R2) receiving SHA-1 deprecation has had consequences. We're actively working to address this situation as quickly and safely as possible.

[LarsF]

Does this issue also apply for Server 2012 in the future?

I understand that Server 2012 is going to SHA-2 later this year.

[MarceloCtba]

Somebody knows if in addition to the patches, Symantec will be releasing a Full package (RU2 or whatever) on the 08/22?

 

Thanks!

[ultron]

Forgive me for asking what is more of a contractual question.

From what I'm reading, Symantec will make hotfixes available for 14.2 customers - this presumes entitlement to that version. In the case that customers hold perpetual licenses, but do not have active support contracts, and accordingly cannot upgrade to 14.2, are these legacy customers permanently bricked?

 

 

[thedman27]

Thanks John.  

[John Owens]

Other OSs are not impacted as they have a newer driver already that will work with the updates. This only affects legacy OSs.

SEP 14.2 RU2 has an ETA of late October and will not be released on 8/22. It will have the fix in it though.

 

[John Owens]

@Ultron

If you do not have an active license then you will not be able to download the Hotfixes. What license do you currently have?

[John Owens]

SEP 14.2 RU1 MP1 will be a full build.  Others hotfixes will be able to be imported into the SEPM and either pushed out or exported out.

[MarceloCtba]

Thanks John! Can you please confirm this full build will be also released by end of the next week?

[ultron]

14.0 MP2 Build 2415, thousands of perpetual licenses but no active support contract, as this product is complementing other security products but still happily receiving definition updates.

[John Owens]

That is the plan. Current ETA is 8/22.

[John Owens]

@Ultron

You will want to work with our Customer Service/Licensing team to see what can be done.

[John Owens]

Hotfix builds:

SEP 14.2 RU1 MP1

SEP 14.2 RU1

SEP 14.2 MP1

 

[ultron]

Noted - thanks for the candor on this forum.

[[e-mail]]

Who at Symantec would we send the bill to for all the extra work this is going to cause us? /s

They really dropped the ball on this and someone's head needs to roll.

 

[Allen Carter]

What is the deployment method for the hotfix? We are on 14.2 MP1. Does it requred a reboot of the endpoint?

 

 

[Deactivated User]

John,

 

Microsoft has issued new patches today, August 16, for various OS versions including Win 7 / 2008R2.  It is KB4517297.

https://support.microsoft.com/en-us/help/4517297/windows-7-update-kb4517297

This new patch was issued to fix problems with VB, VBA and VBscript that were introduced with the patches from August 13.  In other words Microsoft is fixing their broken patch with this new patch.

 

At the bottom of that page there is this:

"Note This update contains all the quality and security changes in KB4512486 (released August 16, 2019). While it does not replace KB4512486 on Windows Update, if you install this update you do not need to install KB4512486."

 

While KB4512486 was one of the patches that Symantec worked with Microsoft to block installation if SEP was detected, this new patch is NOT blocked via either Windows Update or WSUS.

If this new patch KB4517297 contains all of the content of the blocked patch, this would seem to be contradictory.

I know that Symantec is not responsible for the patch, but what is Symantec's stance on installing this?

Did Microsoft not check with Symantec about this new patch in its haste to correct the broken earlier patch?

 

-Regards

 

 

[RicorGuy]

KB4517297 indeed is not blocked by Microsoft due to the SEP/SHA-2 concern.  I'm glad that the VB known issue is being addressed.  We all look forward to the Symantec release of a SHA-2 capable hotfix.

Rick

[nzdude]

Would a valid workaround be to turn off SONAR on any remaining Server 2008 R2 boxes that users don't log on to, patch manually, and turn SONAR back on when we get the new version of SEP (on Wednesday?)?

[nzdude]

There is a block now, as stated in the Known Issues:

Microsoft has temporarily placed a safeguard hold on devices with an affected version of Symantec Antivirus or Norton Antivirus installed to prevent them from receiving this type of Windows update until a solution is available. We recommend that you do not manually install affected updates until a solution is available.

[Deactivated User]

It depends on how / where you look for your updates.  And apparently also when you looked.

The KB article at:  https://support.microsoft.com/en-us/help/4517297/windows-7-update-kb4517297

has since its original pulication on Aug. 16, 2019 asserted in the Known Issues section that it is blocked for installation on systems with a SEP client.

Our testing on Aug. 16 found that the update was NOT blocked either via Windows Update or WSUS.

Testing on Aug. 19 at approximately 16:00 GMT is showing that it is indeed blocked via Windows Update.

However via WSUS it is NOT blocked.  The WSUS Release Date as well as the Arrival Date (when it was downloaded) both show Aug. 16.

It appears that Microsoft made a mistake with the very first issue of the update and did not place the block with the update.  They then corrected the problem for the patches on the Windows Update web site, but did not republish patches with the block for WSUS synchronization.

-Regards

 

[VivaldiR@di0]

I believe next month Microsoft will expand its SHA-2 only to Windows 8.1, Server 2012, and Server 2012 R2 systems.

[VivaldiR@di0]

I'm being told by my Symantec TAM that a 12.x fix will also be made available: SEP 12.1.6 MP10

[zanda]

there is no problem with office updates, right? they still have sha1 and only the windows updates are the problem?

[S_K]

I see on the MySymantec portal in Downloads section that there is 14.2 RU1 MP1 from today. Any information from Symantec about this?

Does this version include the fix for the issue?

[Alexander.S]

I'm also interested in this, skimming the release notes for 14.2 RU1 MP1 i see nothing on the windows update sha-2 fix?

[S_K]

In the link below, it shows that the latest 14.2 RU1 MP1 version was released on Aug 5th, it is build 14.2.4811.1100.

https://support.symantec.com/us/en/article.tech154475.html

And I downloaded the one from MySymantec showing available from today, it is build 14.2.4814.1101.

Anyone from Symantec to share something?

[John Owens]

SEP 14.2 RU1 MP1 Hotfix is available. 14.2 RU1 and SEP 14.2 MP1 is still slated for 8/22

[S_K]

Thanks for the update John.

Just one additional question. In any case we will have to upgrade the Win 7 and W2K8 machines to the hotfix version (and I think everyone will prefer to upgrade to the latest available) so why you are releasing hotfix also for the older SEP 14.2 versions (like RU1 and MP1) and not just for 14.2 RU1 MP1?

 

 

[Anand_N]

"SEP 14.2 RU1 MP1 Hotfix is available. 14.2 RU1 and SEP 14.2 MP1 is still slated for 8/22". Are these patches or client versions ? Hopefully there would also be client installer packages that can be imported into SEPM, made available to the public

[kays]

Nice! thank you!

My Windows 7 Test VM now shows the August updates.

 

...but i only spottet "EN" language.

What are the plans for other languages?

[John Owens]

14.2 RU1 MP1 has both.  The other ones will be packages that can be imported into the SEPM for deployment.

[almost there]

So is the fix included in the latest release? I see a release from today - 8/20. However, no mention of the fix in the release notes. Thanks. 

[John Owens]

The latest release from 8/20 is a refresh of SEP 14.2 RU1 MP1 that includes the fix. It has not been added to the release notes and I am not sure it will be added. Please see the KB for this issue for more information.

[Deactivated User]

Hi,

Could someone please post the long verion number of the latest release with the fixes?

At  https://support.symantec.com/us/en/article.tech154475.html    last updated date of August. 8, 2019.

The latest version 14.x that I see listed is 14.2.1.1 (14.2 RU1 MP1)     14.2.4811.1100     August 5, 2019.

I'm not seeing anything regarding an August 20 release.

And the Tech article at  https://support.symantec.com/us/en/article.tech255857.html  has a last updated date of Auguest 16, 2019. 

I'm able to check from multiple locations around the globe, so I don't think that I'm seeing cached content.

 

-Regards

 

 

 

[Anand_N]

How can the 14.2 RU1 MP1 hotfix be obtained?

[John Owens]

14.2.4814.1101 is the Hotfix build for SEP 14.2 RU1 MP1.  It should be available for download on My Symantec.

[John Owens]

@SK

We are releasing other versions because some customers are not able to upgrade as testing has not been completed.

 

[John Owens]

Other languages should be available tomorrow.

[Anand_N]

A support contract is needed to access the hotfix?

[John Owens]

Yes. As always a support contract is required. You will only be able to download from My Symantec.

[nzdude]

John,  does the server need to be updated as well, or just the clients?  

We see a few releases of clients to fix security vulnerabilities, but our deployments are always slowed down by the need to update the server first.  Can you please consider relaxing your support policy to allow +/- 1 MP for th server version rather than "the same or newer"? 

We could then grab the updated clients, eliminate the vulnerability, and then get to work on the server.  If there weren't so many new versions to fix holes, this wouldn't be an issue.  :(  

Or.... get the server to self-update via a similar method to definitions.

[John Owens]

This depends on what version your SEPM is currently running.  If you are running SEP 14.2 RU1 SEPM then updating the clients to the RU1 MP1 build and then upgrading the SEPM later should not be a problem.

For some releases of the SEPM we add new Polices/features that only that same build of SEP Client can use.  There could be some reporting issues as well with a newer SEP Client than the SEPM.

SEP 15 offers auto upgrades to the Cloud Portal and clients. May be something to look into.  The SEPM will not be able to be updated automatically with On Prem deployments.

[fe007]

Hi John,

Will a hot fix be released for version 12.1?

 

 

[WHairstonLOI]

@fe007:

I asked the same question in another post: https://www.symantec.com/connect/forums/does-windows-update-sha2-problem-affect-sep-121x?list_context_id=3377631&list_context_type=symantec_product

It looks like they will NOT update v12.1.x, so we're going to have to update to v14.2.x 

[onetrust]

I downloaded the newest version + hofix. On Windows 2008 (no R2) it says 7Symantec Endpoint Protection only be installed on Windows 7 / Server 2008 R2 and later.

Here: https://support.symantec.com/us/en/article.info3982.html it says 

• Windows Server 2008 (32-bit, 64-bit; RTM, R2, SP1, and SP2)

So what? This server version is still supported by Microsoft and Symantec?! But i cannot install the hotfix to get the patches?

--

Ok also found the article / post that Win 2008 without R2 was dropped. Great! Does not help at all.

 

 

[John Owens]

@steppe

 

You will need to use the SEP 14.2 MP1 patch for non R2 2008 Servers.  You can open a case and request this build if needed.

[Truman Devor]

What is the ETA for a patch for users of SEP Cloud (22.18.0.213)?

[John Owens]

@Truman

I don't support that product, so I am not sure. I would open a support ticket to request that information.

[Jforge]

The versions I see for download are 14.2.1 MP1, 14.2.1 and 14.2.0.  Which version would work with server 2008 (not R2) ?

[John Owens]

14.2.1057.0103 is the hotfix version for SEP 14.2 MP1.

[fian]

Can we get a straight answer from Symantec on the hotfix availability for 12.1? I contacted support twice and got two different answers. One person said that Symantec would be releasing a hotfix for 12.1 and another person said that we would need to upgrade to 14.

 

Can someone clarify this either way? If we need to upgrade we need to start planning yesterday.

[fian]

Can we get an answer from Symantec on if version 12.1 will be fixed or will we have to upgrade to version 14?

 

We are getting conflicting information from Symantec on if we should wait or begin the upgrade process now.

[fian]

Will a hotfix be released for version 12.1?

[fian]

Hi John,

Will a hot fix be released for version 12.1?

[AN9]

I think it would be either 32bit if server 2008 is 32bit. Any version build of 14.2 should work

[John Owens]

SEP 12.1 is end of support life. You should go to 14.2... unless you have purchased extended support.

[Jforge]

I spoke with Symantec and for 2008 non R2, the asked to use the latest release 14.2.1 RU1 MP1.   I am getting conflicting answers here.. 

 

Symantec need to update their documentation https://support.symantec.com/us/en/article.tech255857.html with instructions regarding 2008 SP2 (non R2)  this is clearly listed in the Microsoft KB that it imapcts 2008 SP2 (non R2) as well.  https://support.microsoft.com/en-us/help/4472027/2019-sha-2-code-signing-support-requirement-for-windows-and-wsus   "Customers running legacy OS versions (Windows 7 SP1, Windows Server 2008 R2 SP1 and Windows Server 2008 SP2)"

[John Owens]

What is your case number JForge?

[John Owens]

We will work to update the Q/A section of the KB to reflect that.

[onetrust]

@john_owens

Why don't you provide the version just for download? I opened up the case (30331485) and nothing happend.

I think there will be mass of requests for this version. Just add the client binaries to the download central and let us download them.

Due to compliance reasons i need this versions asap

[mwit]

I opened a case (30334507) as well, haven't heared anything from support as of yet.