Symantec 14.x, RealTek NIC and Dell XPS laptops - LAN drop out & BSODs

[DAVID MATTHEWSON]

Hi

We run a couple of small (200 user) sites which all use EPP14.x with avariety of (mainly) Dell Windows 10/11 PCs and laptops, normally with no problems. Recently we have been having issues with 3 Dell XPS laptops, running W11 and using Dell (& 3rd party) USB3 to Ethernet adaptors which use the RealTek NIC chipset and drivers. The problem occures when d/ling large - 100MB and up files, which start off OK but then produce 'Network error' and then fail.

Using Wi-Fi on the same laptop or a non RealTek adapter works fine. The BSOD dumps show the RealTek driver being to blame.

If we fully remove the Symantec client the problem seems to go away. Dell 'blame' Broadcom, of course! 

Has anyone else seen this?

Thanks

[keith knowlton]

Hi David, we are seeing the same issue. So far Broadcom blames Microsoft signing drivers incorrectly. We have provided endless amounts of logs and dumps, not getting any traction on a resolution (for months). If we remove Symantec we no longer see the issue. The issues needs to be reproduced on their end, so they are not putting the diagnostics all on the paying customer. I'm even willing to ship them a laptop so they can see the problem first hand. 

[DAVID MATTHEWSON]

Thanks for that - always good to know one is not alone! Is this on a Dell XPS or other device? We find it only happens on Dell XPS laptops running Win11 and have tried numerous drivers from Dell, MS & direct from RealTek. No problems with non RealTek USB <=> Ethernet drivers / interfaces. 

As you say, remove Symantec and it works fine... Hello Symantec - any thoughts please? Thanks.

[Srt-Rtee]

Well I thought it was just me, April we installed 14.3 RU7 and after a day or so it ground to a halt, tried all sorts of fixes but no network traffic was possible. 

Went back to RU6 and it seemed more stable, but no we have had to restore an HP server four times from a frozen state.

What is going on here? 

Yes uninstall Symantec and the problem goes away. 

[Srt-Rtee]

So glad this is fixed between the manufacturers and our Proliant, five months breaks, restores and finally the HPE August firmware and drivers are out, so the machine is like new, never as good as it is now with SEPM 14.3_RU8 and all the bolts tightened to high torque ! :)

[keith knowlton]

We have seen issues on new Dell Latitudes 3420 and even an Optiplex (can't remember the model) so far all Windows 11.

[DAVID MATTHEWSON]

OK, this may not be 100% helpful but...

We also have some Win10 XPS L/Ts which use RealTek usb<=>Ethernet adapters, and these work fine with Symantec installed. It's just the W11 versions that generate BSODs and other errors.

Edition  Windows 10 Enterprise

Version 22H2

Installed on         ‎06/‎12/‎2021

OS build               19045.3155

Experience         Windows Feature Experience Pack 1000.19041.1000.0

Processor            Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz   2.21 GHz

Installed RAM    32.0 GB (31.7 GB usable)

Device ID             8FB004B9-2759-42B2-91CE-F9B7B4AC6B21

But put the same driver on a W11 box and it BSODs, fails to DL files etc. Remove Symantec and it seems OK.

[Vova_NetWiz]

Same issue here! Here's what I've found with my own, recently deployed Latitude 9430 connected to a WD22TB4 dock, running Win-11-Pro w/SEP 14.3.RU7...

In working with Dell's L4 support, they've determined from the minidump files that the BSOD culprit is the SymELAM.sys file, which presumably controls ethernet traffic. In speaking to Broadcom's support, I was informed that 2 weeks ago there was an engineering meeting between Broadcom, Realtek, Microsoft, Dell, HP, and others to address this specific issue. Unfortunately for us, these wheels do not turn fast enough.

BTW, problem disappears as soon as I disconnect the ethernet cable and operate on Wi-Fi. Additionally, same issue does NOT present itself on multiple instances of Latitude 7420 & 7430 connected to a WD19TB dock w/Ethernet connected, also running Win-11-Pro w/SEP 14.3.RU7.

[DAVID MATTHEWSON]

Thanks - so a real issue - I too will open a case on this - the more we have the quicker this will, I hope, get resolved! It's killing our operations as of now!

and yes, wi-fi or a non RealTek usb<=>Ethernet adapter works fine.

[DAVID MATTHEWSON]

Concerning ELAM.. we find version 2.5.0.120 dated 4/4/2013 on our systems, but interestingly the MiniDumps suggest TCPIP.sys as the defaulting component.

Anyone else care to confirm this? - or not!

Interestingly, SymELAM has been around Symantec & Norton for years [Early Launch Anti-Malware (ELAM)] and has a history of issues, but I wonder if this might be a seperate issue to the RealTek one? More information & reports needed I guess.

[DAVID MATTHEWSON]

Case created under reference 

Case 33481213

[Vova_NetWiz]

Same version of SymELAM.sys on my system. Issue seems to be limited to Win-11. Microsoft must've tweaked/introduced something that triggered it.

[Marvin Philipp]

We have the same issue. Also Windows 11 + Realtek Ethernet Adapter. Bluescreens and Network drop outs. Wifi works fine. Also opened a ticket with the reference to this Post here. 

[Marvin Philipp]

I have also the same version of the SymELAM.sys - 2.5.0.120 - when I uninstall the Realtek driver, it switches to Intel Default Driver. Then the issue does not appear.

[DAVID MATTHEWSON]

Thanks - but the Intel driver can't be using the RealTek chip set I think....doesn't it just manage the Wi-Fi NICs?

[keith knowlton]

In our case the minidumps are pointing to ndis.sys (Network Driver Interface Specification).

[DAVID MATTHEWSON]

Our mini dumps also point at ndis.sys

[Vova_NetWiz]

In my case, it's the Realtek USB GbE Family Controller, because the Ethernet port is on the dock.

[Marvin Philipp]

@ DAVID MATTHEWSON sorry - to clarify: I mean in my case it was also Realtek USB GbE Family Controller because of the docking station. If i plugin directly to the laptop an intel Ethernet Driver is used, which works fine.

[DAVID MATTHEWSON]

Ah - thanks for the clarification - looks like we all have the same problem..

[Marvin Philipp]

Any news at your side from the support? We noticed, with older Realtek Drivers it is working fine as well. For example with Verson 10.45.20.308 from 07.03.2021.

[DAVID MATTHEWSON]

I don't think I'm breaking any confidences to report that Dell had an internal meeting on this today and are looking into the issue. They have asked us for more information of what does & doesn't work.

Broadcom have not, as yet, responded.

[Srt-Rtee]

Locked up solid again, looks like another restore, NIC is active for five minutes before going offline. 

[DAVID MATTHEWSON]

Just a minor update...

We have a case open with Broadcom on this and have been doing tests with them.

Bottom line is that a Dell XPS with W11 using any sort of RealTek copper driver for a dock or usb3 adapter consitently fails to d/l large files from the 'net when 14.3 ru7 6000 is installed.

If we switch from copper to wi-fi and repeat the tests, the d/ls work fine.

An XPS with W10 and the same RealTek driver and 14.3 ru7 6000 also works fine.

So some odd interaction between Microsoft, Broadcom, Dell & RealTek perhaps....

Whatever it is we need to get it sorted!

[John Owens]

Please try the following to work around the issue:

SMB transfer speeds decrease after installing Symantec Endpoint Protection (broadcom.com)

------------------------------
John Owens
Strategic Support Engineer | Symantec Endpoint Security Division (SES)
Broadcom Software
------------------------------

[DAVID MATTHEWSON]

Thanks John - I'm reluctant to do that as (i) the problem only occurs in NICs with RealTek chips (ii) I don't want to degrade the effectivness of Symantec & (iii) it only occurs on W11 machines - W10 ones with the RT chip set are fine. 

Thanks!

[John Owens]

Hi David, 

Understood. This is the fix. So please try and report back.

------------------------------
John Owens
Strategic Support Engineer | Symantec Endpoint Security Division (SES)
Broadcom Software
------------------------------

[keith knowlton]

Not a fix, but did stumble across a work around, not sure what feature made a difference, but I turned off all (on the NIC) the Wake on LAN features and any power saving features and I have not had any BSOD for a day or two. 

[DAVID MATTHEWSON]

Thanks Keith - workarounds are good! We'll try that next week & revert.

[DAVID MATTHEWSON]

Thanks John

We'll look at this next week & revert.

[DAVID MATTHEWSON]

John

Thanks for that. I've now implmented your suggestion and run a scripted 1GB d/l test for 24 hrs on a Dell XPS with a RT usb <=> Ethernet adapter and had no BSODS or breaks in the d/ls, so a good solution until RT come up with a better driver. 

Many thanks - great support!

[Mikkel Elgaard]

Hi David 

We're experiencing the same issues. We're using Lenovo laptops. The built-in ethernet is Intel and works well. But all the docks come with Realtek NICs, and we're getting BSODs...

I hope they find a fix to this soon! 

Best, 

Mikkel

[DAVID MATTHEWSON]

Thanks Mikkel

Interesting to have more evidence that this seems to be RealTek NIC related rather than 'just' Dell or Lenovo etc. I assume you are using W11 too? 

Currently we have support cases open with Broadcom & Dell (as our boxes are XPS laptops) but we've had to pause our own investigations due to pressure of work... we'll resume tests next week I hope & report back here.

[Mikkel Elgaard]

Yep, Windows 11. There is no problem with Windows 10. If I install the old Realtek driver (10.45.20.308) from Windows 10 it works without any BSODs.

We've switched most users to using WiFi and for some, we have bought D-link network adapters for those who really need more speed. 
But it is really frustrating with 350+ clients... 

Right now, I am working on a workaround for installing the old Windows 10 driver. Making sure Windows doesn't update that driver. But we have Lenovo Vantage (Lenovo driver update, etc.) software and that one likes to update the driver too.. So we might have to get rid of the Lenovo software also..  

Please keep me updated :-)

[DAVID MATTHEWSON]

Thanks for the prompt update - and yes, let's keep in touch!

We too have tried with the W10 drivers but as you say, it's very difficult to stop them being 'updated'. And frankly, something we just can't risk for our users.

We have found that usb<=>Ethernet adapters using the ASIX 88179 chipset and drivers also work fine. ASIX Electronics Corporation

Asix		remove preview
ASIX Electronics Corporation ASIX is a Leading IC Design Company for Industrial/Embedded Networking and Connectivity Solutions. email to [e-mail] or call 886-3-5799500 for further products information. View this on Asix >

and these can be purchased for 20 Euro/ USD or so on Amazon etc. They are fast, stable and work well with W11. But like you, we have lots of docks which provide way more functionality but use Realtek chip sets. And crash, corrupt large downloads and so on.

Do you know what chipset the D-Link adapters you use employ? (& perhaps D-Link model number?)

So it's all pointing to the W11 driver for the RT chip set....

More next week..

 

[Mikkel Elgaard]

Sorry - It was not completely correct what I typed. I just spoke with one of our supporters. 
We're using the d-link dub-e250, but it seems to be based on Realtek chipset, why we're also experiencing some BSODs on that one, but not as often. The support says, some users are successful using the dub-e250 and the rest are using the Lenovo USB Ethernet adapter which is based on Intel. Which works better than the dub-e250

The Lenovo USB ethernet adapter costs around 20EUR
 https://www.dustinhome.dk/product/5011104180/thinkpad-usb-30-ethernet-adapter

[Mikkel Elgaard]

Thanks, John. We just implemented option 2. This works and improves performance dramatically. 

Best, 
Mikkel

[John Owens]

Please see: BSOD with DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1) with Realtek Network adapter and Teefer.sys (broadcom.com)

------------------------------
John Owens
Strategic Support Engineer | Symantec Endpoint Security Division (SES)
Broadcom Software
------------------------------

[DAVID MATTHEWSON]

Thanks John - good to have confirmation it is a driver issue.