ESXi

 View Only
Expand all | Collapse all

Intermitent network disconnect in VM using E1000E NIC

  • 1.  Intermitent network disconnect in VM using E1000E NIC

    Posted Jan 30, 2013 03:30 AM

    Hi,

    We just installed three new Windows Server 2012 and one new Windows 8 VMs on ESXi 5.1. On those VMs, we get a lot of errors in the System Event log about network disconnects. According to the logs, the network disconnects for about one second as we get another event saying that a network connection was established at 1 Gbps just after. Just on the day of January 28, this problem occured 85 times on a single server VM.

    Those VMs are using the recommended E1000E NIC with vmx-09 hardware version. The Tools are installed and are current. All VMs are connected to the same vSwitch which has a dedicated physical port to the gigabit network switch (no NIC teaming and no VLAN tagging). The VMs are running from the same datastore that uses local disks setup in RAID 10 with a hardware RAID controller (LSI 2108 chipset).

    Of course, when the alledged disconnect occurs, Windows shuts down momentarily network bound services (such as NetBIOS Helper service) and brings it back right away when the reconnection occurs. This annoying problem also causes interruptions in file transfers on network shares, so it is not just a harmless error that can be ignored.

    Here is an example of the disconnect event:

    Log Name:      System
    Source:        e1iexpress
    Date:          2013-01-29 10:53:13
    Event ID:      27
    Task Category: None
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      --- censored by me! ---
    Description:
    Intel(R) 82574L Gigabit Network Connection
    Network link is disconnected.

    Here is the reconnection 1 second later:

    Log Name:      System
    Source:        e1iexpress
    Date:          2013-01-29 10:53:14
    Event ID:      32
    Task Category: None
    Level:         Information
    Keywords:      Classic
    User:          N/A
    Computer:      --- censored by me! ---
    Description:
    Intel(R) 82574L Gigabit Network Connection
    Network link has been established at 1Gbps full duplex.

    Here are some interesting facts about this problem:

    • We checked the physical link to the host and everything is fine. We don't have any disconnection reported by the host or the switch.
    • The problem occurs in all VM which have E1000E NIC, but not at the same time.
    • Changing the VM NIC type to E1000 solves the problem: no more network disconnection reported by the VM and file transfers are not interrupted anymore.
    • The disconnects are not evenly spaced in time. It may occur many times in the same minute and then nothing for more than an hour.

    Has anyone had this kind of problem before? Any clue about what may be the cause?

    We can still use the workaround we found and change the NICs to E1000 but it is not the one recommended for Windows 8 and Windows Server 2012.

    Thanks for your help!



  • 2.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Jan 30, 2013 06:49 AM

    Try changing the NIC's network adapter link speed from auto negotiate to manuall as per your requirement to 100 MBps or 1000 MBPS.

    This might solve your issue.



  • 3.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Jan 30, 2013 05:22 PM

    Forcing the NIC to 1 Gbps did not do anything. The disconnections still happen.

    However forcing the NIC to 100mbps solves the problem, but it is not a viable solution for a server, expecially a file or database server. I would prefer to change the NIC type to E1000 and run at 1 Gbps.



  • 4.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Feb 12, 2013 10:04 PM

    I am getting the exact same issue. Also a Server 2012 machine on ESXi 5.1 with the latest guest tools installed.



  • 5.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Feb 13, 2013 03:14 PM

    Here is an update about this issue.

    The E1000 NIC also show disconnects but not as frequent as the E1000E. So both Intel NICs are problematic. I tried installing the latest Intel drivers (downloaded from www.intel.com) without any significant results, the network still disconnects as often as before.

    In the meantime, I have switched all NICs to VMXNET3 and it works very well.

    But I am still looking for an answer to this problem because two of the three NIC types recommended for Windows Server 2012 are not viable.



  • 6.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Feb 13, 2013 03:54 PM

    Is anything logged in the vmware.log for the virtual machine or the vmkernel.log for the host?

    I know there's an MSI X issue doing the rounds. Would be interesting to see if that's what's being hit

    Also could you run the following command and share the result?

    esxcfg-nics -l



  • 7.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Feb 13, 2013 06:52 PM

    There is no matching event in vmware.log nor in vmkernel.log. Here are the entries in vmware.log just before and after the time of the event I reported earlier. My VM is using the EST timezone and ESXi is using UTC, so that's why there is a 5 hour difference in the times.

    2013-01-29T15:47:00.501Z| vcpu-0| I120: CDROM: Emulate GET CONFIGURATION RT 2 starting feature 81
    2013-01-29T16:04:44.590Z| vcpu-0| I120: CDROM: Emulate GET CONFIGURATION RT 2 starting feature 42

    I reproduced the error today (February 13). The issue occured at 13:14:54. There are no entries in vmkernel.log matching that time. Here are the entries before and after:

    2013-02-13T18:13:54.260Z cpu3:4099)ScsiDeviceIO: 2329: Cmd(0x412400759580) 0x1a, CmdSN 0x4c1 from world 5059 to dev "naa.600605b000f90bb018a2f4d06a773444" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
    2013-02-13T18:16:57.722Z cpu1:4097)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124007da540, 0) to dev "mpx.vmhba37:C0:T0:L0" on path "vmhba37:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    Here is the output of the esxcfg-nics -l command:

    Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description
    vmnic0  0000:06:00.00 e1000e      Up   1000Mbps  Full   30:85:a9:a7:cc:0d 1500   Intel Corporation 82574L Gigabit Network Connection
    vmnic1  0000:07:00.00 e1000e      Up   1000Mbps  Full   30:85:a9:a7:cc:0e 1500   Intel Corporation 82574L Gigabit Network Connection

    vmnic0 is used for VMKermel (management) - vSwitch0

    vmnic1 is used for VM network traffic - vSwitch1



  • 8.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Feb 13, 2013 08:51 PM

    Exactly where are you installing the Intel network drivers?  If you are installing them in the guest OS that is a problem, as the driver for the NIC is built into the VMware Tools.  It is a driver optimized for VMware that emulates an Intel NIC to the guest OS.  I also have to question use of the E1000/E1000E NICs for that vmware/os combination.  I would stick with the vmxnet adapter.



  • 9.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Feb 13, 2013 09:06 PM


  • 10.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Feb 13, 2013 09:33 PM

    EnsignA wrote:

    Exactly where are you installing the Intel network drivers?  If you are installing them in the guest OS that is a problem, as the driver for the NIC is built into the VMware Tools.  It is a driver optimized for VMware that emulates an Intel NIC to the guest OS.  I also have to question use of the E1000/E1000E NICs for that vmware/os combination.  I would stick with the vmxnet adapter.

    I tried installing the Intel drivers inside only one VM (guest OS) but that was AFTER installing a bare Windows with the VMware tools and seeing that the problem was occuring. I since uninstalled the Intel drivers because it did not make any difference.

    Strangly, the driver used for the E1000E adapter in the guest OS even after installing the VMware tools is provided by Microsoft and not by VMware. The tools were installed using the "Complete" option.

    Driver provider: Microsoft
    Driver date: 2012-02-29
    Driver version: 12.0.150

    So far, I came to the same conclusion as you regarding the E1000/E1000E vs the VMXNET3. However, after reading a few articles on the web comparing the performance and overhead of the E1000, E1000E and VNXMET3 adapters, it seems that there are no significant performance degradation when using the emulated Intel adapters, especially between the E1000E and VMXNET3 on a gigabit physical network. So, since I don't need 10 Gbps link, I would prefer to use the driver that is actually recommended by VMware, the E1000E.



  • 11.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Mar 05, 2013 06:31 PM

    Has anyone else come across a solution for this?

    I'm experiencing the same issues that only happen with ESXI 5.1 and Windows Server 2012.

    Our Windows Server 2008 boxes on the same host with the same driver do not have this issue. It's only Server 2012.



  • 12.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Mar 21, 2013 07:22 AM

    Hello,

    just to let you know you're not alone...

    Same issue here, but still no solution.

    ESXi 5.1 build  914609 + Windows Server 2012

    Cheers

    David



  • 13.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Mar 30, 2013 04:56 PM

    Guys, would help if a parallel support ticket was opened with both MS and VMW if there are multiple instances of the issue being reported



  • 14.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted May 13, 2013 11:00 AM

    same problem here out of the blue. didnt change the config for months and now this exact same thing happens on different vmware-systems at the same time :smileysad:



  • 15.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted May 23, 2013 01:12 PM

    Hello ,

    Just a help for everyone with this problem :

    After 10 days of research for our virtual datacenter behind a vcloud as iias ,  i found a solution .

    I only run windows 2012 server .

    Install vmware tools , remove e1000 network card after a vm stop, add a card same vswitch etc BUT with a vmxnet3 TYPE . DONT USE DEFAULT CARD

    Start and enjoy this solution . E1000 and E1000e are just unstable in windows8 / windows 2012 , the network card reset randomly with or without heavy IO. No log in windows , and as a vcloud user i don"t have esxi logs (damn i don't like cloud) and provider don"t know why ...



  • 16.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted May 23, 2013 01:36 PM

    And read that kb :

    Choosing a network adapter for your virtual machine (1001805)

    When we talk about windows 2012 we talk about vmxnet3 .



  • 17.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Nov 06, 2013 03:44 PM

    vSphere 5.1

    ESXi build 1065491

    We are experiencing a similar disconnect issue with our new Server 2012 VMs, however ours is a complete drop of connectivity which does not come back.  Windows claims it has "Internet" connectivity, but the VM is completely dark. 

    Our NIC type was the old e1000; we are trying VMXNET 3 now.  Thank you for the tip!



  • 18.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Sep 11, 2014 03:12 AM

    Hi Leyuka,

    Thank you for your reply and sorry such us delayed.

    It seems  vmxnet 3 is  right solution as you said and I found article as below;

    Choosing a network adapter for your virtual machine

    http://kb.vmware.com/kb/1001805

    -----

    VMXNET 3 is supported only for virtual machines version 7 and later, with a limited set of guest operating systems:

    32- and 64-bit versions of Microsoft Windows 7, 8, XP, 2003, 2003 R2, 2008, 2008 R2, Server 2012 and Server 2012 R2.

    -----

    Many thanks!





  • 19.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Oct 27, 2014 01:52 PM

    Greetings,

    Ditto.  After switching VNIC from E1000 and E1000E on two different VMs (Windows Server 2008 R2) from E1000, to VMXNET2 (Enhanced).  The random 'disconnects' reported by the guest OS have stopped.

    ESXi hosts 5.1.0 1483097

    VM Version 7 (both guests)

    VMWare Tools - current.



  • 20.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Apr 14, 2015 06:50 AM

    Some of the VMs were having issues on this and I have tried to apply your solution Leyuka, will monitor for about a week to see how stable it is (on E1000E the disconnections occur 2-3 times per day on a VM).

    Update: The VMs with the VMXNET3 NICs tend to hang and will not be powered on until changed back to E1000E. Not a fix. Message was edited by: shaikhdaniaal



  • 21.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Oct 05, 2015 06:46 AM

    Hello,

    to keep this thread up I´m also raising my that my Windows 2012 R2 machines have the same Problem (only 2012 R2, the 2008 R2 does not have the issue!)

    EventI27 e1iexpress with the Network Adapter E1000e.

    The VMware version is 6.0 an the issue is still there.

    I tried to switch the network adapter from a E1000e to a VMXNET3 in an windows 8 machine, the errors are now gone but now I got an other problem.

    Now the machine does not lose the connection for a second but forever. When i connect via remote console the network of the machine works again and after about one hour of doing nothing the network of the machine is gone.

    So switching to VMXNET3 is not a real option. Anyone got new info on how to solve this problem?

    Greetings

    B. Abraham



  • 22.  RE: Intermitent network disconnect in VM using E1000E NIC

    Posted Feb 12, 2016 06:39 PM

    I am seeing the same e1iexpress warnings in the system logs of my Win 2012 R2 x64 VMs running with e1000e Nics as others have stated.

    VMware support refeerenced me to this article -  Win 2012 virtual machines using E1000/E1000e driver experience loss of network connectivity (2109922)

    They advised that I change the nic type to the VMXNET3 nic as it is the "recommended" nic type for this OS.  Can anyone confirm that the connectivity issues have in fact been resolved after change the nic type to VMXNET3?  I really do nto want to create more problems in doign this as some people have posted.