ESXi

 View Only
  • 1.  Sudden network problems (packet replay?)

    Posted Feb 14, 2022 10:04 AM

    Hello everyone,

    we are currently experiencing some very strange network problems. Since about 1,5 weeks ago some of our employees are complaining about lost connections to their PC via Remote Desktop. Because of the current situation, most of our employees work from home over an OpenVPN server which is running on one of our ESXi servers. After a bit of research (a lot of pings to googles DNS from different maschines in our network) we determined that the problem must come from our ESXi server, as we saw some strange behavior on all VMs running on the server. Here is an excerpt of one of those pings:

     

     

    Normally we should get one reply each second but then suddenly the replies just stop for a couple of seconds, before coming all at once. This would explain why our employees are experiencing connection loss on the VPN side.

    I then went on to check the logs/events on the ESXi host and found these messanges:

     

     

    They basically say, that the connection to our datastores was lost, and its trying to reconnect. The timing of these messages and the length of connection loss fits with the timing of the network problems.

    We suspected a hardware problem at first so we switched to our backup ESXi server. We do nightly replication to that server. With every VM startet on the new server, we experienced the same problem. So the theory of it being a hardware problem seems highly unlikely now, as it is super unlikely for both systems to have the same hardware problem all out of a sudden.

    Later on we moved the VPN VM (and only that VM) back to the old ESXi host and the problems (atleast for the OpenVPN VM) were gone. No connection problems were reported anymore. 

    Did any of you ever experience similar problems? Does anyone have an idea what is going on? I feel like I am going nuts trying to find a credible explanation and a solution... If I can provide anymore information, just let me know

    Thanks in advance and kind regards,

    Jan



  • 2.  RE: Sudden network problems (packet replay?)

    Posted Feb 14, 2022 06:38 PM

    There is a known issue in vSphere 6.5 and 6.7 in which slow storage operations can cause ESXi hosts to go offline: https://kb.vmware.com/s/article/1003659

    I would recommend following the steps in the KB to troubleshoot the issue.

    This issue has been fixed in vSphere 7.



  • 3.  RE: Sudden network problems (packet replay?)

    Posted Feb 15, 2022 07:01 AM

    Thanks for the information. The KB article only references network storage, we are using local storage connected via a raid controller. I am unsure if we can Update to vSphere 7. We have a vSphere 6 Essentials kit (with valid subscription).

    Thanks again for helping.



  • 4.  RE: Sudden network problems (packet replay?)

    Posted Feb 15, 2022 10:37 PM

    AFAIK it applies to any slow I/O operations, which could apply to local storage as well. In most cases it used to be tied to operations like backup which would result in the slow I/O operations.

    If you have a vSphere 6 license, I don't think you can upgrade to vSphere 7 unless a new license is purchased.



  • 5.  RE: Sudden network problems (packet replay?)

    Posted Feb 16, 2022 11:08 AM

    Once again thanks for helping us. I will start/am doing some Performance profiling and will have a look, if the issues where connected to spikes in I/O heavy operations.
    Yeah, that's what I would have guessed too. Will look into upgrading the license then (on the technical side the servers are compatible with vSphere 7 AFAIK).