VMware vSphere

 View Only

  • 1.  Help Troubleshooting Throughput Problem

    Posted Dec 15, 2012 08:17 PM

    I'm seeing some really poor network performance on a specific VM and I'm wondering how to best troubleshoot this.

    • Is it the NAS?
    • Is it the Host?
    • Is it the physical network?
    • Is it the virtual network?
    • Is it the VM itself?
    • Is it the destination VM?
    • Something else?

    While transferring some data to an AWS EC2 Instance, I noticed it was taking an incredible amount of time to transfer a small amount of data.  Even after several minutes the box had only managed to transfer 400MB which to me (on a 25/25 connection) wasn't normal.  I decided to run through some very simple tests on the VM in question then on the destination EC2 instance.  (Just so we're clear: the destination machine is not hosted here, its an Amazon EC2 VM.  So all this data is going out over the Internet to the destination machine.)

    I found a 30MB file on Softlayer's site that I've been using to build a very simple baseline:

    wget --output-document=/dev/null http://speedtest.sea01.softlayer.com/speedtest/speedtest/random4000x4000.jpg

    I ran through these tests from (yes I have wget for Windows)

    1. a physical desktop (Windows 7): 3M/s
    2. the NAS (via ssh): 3M/s
    3. the CentOS VM in question: 600-700K/s
    4. a Windows 7 VM on the same host as the CentOS: 3M/s
    5. the destination EC2 instance: 3M/s

    Note: I realize this only tests downloads & not uploads.  I expected to see maybe high download speeds & a poor upoload speed but when the download itself was low, I figured there may be some sort of misconfiguration somewhere.  I ran the GUI upload test (http://speedtest.sea01.softlayer.com/speedtest/) on the physical and virtual Windows 7 systems, and the results were roughly 18Mbps up and 25Mbps down.

    With 110GB to transfer, I'm a little worried it'll take ages at this rate!  Help!

    I don't know what information might be useful at this point so I'll give you what I have.

    Local VM Info
      -  CentOS release 5.8 (Final)
      -  Linux version 2.6.18-308.13.1.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-52)) #1 SMP Tue Aug 21 17:10:18 EDT 2012
      -  1 vCPU / 2GB RAM

    Remote VM Info

      -  Linux version 2.6.18-xenU-ec2-v1.2 (root@domU-12-31-39-06-0D-E1) (gcc version 4.1.2 20070626 (Red Hat 4.1.2-13)) #2 SMP Wed Aug 19 12:57:15 UTC 2009

      -  Similar CPU & Memory configuation

    Host Info
      -  Dell PowerEdge T710 Dual Quad Xeon E5504 12GB
      -  8x500GB SATA 7200RPM 32MB Cache 2xRAID5
      -  Storage via iSCSI LUNs
      -  ESXi 4.1.0 800380

    Storage Info
      -  Synology DS411+ii
      -  4x3TB SATA 7200RPM 64MB Cache RAID5

    Local Network Connectivity
      -  FiOS 25/25
      -  Gig connectivity all around
      -  No firewalls between host and NAS

      -  No complex network configuration on switches/routers (other than VLAN tagging)
      -  No complex VMWare network configuration (other than VLAN tagging)



  • 2.  RE: Help Troubleshooting Throughput Problem

    Posted Dec 16, 2012 04:28 PM

    For what its worth: I'm scp'ing a 60+GB tar'd.gz'pd file to Amazon and it seems capped at roughly 1MB/s but it shoudl be closer to 3MB/s.

    I can't figure out if its the VM, the host, FiOS or Amazon and I'm not sure what the best testing methodology is for ruling out one or the other.