Brocade Fibre Channel Networking Community

Expand all | Collapse all

latency with vmware virtual machines.

  • 1.  latency with vmware virtual machines.

    Posted 02-17-2015 12:37 PM

    We are experiencing intermittent connectivity on our vms both on the network side and on the storage side. When a user complains about a specific virtual machine I ping the machine and it will have latency up to 3983ms but when I ping the blade that the machine resides on its connectivity seems unaffected in the ping status.  I am not for sure if this is a host issue or if something else is going on. Not only does remote desktop sessions fail when it does work the machine has extreme lag to the point of unavailability.

    .

    We currently have 28 hosts in our main cluster. I think VMWARE's recommended limit  is 33 hosts which we are in that range. I don't know where to begin troubleshooting this issue considering the physical infrastructure seems to be working properly. 

     

    Does anyone have any suggestions that would be a sound process in diagnosing this problem? 

     

    Are there any network diagnostic tools for virtual environments to assist in troubleshooting? 

     

     

     

     


    #BrocadeFibreChannelNetworkingCommunity


  • 2.  Re: latency with vmware virtual machines.

    Posted 02-20-2015 01:20 AM

    Hi,

     

    you are here in the SAN-Community. Are you talking about IP-Latency?

     

     


    #BrocadeFibreChannelNetworkingCommunity


  • 3.  Re: latency with vmware virtual machines.

    Posted 02-20-2015 08:23 AM

    Both we are expericing problems with both. We learned there was a storage issue when we were preparing for our exchange servers. We are over the 10ms that microsoft reccomends for write times somewhere around 40-50ms response time.

     

     Also network connectvity to our virtual servers can be slow and laggish at different times. We think it might be all releted to the storage connectivity.If the server does not have good access times to its disk then that would affect its ability to serve data over the network correct?

     

    Vmware is saying that it looks like the hosts has storage issues. 


    #BrocadeFibreChannelNetworkingCommunity


  • 4.  Re: latency with vmware virtual machines.

    Posted 02-23-2015 02:16 AM

     

    Hi,

     

         Virtual Machines not directly connected to SAN. So proper investigation needs to be done on VM level which includes Virtual machine,ESX server , VM Tools and etc.

     

    " Vmware is saying that it looks like the hosts has storage issues " == > Please be specific. ( Proper output from VMware support/investigation.


    #BrocadeFibreChannelNetworkingCommunity


  • 5.  Re: latency with vmware virtual machines.

    Posted 02-23-2015 09:28 AM

    The vmkernel log is showing these errors:

     2015-02-17T23:05:16.033Z cpu32:1082860)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x89 (0x4136c34824c0, 34126) to dev "naa.60002ac000000000000000e8000062a5" on path "vmhba2:C0:T1:L31" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0. Act:NONE

     

    2015-02-17T23:05:16.033Z cpu32:1082860)ScsiDeviceIO: 2338: Cmd(0x4136c34824c0) 0x89, CmdSN 0x100b9 from world 34126 to dev "naa.60002ac000000000000000e8000062a5" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0.

    2015-02-17T23:05:55.462Z cpu34:32811)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x89 (0x4136c263d480, 34126) to dev "naa.60002ac000000000000000eb000062a5" on path "vmhba2:C0:T1:L34" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0. Act:NONE

    2015-02-17T23:05:55.462Z cpu34:32811)ScsiDeviceIO: 2338: Cmd(0x4136c263d480) 0x89, CmdSN 0x626af from world 34126 to dev "naa.60002ac000000000000000eb000062a5" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0.

    2015-02-17T23:18:40.406Z cpu34:2974725)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x89 (0x4136c2921d80, 34126) to dev "naa.60002ac000000000000000e6000062a5" on path "vmhba1:C0:T1:L29" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0. Act:NONE

    2015-02-17T23:18:40.406Z cpu34:2974725)ScsiDeviceIO: 2338: Cmd(0x4136c2921d80) 0x89, CmdSN 0x10386 from world 34126 to dev "naa.60002ac000000000000000e6000062a5" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0.

    2015-02-17T23:25:05.437Z cpu8:4812392)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x89 (0x412ec8bd3680, 34263) to dev "naa.60002ac00000000000000064000062a5" on path "vmhba1:C0:T1:L14" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0. Act:NONE

    2015-02-17T23:25:05.437Z cpu8:4812392)ScsiDeviceIO: 2338: Cmd(0x412ec8bd3680) 0x89, CmdSN 0x17f from world 34263 to dev "naa.60002ac00000000000000064000062a5" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0.


    #BrocadeFibreChannelNetworkingCommunity


  • 6.  Re: latency with vmware virtual machines.

    Posted 02-24-2015 02:22 AM

    Hi there,

     

    the event you posted:

     

    Spoiler
     2015-02-17T23:05:16.033Z cpu32:1082860)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x89 (0x4136c34824c0, 34126) to dev "naa.60002ac000000000000000e8000062a5" on path "vmhba2:C0:T1:L31" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0. Act:NONE

     

    Cmd 0x89 stands for the COMPARE AND WRITE command, used with VAAI ATS.

    SCSI condition H:0x0 D:0x2 P:0x0 stands for CHECK CONDITION which points at the Target device.

    SENSE CODE 0xe 0x1d 0x0 stands for MISCOMPARE DURING VERIFY OPERATION

     

    The VAAI ATS pimitive compares a on-disk sector to a given buffer and if the information in both is not the same, it reports the above event. 

     

    Are there any other events before/after this one?

    Does your storage array support VAAI ATS operatives?

     

    If this behavior is impacting your ESX performance and if the SAN is not reporting any error I would have the array checked by the storage vendor.

     

     

    Rgds,

    Felipon

     


    #BrocadeFibreChannelNetworkingCommunity


  • 7.  Re: latency with vmware virtual machines.

    Posted 02-24-2015 08:33 AM

    We have had our storage vendor check out the device. We are using HP 3PAR storage and it seems that each device on each end of the circuit reports fine. We have also ran the Brocade Analyzer SAN Health utility and are awaiting to hear from Brocade on this.


    #BrocadeFibreChannelNetworkingCommunity


  • 8.  Re: latency with vmware virtual machines.

    Posted 03-10-2015 03:21 PM

    @konfiguratonking let me know if you have received the analysis from brocade for "H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0" what were the steps taken to resolve the issue?

     


    #BrocadeFibreChannelNetworkingCommunity


  • 9.  Re: latency with vmware virtual machines.

    Posted 03-31-2015 03:26 PM

    any update on this? having the same message with 3PAR 7400 3.2.1


    #BrocadeFibreChannelNetworkingCommunity


  • 10.  Re: latency with vmware virtual machines.

    Posted 07-28-2015 02:17 PM

    We have experienced same problem. Did you ever get solution to this?

     

     


    #BrocadeFibreChannelNetworkingCommunity


  • 11.  Re: latency with vmware virtual machines.

    Posted 08-20-2015 06:02 AM

    Hello,

    we have also the problems (Vmware 5.5 U2 Qlogic -> Brocade -> 3Par) :

     

    2015-08-13T17:50:39.763Z cpu24:33647)ScsiDeviceIO: 2338: Cmd(0x413685b9bf40) 0x89, CmdSN 0x237dba from world 34420 to dev "naa.60002ac000000000000000c90000735e" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0.
    2015-08-17T23:07:00.917Z cpu17:33647)NMP: nmp_ThrottleLogForDevice:2322: Cmd 0x89 (0x413689206380, 34420) to dev "naa.60002ac000000000000000c40000735e" on path "vmhba2:C0:T1:L14" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0. Act:NONE
    2015-08-17T23:07:00.917Z cpu17:33647)ScsiDeviceIO: 2338: Cmd(0x413689206380) 0x89, CmdSN 0x1ca1ff from world 34420 to dev "naa.60002ac000000000000000c40000735e" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0.
     
    has anyone a solution?

    Thanks,

    Frank


    #BrocadeFibreChannelNetworkingCommunity