Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

[ArnisR]

I performed a clean install of ESXi 6 with provided HP image (ESXi-6.0.0-2494585-HP-600.9.1.39-Mar2015).
Access to and from datastores over network through vSphere Client (6.0.0-2502222) is very slow - ~10 Mbps (  there are several hundreds under ESXi 5.5)
I'm suspecting that ESXi 6 limits CPU resources used by host system to 230 MHz.
I can't edit "System Resource Reservation" parameters (contrary to ESXi 5.5, where I can edit them and switch to Simple or Advanced view).

[kubasedeep]

Hello, I also my Microserver G8 slow down with HP's customized imaged. I have installed HP's 5.5U2 image and after ugraded with ESXi 6 standard image with better results, but it cannot detect my DVD ROM device. Hope that helps.

David

[ArnisR]

Thanks for the answer. I'll also plan to try 5.5->6 upgrade scenario.
I  haven't DVD drive. I have installed SSD drive in that place and configured through RAID controller as RAID0 (single drive).
ESXi 5.5 and 6.0 see them fine.

Arnis

[Alistar]

Hi there, I can update the system CPU reservation in "thick" client that came with vSphere 6.0 by connecting to the standalone host with it:

[ArnisR]

"thick" client is free or paid option?
I've lost in vmw vSphere vCenter vWeb vClient vVirtual vJungles :/

[nthorst]

Hey ArnisR,

I finally found the problem. In the last ESXi 5.5 and the 6.0 release they have replaced the hpvsa driver (HP vSphere ISO !!). With these drivers, I have experienced major problems with the data stores and thus the B120i (RAID) controller of the HP MicroServer Gen8.

The problem is caused by the last (HP) ISO's (from march);

- VMware ESXi 5.5.0-Update2-2403361-HP-550.9.2.27 Mar2015.iso

- VMware ESXi 6.0.0-2494585-HP-600.9.1.39 Mar2015.iso

I have found out (because I had an older ISO), the ISO VMware ESXi 5.5.0-Update2-2068190-HP-5.77.3-Nov2014.iso works well! With this ISO, all performance problems disappeared :smileyhappy:.

The difference between the ISO's is the following. The older ISO contains the storage driver scsi-hpvsa 5.5.0-88OEM. ISO(s) of March come with newer storage driver scsi-hpvsa 5.5.0-92OEM or scsi hpvsa-5.5.0-90. These drivers are the cause of the problems in the B120i controller!

When installing ESXi 5.5 or 6 from the last ISOs, you must manually install the older storage driver (SCSI hpvsa-5.5.0-88). I've tested this with the vib file scsi HPSA-5.5.0-88OEM.550.0.0.1331820.x86_64.vib. I then adjusted the HP ISO and equipped with the older driver. Now I can install ESXi 5.5 or 6 from the (customized) HP ISO, which includes the older driver.

I hope this helps!

[ArnisR]

Thanks, nthorst

I made some investigation now and found that hpsva ...-90 driver is marked as hotfix and urgent. I didn't find yet what the bug was in -088, but at this moment I don't want to risk with older one.

Arnis

P.S. There is also newer HP iso for ESXi 6 (...HP-600.9.2.38-Mar2015) from 30 March, but it didn't help me also

[nthorst]

Hi Arnis,

ESXi 6.0 iso also uses the wrong storage drivers. As I indicated before :smileywink:.

These are the fixes in the new releases:

5.5.0-90.0(6 Oct 2014)

• Fixes an issue where HDD logical volumes were not being detected at installation.
• Fixes an issue where an updated driver was not persistent after a reboot. Previously, after updating the driver and rebooting, the system would revert to the originally installed driver.

5.5.0.92-1(30 Mar 2015)

• Spin lock update to fix a potential boot hang.

possible, the older driver is stable enough to use :smileyhappy:

[ArnisR]

Ok, thanks nthorst for changelog. (BTW, where they are?)

Installed 088 driver and indeed in Ubuntu 14.04 LTS speeds goes up tenfold.
vSphere client however still doesn't work fast - ~100-150Mbps from SSD datastore to desktop SSD.

Also problem is still when I copy from one VM to another (Ubuntu<->Win10) on the same SSD. Speed often fall below 1 MB/s and even freezes for some time.

I decided to try Hyper-V. I also have large HDDs and could stuck in 2 TB limit with esxi.

Thanks again

A

[yuanlinios]

Hi nthorst

Thanks a lot for the post! Do you know if this driver issue also applies to AHCI mode?

I am using AHCI mode for 2 local SATA disks, and got about 80MB/s to 120MB/s during VM copy between these 2 local disks in the same ESXi. Is this performance normal?

[flippyg]

Hey nthorst,

I have the same problem I think...

I upgraded my ESXI 5.5.0 to ESXI 6.0 today and my top speed is 20 MB/s sometimes lower... Got RAID 10 with 4 hard disks...

How can I downgrade the storage driver?

Can you help me maybe?

Thank you very much!

Edit:

Found my old installation ISO: VMware-ESXi-5.5.0-1331820-HP-5.71.3-Sep2013.iso

This one worked fine. But I dont know how to install only that driver package :smileysad:

Edit 2:

Installed now "scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib" from here.

ESXI 6.0 running better now! Speed collapsing to 120 MB/s (I think it should be faster :smileysad:, maybe some of you guys have any idea) but it runs smooth again.

[username2015abc]

How to install scsi-hpvsa-5.5.0-88 on ESXi 6 on HP microserver gen8

(if an other  scsi-hpvsa-driver is installed)

1. Stop all VMs

2. Enable ssh-conection

3. Copy "scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib" to /tmp  (using WinSCP)

4. Start ssh-conection (using putty)

5. chage directory to /tmp

cd /tmp

6. Copy the vib file to /var/log/vmware

cp scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib /var/log/vmware/

7. Start maintenanceMode

esxcli system maintenanceMode set --enable true

8. Deinstall the running  scsi-hpvsa driver

esxcli software vib remove -n scsi-hpvsa -f

9. Install scsi-hpvsa-5.5.0-88

esxcli software vib install -v file:scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib --force --no-sig-check --maintenance-mode

10. Restart ESXi, stop maintenanceMode, disabel ssh and start VMs again

[speedunlock]

ls how to install this driver

!i used mac os

[Twister5800]

Thanks a lot for this thread - I was almost going to throw out the server, but know....vrrroooommm :-)

Br. Martin

[pauliem]

Just thought I would add my results to this. I was having performance issues with Microserver, although apparently not as severe as some of you were having. I decided to run a basic benchmark before and after changing the drivers. Results were quite dramatic and I could tell as soon as I booted the VM that the performance had changed.

I ran Crystal Disk Mark (all default settings) and the output is as follows:

ESXi 5.5.0 U3a

Before (v100 drivers):

After (v88 drivers):

Don't really understand the significance of the numbers that Crystal Disk Produces, but the numbers seem a lot higher!

Thanks for the help all.

[YourAndroid]

Just wanted to share my results with you guys.

I've startet with this setup.

v88 with ESXi 6.0 Build 4510822

Then i have updated my ESXi to the latest build and installed the latest hpvsa driver (v102)

v102 with ESXi 6.0 Build 4600944

And because of the bad results, i've switched back to the old driver (v88) with the new ESXi Build

v88 with ESXi 6.0 Build 4600944

Keep at v88 if you want to have good performance.

[leofmc]

Just wanted to update to check if HP has fixed this slowness with the latest drivers and you saved me a lot of time :smileyhappy:

[Migueis]

Where can I found a good iso working well with Ml310e?

[hbokh]

Date: 20170127.

The issue returned in VMware ESXi 6.5.0 (Build 4564106) on HP Microserver Gen8 and it seems 5.5.0-88 isn't working on ESXi 6.5.

Anyone got a workaround?

Update: 20170219

Got scsi-hpvsa 5.5.0-88 working on VMware ESXi 6.5 - see steps taken here:

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

[SamBGB]

I would be interested in this too,

[hbokh]

I finally managed to get the driver "hpvsa-5.5.0-88" working on ESXi 6.5 by taking these steps (in maintenance mode):

[1] Remove the current driver:

[root@esxi01:/var/log/vmware] esxcli software vib remove -n scsi-hpvsa -f

Removal Result

   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

   Reboot Required: true

   VIBs Installed:

   VIBs Removed: HPE_bootbank_scsi-hpvsa_5.5.0.102-1OEM.550.0.0.1331820

   VIBs Skipped:

[2] Install the hpvsa-5.5.0-88-driver instead:

[root@esxi01:/var/log/vmware] esxcli software vib install --viburl=/vmfs/volumes/datastore1/VIBs/scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib

Installation Result

   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

   Reboot Required: true

   VIBs Installed: Hewlett-Packard_bootbank_scsi-hpvsa_5.5.0-88OEM.550.0.0.1331820

   VIBs Removed:

   VIBs Skipped:

[3] Disable fallback on AHCI-driver:

[root@esxi01:/var/log/vmware] esxcli system module set --enabled=false --module=vmw_ahci

[4] Reboot and exit maintenance mode.

References:

http://homeservershow.com/forums/index.php?/topic/11872-esxi-65-b120i-hpvsa-hp-microserver-gen8-low-performance/#entry124634

http://homeservershow.com/forums/index.php?/topic/9602-esxi-notes/

Now provisioning a new VM with "salt-cloud" (SaltStack) has given me a speed-boost of app. 3x !!

[t40t40]

Hi,

I have same problem. -I think.

I am on older HP Microserver Gen 7 N54L I also see realy slow storage.

Is there any place I can get the correct driver / guide to fix this problem.

Thanks

[Tittan]

Hello to all,

I have this same problem with my HP Proliant Microserver Gen8 1610T with HP Dynamic Smart Array B120i Controller and my raid volumes.

I have ESXi 6.5 HP custom with driver scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib and ahci disabled. I have RAID1 SATA6, RAID0 SATA3 and RAID0 SATA3 SSD volumes (datastores) and my write speed does not go beyond 35MB.

I don't know what else to do and any help would be very apreciated.

[darksync]

6.7 here.

loaded the driver and made chnace to ahci fallback.

 

Still very slow HDD and ssd

 

Any help?

[ijdod]

Just for reference: driver version 100 in the June 2015 HP ESXi 6 image hasn't fixed the problems, version 88 as described above works like a charm.

[DaeronBox]

I was a little skeptic about this hp faulty driver, so i made some test in my lab made of two identical MicroServer Gen8, same G2020T cpu, 16 MB memory and same raid-zero lun as datastore built on two WD black 1TB drives.

As testing VMs I used two clones of CentOS 7 minimal with fio 2.1.10 installed, testing to /dev/sdb1 (thick eager zeroed vmdk).

No other vms were running during test.

fio command lines used

fio --filename=/dev/sdb1 --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=8k --rwmixread=70 --iodepth=16 --numjobs=16 --runtime=120 --group_reporting --name=8k7030test

fio --filename=/dev/sdb1 --direct=1 --rw=read --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --iodepth=16 --numjobs=16 --runtime=120 --group_reporting --name=4kseqread

	host1 : scsi-hpvsa 5.5.0.100-1OEM	host2 : scsi-hpvsa 5.5.0-88OEM
8k7030test	read : io=358192KB, bw=2971.9 KB/s, iops=371, runt=120529msec write: io=153376KB, bw=1272.6 KB/s, iops=159, runt=120529msec	read : io=341192KB, bw=2832.1 KB/s, iops=354, runt=120438msec write: io=145792KB, bw=1210.6 KB/s, iops=151, runt=120438msec
4kseqread	read : io=5732.3MB, bw= 48'906 KB/s, iops=12226, runt=120024msec	read : io=43512MB, bw= 371'288 KB/s, iops=92821, runt=120005msec

The randow rw test are equivalent and i even got some more iops with driver version 100,

but the sequential read test was very bad.

After downgrading hpvsa to version 88 and rebooting, 4kseqread test now gives me : read io=39374MB, bw=335981KB/s, iops=83995, runt=120005msec.

I think I'll stick with driver version 88.

[Baltasar]

I was trying to report this problem to HP. I made a lot of effort to give all the information that HP needs to take a look.

They closed my case after I send my system config, because the harddisks and USB Stick are "not certified by HP".

<HP>

How i promised i forward the information to my colleague of our second technical Level.  Unfortunately the Microserver is in not support configuration and that’s why it is not possible for us to reproduce the issue in the lab.

From the Report is to be seen that the HDD’s are not HP branded:

Slot 0,1I,0,1       WDC WD60EFRX-68M  SATA  WD-WX31D45RFL3U         0A82      Unknown

Slot 0,2I,0,2       WDC WD60EFRX-68M  SATA  WD-WX31D45RF8DC         0A82      Unknown

The firmware on the hard drives didn’t correspondent to the firmware on the controller. On the other side HP recommends to use also HP branded USB in order to use it with VMware.

HP Flash Media Kits for USB Drives

NOTE: HP qualified blank USB key for use with HP ProLiant servers that support the VMware virtualization environment and other HP value-add software. HP recommends this industry standard USB flash device for use with VMware ESXi. USB device must be installed in the internal slot of the ProLiant server for use with VMware ESXi. Refer to HP VMware Getting Started Guide for installation instructions.

HP 8GB USB Enterprise Mainstream Flash Media Drive Key Kit 737953-B21.

I am sending you the quickspecs of this Microserver with all supported components:

http://www8.hp.com/h20195/v2/GetPDF.aspx/c04128132.pdf

If you can test the server with supported configuration and you still see the slow performance please provide us new AHS report with the more information about the test so I can forward the case to Level 2 / 3.

Please do not hesitate to contact me if you have more questions.

Thank you.

Mit freundlichen Gruessen / Regards

Andrey Andreev

Response Center Engineer

Industry Standard Server Support

Customer Solution Center Sofia

</HP>

HP can of course do that, but it is plain stupid, because i am really sure the problem is not caused by WD harddisk.

Does anyone have HP certified hardware and can open a case with HP ?

[ijdod]

Just for information: The 28 Oct 2015 ESXi 6.0 U1 image by HP still uses the v100 drivers.

[ESXiDK]

I just came by this thread, and wondering if this release could fix something:

Drivers & Software - HPE Support Center.

Changes in my case:

Currently installed: 2.0.10.1OEM - build 4240417

This new update is build 4593811

Not updated it yet, but looking for solutions to the B120i slow disk speed with VMware ESXi 6.5, using the HPE Custom Image from november 2016, and latest HPE Proliant Support Pack.

[hbokh]

Makes no difference, since the offline bundle contains a VIB with the same version 4240417... :smileyangry:

# esxcli software vib update -d /vmfs/volumes/datastore1/VIBs/VMW-ESX-6.5.0-nhpsa-2.0.10-offline_bundle-4593811.zip

Installation Result

   Message: Host is not changed.

   Reboot Required: false

   VIBs Installed:

   VIBs Removed:

   VIBs Skipped: Microsemi_bootbank_nhpsa_2.0.10-1OEM.650.0.0.4240417