ESXi installation drive and IBM RAID card

[7007VM7007]

I am setting up my first ESXi 5.5 host and have a couple random questions that I hope someone can assist with:

1. I have two SSD drives that will be mirrored and used as a datastore for my VMs. From what I have read there are two options when installing ESXi: To a USB flash drive or to local storage. I was thinking of installing ESXi to the mirrored SSD drives as it will provide good performance and redundancy. Although my motherboard has a USB port on it, I am not keen on using a USB flash drive for booting ESXi as these drives do tend to fail. So is it ok to just install ESXi on the local storage?
2. I have an IBM MegaRAID M5015 RAID card (a rebadged LSI 9260-8i) which has two Samsung Pro 840 drives connected to it and it is working great. Has anyone tried the Samsung Pro 850 SSD drives connected to this particular RAID card? The 850 drives are not on the compatability list but I was curious to hear if anyone had successfully used them.

Thank you! :smileyhappy:

[JPM300]

Hey 7007VM7007

Yes you can install ESXi to the local SSD mirror.  ESXi will use the minimum amount of spare to install ESXi on it, the remaining space will be turned into your Local Datastore for VM usage.

As far as trying the Samsung Pro 850's I haven't tried them and if they are not on the compatibility list its always a gamble.  However the 840's and 850's are very similar, I believe the controller on the drive is the only difference which gives it the better speed.  So it "could" work.  However if you are planning on using this ESXi server for any kind of production VM's I would stick to hardware on the HCL.

[MMabrouk_VMWARE]

I'M trying to install esxi 7 at IBM HX5 blade server with SSD  raid card  and i have issue 

the esxi loaded and start  the installation process but when it come to the step to choose where i want to setup the os ,the local disks doesn't appear ,only the USB is shown 

i try for 3 days and i can't find solution for this issue 

[Kinnison]

Hello,

You have to make sure, consult the HCL list, that your system is supported for the version of ESXi you intend to install and it is generally advisable to use the "custom" builds released by the manufacturer of the same which include more specific drivers for the underlying hardware.

Regards,
Ferdinando

[E_Espinel]

Hello.
The IBM HX5 blade server is very, very old. according to the documentation it only supports up to vmware 5.5.
You could try with version 6. But these versions are also very old.

Another tip would be to update all the firmware of the blade server hardware.

https://lenovopress.lenovo.com/tips0824#supported-operating-systems

 

 

[JarryG]

Ad.1: You can install it on local raid1-array (build with SSDs), but IMHO, it is useless. ESXi is just loaded during boot-up, and then never accessed. It holds the whole image in RAM. If you do not like USB, you could install ESXi on small CF-card (and attach it with CF-to-SATA adapter). Those flash-cards are targeted on "pro" market, and hold very long.

Ad.2: 850/Pro are not much different from 840/Pro, except they use some vertical/3d NANDs. No need to be affraid of missing them on compatibility list. Even 840/Pro has been added later (more or less on users' request)...

If I may give you a tip: leave at least 20% on those SSDs free (not in raid-array) as extra overprovisioning (in addition to those factory-set ~7%). It helps keeping constant R/W speeds, even without trim (no HW raid-controller supports trim in raid1).

[7007VM7007]

Ok, so lets say I decide to install ESXi on a USB flash drive. What happens when this USB glash drive fails?? What can I do in preparation for a failed USB flash drive? How should I back it up?

I know some servers let you mirror a USB or SD card but mine can't so I am very interested to hear how (or what options there are) for backing up a bootable ESXi USB flash drive!

JarryG wrote:

If I may give you a tip: leave at least 20% on those SSDs free (not in raid-array) as extra overprovisioning (in addition to those factory-set ~7%). It helps keeping constant R/W speeds, even without trim (no HW raid-controller supports trim in raid1).

20%, damn. Thats a lot of SSD space to not use! So lets work this out. Lets say I have a 512GB SSD drive (not RAIDed, just single drive). Formatted I will have about 470GB of space. Are you saying I should only use 376GB on the SSD drive for VMs? So when I add the SSD to the datastore do I only select 376GB as the usable space? Or is this done on the IBM RAID card?

Do you really see better/constant R/W speeds doing this? Its the first I have heard of this so am keen to hear peoples experiences.

One other question. I don't see the Samsung Pro 840 1TB drive listed on the LSI 9260-8i HCL (the 128/256/512 are). Will I be able to use the 1TB drive with my RAID card?

Thank you.

[JPM300]

Hey 7007VM7007,

20% free space on a Datastore is the standard / best practice.  This room is saved for snapshots, thin provision expansions if you use it, and memory overhead.  If your datastore runes out of space and your VM requests more space it will crash if it can't get it.  Example: Your Datatsore has 1GB of free space on it, and you shutdown your VM and boots the memory up 4GB.  This VM will no longer start as there isn't enough memory swap space on the Datastore.

Also if you have vCenter if you don't leave 20% free you will have to tweak your datastore alarms so it doesn'ts scream at you every day.

If you boot off a USB stick or some form of memory stick and it fails your host will go down.  However all you have to do is put another stick in and setup your ESXi host again.  The ESXi installation process is pretty quick, 15-20min tops.  The configuration if it is complex is where it takes time.  However you can store these in host profiles if you have vCenter.  If you don't have vCenter and you want to reduce single points of failure or reduce your recovery time you could always install ESXi the same way on two different USB/Memory drives.  That way if one fails you can boot up off the other and get things back up and running fairly quickly.

Its up to you.  I know in blades the SD card is pretty popular, however in most cases the cluster has 2+ more hosts in it so if one host goes down there is little downtime to the VM's as the VM's will start up on other hosts in the cluster while you fix the blade that had a SD failure.

[JarryG]

You can always use 2xUSB. You can even create raid1-array (with USB-to-SATA converter). I personally have 2x CF with "nearly" identical ESXi, but not in raid. Reason is: if I screw something up during patching, I can still boot from the other CF. It saved me once already!

Concerning that 20%: you are right, it is too much, but unfortunatelly necessary. You can read about it (i.e. on servethehome.com forum, you can find even some tests there). I have raid1 with 2x SM843T/480GB (which are basically the same as yours, with extra-capacitors and much higher price-tag), but I created raid1-array ony from 2x400GB. As I wrote: default factory-set over-provisioning ~6-7% is sufficient if TRIM (garbage-collection) works. Unfortunatelly, it does NOT work on raid1-arrays (AFAIK, not a single LSI-chip based raid-controller support TRIM on raid1, but some of them support TRIM on raid0). That's why you'd better leave more free space for SSD-controller to deal with this problem...

You can read something about it here:

http://www.lsi.com/downloads/Public/Flash%20Storage%20Processors/LSI_PRS_FMS2012_TE21_Smith.pdf

[7007VM7007]

Thanks for all the terrifc replies.

So to protect myself from USB flash drive failure I have two options:

1. When my ESXi server is configured the way I want it I should clone the USB drive OR
2. I could use host profiles (I know nothing about these, how do they work?)

JarryG wrote:

Concerning that 20%: you are right, it is too much, but unfortunatelly necessary. You can read about it (i.e. on servethehome.com forum, you can find even some tests there). I have raid1 with 2x SM843T/480GB (which are basically the same as yours, with extra-capacitors and much higher price-tag), but I created raid1-array ony from 2x400GB. As I wrote: default factory-set over-provisioning ~6-7% is sufficient if TRIM (garbage-collection) works. Unfortunatelly, it does NOT work on raid1-arrays (AFAIK, not a single LSI-chip based raid-controller support TRIM on raid1, but some of them support TRIM on raid0). That's why you'd better leave more free space for SSD-controller to deal with this problem...

Well I am thinking of using two Samsung Pro 840 128GB SSD drives and two Samsung Pro 840 512GB SSD drives. I will not be using RAID at all. So, will TRIM work in this kind of "RAID0" setup?

From what I understand there are two ways I need to manage the space on my SSD drives:

1. If TRIM works then I only need to not use 6-7% of the SSD. So for a 512GB drive I would use 476GB.
2. If TRIM does NOT work then I would need to use only 380GB for a 512GB drive.

Where do I limit the drive to always have 20% free, on the RAID card or in ESXi? In other words, do I limit the size of the SSD drive in the RAID card when creating a RAID0 drive or when I create the datastore in ESXi by limiting the size of the datastore on the SSD?

Any idea if a 1TB Samsung Pro 840 SSD will work with my RAID card?

[7007VM7007]

Having read some more about "over provisioning" (OP) I am still a bit confused about the details of implementing this. Hopefully someone can help me understand this. Let me use an example.

Lets say I have a 512GB SSD drive. This drive has 476GB of usable space. So I assume the 36GB of space is reserved for OP by the manufacturer. In this case it works out to about 7.5%.

But I want to have 20% for OP as was mentioned in this thread. So 20% of 512GB equals 102GB. So on my 512GB drive I will have to set OP to 138GB (102GB for the 20% and 36GB for the manufacturers 7.5%). Therefore, I will have 374GB of usable space...is this correct?

Assuming my calcualations are correct, then when I configure a 512GB SSD drive on the RAID card, do I set the size of the drive in the IBM WebBIOS to 374GB? What I don't understand is, someone said I should have 20% free space on the datastore, does that mean I lose ANOTHER 20%?

Or have I go this all wrong and just need to add 476GB of space to the SSD drive when adding it to the RAID card and then ensuring that the datastore always has 20% free?

Please help :smileyconfused:

[JarryG]

You have 512GiB (binary) SSD, that is 512 * 1.073.741.824 bytes

Out of this, you see 512GB (decimal) capacity, that is 512 * 1.000.000.000 bytes

This is ~7% factory-set overprovisioning.

That is apparently not enough in case you do not have TRIM (as in HW/raid1). Write-amplification factor is very high for just 7% OP (about 6-7). It means, for every MB you want to write (from system), 6-7MB are written. It could shorten your SSD-life signifficantly. But write amplification factor drops to 3 (which is quite acceptable compromise) at about 27% OP (page 11 of the above mentioned pdf).

So it is advisable to add 20% "user-level" overprovisioning. The way to achieve this with raid is that you pick SSDs (in WebBios, or Megaraid Storage Manager, or StorCLI/MegaCLI), but you do not use its whole capacity for raid-array, but just about 80%. Beware: SSD must be in "factory-reset" state (never used, nothing written); if not, you have to re-set it first. That remaining 20% will be left as "unused", and controller will use it for wear-leveling, garbage collection, etc.

After that, when raid-array is constructed (with only ~80% of SSD capacity) you create partition/s (using all the capacity array is reporting), and format it. How much real space remains depends on filesystem you use (block/subblock size, metadata, etc).

[7007VM7007]

JarryG wrote:

You have 512GiB (binary) SSD, that is 512 * 1.073.741.824 bytes

Out of this, you see 512GB (decimal) capacity, that is 512 * 1.000.000.000 bytes

This is ~7% factory-set overprovisioning.

That is apparently not enough in case you do not have TRIM (as in HW/raid1). Write-amplification factor is very high for just 7% OP (about 6-7). It means, for every MB you want to write (from system), 6-7MB are written. It could shorten your SSD-life signifficantly. But write amplification factor drops to 3 (which is quite acceptable compromise) at about 27% OP (page 11 of the above mentioned pdf).

So it is advisable to add 20% "user-level" overprovisioning. The way to achieve this with raid is that you pick SSDs (in WebBios, or Megaraid Storage Manager, or StorCLI/MegaCLI), but you do not use its whole capacity for raid-array, but just about 80%. Beware: SSD must be in "factory-reset" state (never used, nothing written); if not, you have to re-set it first. That remaining 20% will be left as "unused", and controller will use it for wear-leveling, garbage collection, etc.

After that, when raid-array is constructed (with only ~80% of SSD capacity) you create partition/s (using all the capacity array is reporting), and format it. How much real space remains depends on filesystem you use (block/subblock size, metadata, etc).

Thanks for the reply.

Sorry to be ultra pedantic about this but I want to make sure I do this correctly!

Lets say I start with 512GB SSD drive. Its a brand new drive with no data on it. So I connect it to RAID card and go into WebBIOS. Now I start configuring drive in WebBIOS as a SINGLE drive in RAID0. So when I add the drive do I add it with 409GB (512GB x 20% OP)? Therefore I will have a 409GB drive listed on the RAID card?

Assuming the above is correct: Now in ESXi I create a new datastore and add the 409GB SSD drive as storage. Do I need to keep 20% of the 409GB datastore free as well??

[JarryG]

"...Do I need to keep 20% of the 409GB datastore free as well??..."

No. It is enough to have 1x 20% (you saved in WebBios), not twice. Apart from that, I do not see any reason to have single drive in raid0 (if it is possible at all). Instead of that, you can configure it as single JBOD. That way you omit all those raid-array metadata...

[7007VM7007]

JarryG wrote:

"...Do I need to keep 20% of the 409GB datastore free as well??..."

No. It is enough to have 1x 20% (you saved in WebBios), not twice. Apart from that, I do not see any reason to have single drive in raid0 (if it is possible at all). Instead of that, you can configure it as single JBOD. That way you omit all those raid-array metadata...

Ok, I will reserve 20% then for OP. Thats very helpful. Just to clarify, with my 20% OP and the manufacturers OP, does this mean I'll have about 27% OP for my 512GB SSD drive?

The only reason I have the RAID card is for the caching since ESXi doesn't do this. So can I setup this card in JBOD mode AND use the cache to improve the performance in ESXi?

You can setup a single disk on the M5015 in RAIDo as I have done it when testing with my two SSD drives. If theres a better way to do this in JBOD (with caching) then I would like to hear about it please :smileyhappy:

Is it necessary to keep any free space in an ESXi datastore? If so how much?

[7007VM7007]

If I am running all SSD datastores, is there any point in using vFlash and/or SSD cache for swap?

[JPM300]

SSD drives for caching for swapping will help when or if your hosts ever need to swap during over commitment time.

vFRC has a lot of requirements and most of the tests I have seen and other setups I've seen the performance number don't seem to justify the cost of the extra drives.  However test it out for yourself.  There is  A LOT of other really good 3rd party apps/plugins for vCenter to assist with Flash Read Cache and Flash Write Cache.  If you want more info on some of the vendors I've tested shoot me a PM.

[7007VM7007]

But if my VM is running on SSD storage AND my swap files for that VM are running on SSD storage then surely vFlash and/or SSD to cache is pointless?

[JarryG]

There is really not much sense in caching SSD on SSD. Only if the other SSD is much faster (i.e. PCIe-card)...

[JPM300]

vFlash wouldn't give you as good of results in that case yes.  However even if your VM is running on SSD storage on your SAN you still have to worry about SAN IO storms, latency, the service processors being over worked on the SAN causing high DAVG times or latency, ect.  vFlash or a 3rd party solution to do read caching locally on a SSD driver or write caching buffers out all these problems.

However if you are running no shared storage and only running local SSD drives for everything then yes vFlash will not help you a whole lot if any.

[7007VM7007]

That pretty much confirms what I was thinking: SSD cache for VMs on SSD storage is a waste.