VMware vSphere

I'm new to ESXi, so excuse me if this is trivial. I have read that many run ESXi from a USB stick rather than internal hard disks. Can someone please explain why and what the benefits are?

I am considering ESXi and unfortunately my HP server is not on the HCL. I only have one server and plan on having 3 VMs. Would I install ESXi on a USB stick and use my 2x72.8GB UltraSCSI320 hard drives setup as RAID 1 for storing my 3 VMs or have ESXi and the 3 VMs on the hard drives? What is preferred way of setting this up?

Thanks

When you install ESXi on an USB stick, upgrading is easy. Just replace the USB stick, no need to install anything on your harddisk. Also, when running multiple ESXi hosts with shared storage, you can run them without any harddisk at all.

Your suggestion to run ESXi from USB and use your harddisks for the VM's is a good one, many people use it like that. Me too :smileyhappy:

Hi microkid. Thanks for the quick reply! A few follow up questions.

- I have read that ESXi installed on a USB stick by HP is available. Does it differ from a USB stick that I would purchase and load with ESXi HP CIM?

- When you say that you just replace the USB stick and upgrade. I assume that you have to power off the server first, perform the upgrade on the USB and then attach it and boot, right?

- If the server fails and I have the 3 VMs on some other backup. Is it enough to just remove the USB stick, attach it to another compatible server and move my VMs to that new server?

- Is it enough to have two disks in RAID 1 for my 3 VMs or would I need more disks? Not in regards to space, but performance/ redundancy?

Thanks

Don't know about the HP thing, but the answer on the 3 other questions is "yes" :smileyhappy:

Don't know about the HP thing, but the answer on the 3 other questions is "yes" :smileyhappy:

Can you please clarify what you said yes too in the below question:

Is it enough to have two disks in RAID 1 for my 3 VMs or would I need more disks? Not in regards to space, but performance/ redundancy?

well, assuming this is a kind of home server, 2 or 3 VM's would only take approx 50-60GB of disk storage. So your 2 x 72GB disks in raid-1 would be sufficient.

Reminder: make sure you use hardware raid and that your SCSI raid controller is on the HCL, because most onboard raid controllers rre not supported.

Thanks. Btw this is not a home server. This is actually my company server. Small business with two users, hence the 2 Windows XP Pro VMs and one Windows 2003 server VM. After looking at the HCL it looks as if my RAID controller HP Smart Array 642 is not supported. I'll give it a try and see what happens.

HP does have preinstalled USB stick available. I wouldn't expect it to be up to date.About $50. It can install to the internal USB port specifically meant for this purpose. The HP versions have specialized monitoring modules. It can be downloaded from HP. I posted something in this forum pointing to the HP download.

Updating using the vSphere update tool can apply patches and updates remotely without removing the stick. It will mean restarting the host so VM would need to be shut down.

I've discussed this before on this forum, and been shot down, so take my statement with a grain of salt.

I've been there done that with the HP USB stick installation. Despite having a four hour response care pack, when that USB stick failed, it took nearly two weeks to get it replaced. Yes, I ended up buying my own, but that still meant:

• A full days downtime before you figure out what exactly is wrong. It's not like USB keys support SMART

• An hour to reinstall and reconfigure networking. Most of the proponents of this system argue this could be done quicker, and fail to see the rest of the time involved.

• In my case, multiple trips between the server and my current location, two hours in each direction

The other scenario is the one where you install ESXi on the RAID mirrors, one fails, you get an alert, HP come out and hot swap the disk at some point with no downtime.

Edit: And no, there is no difference between the ESXi installation that you buy from HP on a key, and installing it yourself on a key using the HP CIM edition. Unforutnately, when I last checked, HP's agents weren't vSphere suported yet, which meant ESXi 3.5. Manually installing ESXi 3.5 on a USB key is non-intuitive, which is why vendors have taken to supplying preinstalled keys. They charge a lot more for it too.

I do understand the single point of failure. Certainly something to be aware of when making any critical infrastructure decision. I have also had those same single point failures with RAID controllers, motherboards and RAM failures. Yes HP replaced them quickly and I was back in business. To my mind the failure of a USB stick is less disruptive. Easy to have a spare ready to go and I don't need to be concerned that an upgrade will touch the datastore drive.

Very interesting information from both you Josh26 and DSTAVERT. I'm still learning so please bear with me.

DSTAVERT is there any benefit of purchasing HP's USB sticks over another generic make like Sandisk and loading ESXi myself? If I am going to implement ESXi it seems that using a USB stick for ESXi and my RAID 1 for the datastore is the preferred method.

I will try ESXi for the first time this week and I hope it goes well since my HP Proliant ML350G4p is not listed on the HCL.

There is no benefit. VMware posted something but it may have been in regards to the 4 beta. It listed Sandisk, Kingston, Verbatum, I believe. Minimum 1 GB. Sandisk is fine but you want to make sure it isn't U3 or wipe the CD partition if it is.

Not being on the HCL is not a good sign. ESXi 4 requires capabilities beyond just 64bit

I would download the processor check utility (I think it is available from that link). You would probably not support 64bit OS's in any case since I don't think that you would have BIOS virtualization support. If it isn't capable I would strongly suggest that you use ESXi 3.5 if this would be for a production environment.

WRT USB stick size is any data written to it after I've loaded ESXi on it? If 1GB is minimum is there a recommended size, i.e. 2GB or larger?

I have run Intel's Processor CPU ID and my processor is 64bit capable with hyperthreading, but does not support Intel VT.

I am not interested in running any 64bit OS. Also, my HP server is not on the HCL for any ESXi version unfortunately. Is this a big problem? I had it confirmed from another forum member here that he has the same server and runs ESX4.

Should I forget about ESXi? Not use version 4?

Many questions. Sorry :smileywink:

UPDATE: I just had a look at the HP VMware ESXi 4.0 Getting Started Guide. It says and I quote:

VMware ESXi only supports the following installation destinations for HP VMware

ESXi:

• Any HP hard drive supported for a server

• Secure Digital (SD) memory card

• Flash media (USB flash drive)

The following flash devices are supported:

o Kingston Flash Memory Secure Digital Elite Pro: Description—2GB Secure

Digital Card (50x), Part Number: HP-SD/2GB-S available from the Kingston

website (http://www.kingston.com/hpoffer/).

o v100ss 2GB USB Flash Drive, part number FD2GBHP100-BLK-SMA-T

To purchase the v100ss USB flash drive, call PNY at 800-769-4221.

VMware ESXi does not support any other USB or SD card devices.

If it were me, I would only use a platform that had full supportability. There are lots of people happily running ESXi on platforms that have no support. If something didn't go well late some night do you have the background to get yourself going again. Without a platform known to behave a certain way you may not get help when it really counts.

Some servers just didn't get tested for any version and may in fact work well. I would definitely consider my options carefully. If you have time to do some thorough testing you could certainly give it a try.

If it were me, I would only use a platform that had full supportability. There are lots of people happily running ESXi on platforms that have no support. If something didn't go well late some night do you have the background to get yourself going again. Without a platform known to behave a certain way you may not get help when it really counts.

Some servers just didn't get tested for any version and may in fact work well. I would definitely consider my options carefully. If you have time to do some thorough testing you could certainly give it a try.

I take your point and agree that using supported hardware would be the way to go. However, I'm not ready to throw out my almost 3 year old HP Proliant server when it is in perfect condition. Initially, I wasn't even thinking of using ESXi, but after struggling with VMware Server I have reconsidered. I haven't tested ESXi yet, so I really don't know what to expect.

I'm at the research stage and trying to gather as much information on the use and configuration of ESXi before I do some testing.

Thanks

Testing is perfect. You shouldn't have any problems and if you do it is part of learning. Get yourself going with what you have.

If I didn't mention it consider the[online training class|http://mylearn1.vmware.com/mgrreg/courses.cfm?ui=www&a=det&id_course=14798&src=EM_08Q3_VMW_ESXi_Emai1CTA_online%20training%20course].

Well install USB will give advantages over diskless enviournement , specially in blade servers. But managing the USB bit painful cause don't provide the flexibility of service console in that, yeah still tech-support mode is available for doing some testing directly on ESXi or configuring ssh on host server,

But no worries, vMA appliance will be good tool to manage everything..

Best of luck with your testing....AM

Well install USB will give advantages over diskless enviournement , specially in blade servers. But managing the USB bit painful cause don't provide the flexibility of service console in that, yeah still tech-support mode is available for doing some testing directly on ESXi or configuring ssh on host server,

But no worries, vMA appliance will be good tool to manage everything..

Best of luck with your testing....AM

Thank you

You are always welcome for any more queries...

AM

If you found my answer to be useful, feel free to mark it as Helpful or Correct.

The latest blogs and articles on Virtulization:

http://communities.vmware.com/blogs/amodi

Testing is perfect. You shouldn't have any problems and if you do it is part of learning. Get yourself going with what you have.

If I didn't mention it consider the[online training class|http://mylearn1.vmware.com/mgrreg/courses.cfm?ui=www&a=det&id_course=14798&src=EM_08Q3_VMW_ESXi_Emai1CTA_online%20training%20course].

Yep, I am going to do this course: http://store.vmware.com/store/vmware/en_US/DisplayProductDetailsPage/productID.104519700

I'm sure I'll be back with further questions once I get started :smileywink:

Thanks

I'm sure I'll be back with further questions once I get started :smileywink:

Have fun!

If it were me, I would only use a platform that had full supportability. There are lots of people happily running ESXi on platforms that have no support. If something didn't go well late some night do you have the background to get yourself going again. Without a platform known to behave a certain way you may not get help when it really counts.

Some servers just didn't get tested for any version and may in fact work well. I would definitely consider my options carefully. If you have time to do some thorough testing you could certainly give it a try.

I take your point and agree that using supported hardware would be the way to go. However, I'm not ready to throw out my almost 3 year old HP Proliant server when it is in perfect condition. Initially, I wasn't even thinking of using ESXi, but after struggling with VMware Server I have reconsidered. I haven't tested ESXi yet, so I really don't know what to expect.

I'm at the research stage and trying to gather as much information on the use and configuration of ESXi before I do some testing.

Thanks

I've had middle of the night failures, besides having or not having the skils, when it's 2am and your infrastructure is down, it's nice to be able to rely on the guys who wrote the software. It does come down to, come 9am, how do you explain to the CEO that all the servers are down, you're not quite sure how to fix it, and the software manufacturer won't help you because you went cheap on the hardware? . A 3 year old server is fully depreciated. At this point, you are much better off with new hardware that is on the HCL

I've had middle of the night failures, besides having or not having the skils, when it's 2am and your infrastructure is down, it's nice to be able to rely on the guys who wrote the software. It does come down to, come 9am, how do you explain to the CEO that all the servers are down, you're not quite sure how to fix it, and the software manufacturer won't help you because you went cheap on the hardware? . A 3 year old server is fully depreciated. At this point, you are much better off with new hardware that is on the HCL

I fully agree this is what SHOULD happen. But as technical people it's hard to see a business reality, that in many businesses, there's a conversation that's a lot more likely to cost a person his job than the one above. It is the conversation where you attempt to convince someone their "recent big investment" in a three year old server requires further expenditure, let alone replacing it.

I'd certainly advise replacing the server, but you can never judge a poster until you know the circumstances.

I'm very thankful for all the suggestions and bits of information. I may not have been very clear when I outlined my IT infrastructure, so I'll spend a few minutes clarifying my situation and why I am interested in ESXi.

I am the IT administrator/ manager of my father's accounting firm. This is not my job as I am actually a mechanical engineering graduate currently looking for work. I have an interest in all things technical and that is why I took control of my father's IT network in 2004. I am self taught and started looking at virtualisation seriously after having used VMware Fusion for sometime. I am a Mac user for about 9 years.

The following hardware constitutes my father's IT infrastructure:

Hardware

HP Proliant ML350G4p server

RAM: 1GB

Hard drives: 2 x 72.8 GB UltraSCSI320 hot-plug

Smart Array 642 Controller

HP StorageWorks DAT 72 SCSI tape drive

1 NIC HP NC7761

APC SmartUPS 1500VA

Netgear ProSafe Firewall FVS338

ADSL modem

HP Laserjet 4050T + JetDirect

2 x HP Workstations

The server runs Windows 2003 Server R2 and is a member of the company's small network consisting of the server and two HP workstations with WIndows XP Pro. The two HP workstations are very old (ca 6 yrs) and one of them is acting up. I am interested in replacing the two HP workstations with HP thin clients.

I have evaluated an HP thin client T5545 for two weeks and I am very happy with the results and seamless integration with virtualisation. However, I have used VMware Server 2.0.1 for the virtualisation and experienced significant problems with the web access component (if you want to read more you can see my other thread here: http://communities.vmware.com/thread/213044?start=0&tstart=0). This is primarily why I am looking at ESXi and why I have turned to this forum for information.

Finally, my motivation for choosing thin clients as opposed to purchasing full-fledged HP workstations is that I don't need to worry about hardware failure, setup of the hardware is minimal and administration of the VMs is simplified. It is also important to note that I have to manage everything remotely, as I have done since 2004 unless there is hardware failure.

NOTE: THIS IS NOT AN EXHAUSTIVE EXPLANATION, BUT A BRIEF OUTLINE IN RESPONSE TO PREVIOUS COMMENTS!!

If I might suggest, add RAM, you will need to anyway. Then setup Terminal Services. You can use the RDP protocol on the thin clients. It mat be the easiest to setup and admin with the least disruption.

To setup ESX, you would first have to backup the server, load ESX, reload the server and data and setup the workstations. TS is a matter of setting it up, getting a couple of licenses and rebooting the server.

If I might suggest, add RAM, you will need to anyway. Then setup Terminal Services. You can use the RDP protocol on the thin clients. It mat be the easiest to setup and admin with the least disruption.

To setup ESX, you would first have to backup the server, load ESX, reload the server and data and setup the workstations. TS is a matter of setting it up, getting a couple of licenses and rebooting the server.

Yes, I have already planned to upgrade the RAM. I have looked at Terminal Services, but this is not an interesting option. Furthermore, disruption is not an issue since the company is closed for the summer.

Is that spec correct, only 1GB of ram?

If thats the case I'd say you don't really want to waste memory with the overhead of VMs and the duplication of the OS in memory. You'll use up all of your RAM just booting 3 instances of windows on the VM, before anyone even logs in.

You might want to consider using Terminal Services on the server instead, however I feel that with only 1GB of RAM in the server, there isn't really much you can use it for as far as serving clients. If the apps they use are pretty tiny then you might be able to trim down a terminal server install enough to make it bearable, I just can't picture you doing anything very useful with VMs on 1GB of RAM.

I'm not even sure you can get ESXi to run a VM that would give Windows enough ram to be useful, theres just too much overhead. My home server for example uses 1.7G of memory (out of 4) JUST in the overhead and services on it, before any VMs actually start.

ESX does allow you to SWAP VM memory out to disk, but theres no way you're going to be able to stand using a virtual machine that is constantly swapping just to run.

Is that spec correct, only 1GB of ram?

If thats the case I'd say you don't really want to waste memory with the overhead of VMs and the duplication of the OS in memory. You'll use up all of your RAM just booting 3 instances of windows on the VM, before anyone even logs in.

You might want to consider using Terminal Services on the server instead, however I feel that with only 1GB of RAM in the server, there isn't really much you can use it for as far as serving clients. If the apps they use are pretty tiny then you might be able to trim down a terminal server install enough to make it bearable, I just can't picture you doing anything very useful with VMs on 1GB of RAM.

I'm not even sure you can get ESXi to run a VM that would give Windows enough ram to be useful, theres just too much overhead. My home server for example uses 1.7G of memory (out of 4) JUST in the overhead and services on it, before any VMs actually start.

ESX does allow you to SWAP VM memory out to disk, but theres no way you're going to be able to stand using a virtual machine that is constantly swapping just to run.

See my previous comment. RAM is not a problem since I will be upgrading and TS is not of interest. Also, in my tests with VMware Server and one VM running, 1GB RAM has been doable. Hence, if I was to use ESXi I would assume that since there is no host OS and if I was still using 1GB, my VM would have access to more RAM.

Out of interest, how does the thin client access the XP VM? Does it RDP to it, or requires a VDI?

Do you really wnt to go to the trouble and expense of ESX and perhaps View and add the complexity of all that because you dislike TS?

Take it from the people who deal with it daily, ESX is way to complex for your environment.

RDP. I appreciate your comments and I am aware that I am a novice in the virtualisation field. However, I am looking at ESXi and not the paid version ESX, so View is not something I am looking at.

As I've said I have done testing with VMware Server and it worked well with regards to the integration between the thin clients and accessing the Windows XP Pro VMs via RDP.

ESXi is the next step up from VMware Server and I'll give it a try. Experimentation is the best way to learn and I won't know whether it will work in my situation until I've done that. Most likely I will run into hardware compatibility problems and that will rule out ESXi since I have no plans on replacing the current server. I will re-evaluate my options in that case and see what I'll do.

Thanks

Ok. So you have a pair of 72GB drives, I assume mirrored. You are going to load server and 2 instances of XP. That will not leave much room for data. If you are going to use RDP anyway, you are much better off with TS.

It will add essentially zero load to your server. Add 1 or 2 GB of RAM and you are set.

RDP on XP is nowhere near as robust and full featured as on TS.

I do understand the single point of failure. Certainly something to be aware of when making any critical infrastructure decision. I have also had those same single point failures with RAID controllers, motherboards and RAM failures. Yes HP replaced them quickly and I was back in business. To my mind the failure of a USB stick is less disruptive. Easy to have a spare ready to go and I don't need to be concerned that an upgrade will touch the datastore drive.

How is the failure of a USB drive less disruptive than a hotswappable RAID disk?

If it's "easy to have a spare ready to go" I'm guessing your servers aren't in datacenters that are a pain to go visit in the middle of the night.

Datacenter is a pain, always. I just like the convenience. All the server manufacturers are making it available. I use HP and they are now supporting internal SD cards. If it was a really bad decision from a failure rate perspective I would guess it wouldn't have made out of the labs. You had a failure and I understand your reluctance to trust it. Thankfully no one has made it mandatory to have the server boot from USB or there would a lot of angry people.

But USB has a finite access cycle. Given the choice between facing a USB or HDD failure, I'll take the HDD because it will be RAIDed.

So I will ask. Do you have any Cisco routers or switches (or any manufacturer for that matter). Do you depend on then 24/7. Are they critical to your organization. Do they move tremendous amounts of data.

Do they have hard drives? I realize we end up with redundant switches but we also have redundant servers.

We have insisted on raid because we are planning on having failures. They happen all the time and rather regularly. USB has no moving parts. Doesn't have the same heat issues. You can easily solve the write issues by using a remote syslog server. We won't solve the riddle which is better since there will always be a difference of opinion.

USBs have no moving parts, but the memory cells have a finite limit of read/writes. That is why SSD have much more capacity than advertized, they rotate the cells used to lenghten MTBF.

perfect hdd will provide you raid technology what happened in case of failure of two disks and more. but don't forget you have the HA in vmware which can easily move ur production VM on different host. On otheriside if your usb crashed, you can simply reimage the usb key and restore the config from backup that's ur new usb ready for production VM. But hardly speaking I have really some bad times with management of USB key sometime compare to classic one.

AM !![http://vsolutions.compare2shop.com]

USB has no finite numbers for failure. Flash memory does. Flash memory does not fail due to reads it failes due to writes. ESX in a USB/SD boot configuration doesn't do much writing to the device and its highly unlikely that you're going to fail due to wearing out the Flash memory. DO NOT USE A CHEAP USB STICK OR SD DEVICE! The company I work for previously resold branded/customized USB flash drives for various companies, the number one failure mode is cold solder joints breaking lose when we would make the mistake of buying cheap sticks to start with. There are 3 basic classes of USB stick you'll find. Throwaways - these are the ones you get at some tradeshow or as a free gift. You don't want to use these for anything except maybe having one around handy to show how they fail. Then there are the consumer level you can buy at a store from the big three manufactures. These are typically OK and if you use the tip below can be viable, although I'd be hesitant about using it in a production environment, I do use an over the counter one for my ESXi server at home. If you can find a high quality USB stick you are safer than with a hard disk as they are solid state devices and rarely fail. The only problem is finding one of this class, HP gets them, which is why you pay $50 for it instead of more like $10 or so like the going rate for a 2GB device. Unfortunately I can't tell you where you can buy one of these as a end user, our company had to jump through hoops to finally get that level of drive and finally actually found them via a goverment agency that pointed us at where they get theirs. I wasn't part of that process so I'm afraid I can't point you in the right direction.

The reason USB and SD cards are an attractive way to boot an ESX server is because they can be replaced practically instantly, and since ESX doesnt' do a lot of writing to the card they will last forever from a practical perspective. You will occasionally have a failed drive, but due to their nature you are far more likely to lose a high quality SCSI drive than you are to lose a high quality SD device.

I would use SD if at all possible. Even buying a 'high end' SD card at walmart will out perform any USB stick you're going to put into it, the SD interface simply allows for more bandwidth than you'll get out of USB in the real world so you'll shorten your boot times and SD cards in my experience are more reliable due to the manufacturing processes. If you go the SD route, don't use an adapter from one SD size to another to fit into your SD controller. Corrosion and heating/cooling cycles will eventually break the contacts on your adapter cards and you'll end up with a failure thats very hard to track, half the time a reboot will fix it the other half the time it won't. Removing and reinserting the sd card from the adapter and the adapter from the reader usually fixes it, but if you just avoid the adapter you've taken one more point of failure out of your system for no work other than buying the right card. Micro SD cards (possibly mini as well, not sure) use a lower voltage, run a little cooler and will probably last longer but thats just a theory/common sense guess on my part, I have no actually tests to confirm it.

If you have a relatively static configuration such as mine you can give yourself some breathing room with USB or SD in the case of the boot drive failure. After you have the server all setup, vms, networks, boot options and all that configured, shut it down, pull out the USB/SD device and use something to duplicate it. If you're using Windows and making your own USB/SD boot device than you've already used WinImage (or something like) it to get the boot image on the drive. Just use WinImage to duplicate the existing USB or SD device to a second one. I leave my second taped to the server at the datacenter. If all else fails, a call to the data center can get my a general idea of whats on the display and if need be they can just swap the boot device for me and power cycle it back to a working configuration.

My personal reason for going with USB/SD is this: Easy replacement of the entire ESX hypervisor without touching my data disks. If you end up with some weird breakage or stupid mistake like I've done (upgrading to new versions which no longer support your old non-VT supporting processors :smileyhappy: you end up needing to essentially reinstall or do a bunch of work by hand to fix your mistake. With the USB device you can simply remove it, reimage it and start reconfiguring (if you don't have a backup image of your previous USB/SD boot device) and you're off.

I probably use ESX just slightly more so than you intend to to start out with, but our company is not really a hard core ESX user. We only have 2 machines and they are used to run a ton of QA and testing setups, we only have backup domain controllers on them as far as production/important VMs so our ESX server going down is a hassle to a couple developers, but they can just run most of the VMs directly via file shares in a pinch if the server is down. I think SD/USB is the better way to go for you, but if my butt was on the line because the ESX server ran critical machines, I would likely use a good RAID and just use the old tried and true methods since those really are the best tested and most likely to have the least problems. Fortunately for me, my butt isn't on the line and SD card take home the win. :smileyhappy:

It's really too bad you didn't give your response more thought before you posted. :smileylaugh:

Great post and thanks. Earlier in the post there is a link to the HP supported media.