FC or iSCSI ?

[GrantLui]

Hi All,

After having tried the free ESXi 4 virtualization in our business, it is time for us to make a step ahead. We now have:

- 2 x IBM x3650 XEON 5405, 20GB RAM and 500G HDD

- 5 x VMs running on one of the machine (Exchange 2007, SQL, SMB file sharing)

The thing is with the free version of the ESXi 4, we cannot enjoy functions like HA and FT, and therefore one of our spare x3650 is wasted. Our objective is that we wanna to have all 5 of the current running VMs being protected by the spare server just in case of hardware failure, we also wanna the storage volume like Exchange 2007 storage, SQL and SMB file sharing being deployed on a larger shared storage pool rather than inside the VM's partition volume.

In order to achieve this, we know we will have to, on the software side to purchase a vSphere 4 Advance, and on the hardware side a share storage SAN.

Here is the question, wether we should go for a FC (FC adapter + FC switch + IBM DS3400) or a iSCSI (iSCSI adapter + 1G Eth. switch + IBM DS3300). We know FC is fast with lower latency than iSCSI and is more expensive, but we are not going to discuss here which one is more affordable, because we will present both configurations' quote to the management and they will decide it anyway. What I really want to know is the feasibility of using either of the solution and which solution is easier to setup.

As for FC, would it be easy to setup ESX host to connect to the SAN (e.g. drivers for the FC adapter or even no driver setup is necessary?), I think it might be a bit simple if it's on iSCSI as I did found some articles about implementing ESX over iSCSI, but yet I never found anything about over FC. Second thing is that, just as I mention we wanna to deploy several storage volume like Exchange 2007, SQL and SMB file sharing on to the SAN shared storage pool and divide them apart from the native VM's partition volume. The iSCSI initiator inside the VM may easily do the job, but it is however what about the iSCSI target? I'm not sure if you can create an iSCSI target over FC?

Will appreciate if there are someone can give me some suggestion. Thanks!

[JimKnopf99]

Hi,

thats not so easy to explain and it depends realy on what it should cost. I know, you do not want to speek about it but

FC need a seperate Network environment. Ifyou do not have one. FC Switche and so on.

iSCSI can use the "normal" Gigabit Switche. Thats a big advantage.

But keep in mind that FC is the only protocol that is encryptet. I know that you can use chap for the iscsi communication. But what is chap.......

And that is, in my meening the realy big different. Maybe if you work at a bank or something like that. And not use FC for your SAN. And it happend that someone steels a complete vm, and that is not impossible, then you have realy big problems. We saw that in a workshop how easy it is to steel a complete vm if you use iSCSI....

If you keep that in mind and describe that in your management document. Maybe you get FC Storage.

Only my 2cent.

Frank

If you find this information useful, please award points for "correct" or "helpful".

[GrantLui]

Hi Jim,

Well, if we are going to use iSCSI solution, we will buy a separate Eth. switch for solely that purpose anyway. Except I haven't really get a quote from vendors about the different between a FC and a "good" Eth. switch. Here are some components costs we've conclude so far:

FC setup:

IBM DS3400, single controller, disk enclosure only, USD$3.9k

IBM Emulex Single port HBA PCI controller, USD$1073 x 2 = USD$2.1k

=USD$6k

iSCSI setup:

IBM DS3300, single controller, disk enclosure only, USD$3.6k

IBM Qlogic Single port HBA PCI controller, USD$843 x 2 = USD$1.6k

=USD$5.2k

Well, you see that, yes there is a different, but the different is not huge I mean. And performance difference is 1G vs 4G. So looks like FC is a favorable investment?

But back to the original question, is wether it would be possible to have the VM to access the FC SAN pool, this is something I'm really not sure for a FC setup and I can't find anything if u can make a iSCSI target on FC.

Presumably, I think u'll have to add another NIC on both ESX and inside the VM if u'll have a iSCSI setup and use MS iSCSI initiator to access the SAN pool. However, I'm not sure if it will be the same case on a FC setup.

[kjb007]

The simplicity/feasibility of either depends on which you have more experience with. They are similar in that they are block-level LUN presentation, but FC requires you to configure SAN switch to attach your storage to ESX, while iSCSI requires IP switch config to attach your storage. If you already have an existing IP infrastrucutre, iSCSI seems a logical easy fit into your current environment, while fc requires that additional investment.

If you're going with the 1 GbE route, then FC will probably be faster, as you'd be hard pressed to find anything slower than 4 GB FC at this point. Both will allow you the sharing you mention in your post.

If you don't have a SAN infrastructure, then iSCSI will probably be faster to setup. You can do iSCSI/FC both and add directly to a VM as a raw device mapping, which wouldn't require using some MS iscsi initiator. Both can do this without having to create a datastore to see that storage LUN. You can't create an iSCSI target over FC, but you can add fc storage to a server, and turn that server into an iSCSI target.

-KjB

[GrantLui]

Hi KjB

We do not have any infrastructure on shared storage at all, neither iSCSI or FC, we are currently still using the DAS approach, so we are quite new to this technology. Now we are still at researching stage and it is just because it comes to us the objective in achieving HA and FT will definitely require a SAN setup and that is why this will be something that we must acquire this coming few months.

Just as mentioned in replying Jim's post, we initially found the hardware cost difference between iSCSI and FC setup was not of huge. Of coz, we know there are some other cheaper iSCSI solutions such as Starwind or some other NAS enclosure such as Synology can be configured as iSCSI target with very competitive price, but I think there must be at least some difference, at least for NIC, we even found a series of product from Qlogic specifically for use with iSCSI applications. Once I read an article published by Microsoft regarding SAN performance, a chapter specifically mentioned that the overall throughput can be significantly be impacted by some entry level NIC or standard home/office grade switch. We therefore may in favour in looking at some commercial class product.

I think there is one more reason that we may need higher bandwidth SAN, it is because we are running XenApp in one of the VM, and I believe it is one which has most of the I/O (staffs executing and closing apps).

As for your last suggestion, I'm not sure if I fully understand, do you mean I can build a VM as a iSCSI target and actually the disk resources that the target is using is from a raw device mapped from the SAN pool?

Thanks!

[kjb007]

Yes, that is correct. You can stand up an iSCSI target inside of a vm, if needed, even though the disk pool behind it is from somewhere else. Storage is completely abstracted from the virtual machines.

Your question of NIC and workstation home/office class switches is warranted. Most often, those devices are shared, or oversubscribed, if you will, so you have an X number of ports that share a 1 GbE uplink, so you have to be careful. Make sure to get server class NIC, which you will need for ESX, anyway, and make sure you get server-class switches as well. That is, if you go that route. These days, you can also invest in a 10 GbE infrastructure, and increase your bandwidth even further, but those prices have not come down as far as the 1 GbE devices.

That being said, I prefer FC myself for storage, and that may is just my biased oppinion.

-KjB