VT-d support on desktop motherboard

[jellyfish]

Hey. Just wondering if the Intel DQ77MK motherboard has Directed I/O support with ESXi5.x?

The ESXI hardware compatibility list just mentions server boards and not desktop boards - for obvious reasons.

My intention is to passthrough a PCIe NIC expansion card and two on-board NICs

I have absolutely no experience with virtualization but I have read up a lot over the past week so feel free to hit me with anything you feel I should know. This is just a personal project - not a production server.

Although the first page does not say that the board supports VT-d but the following states that the Q77 chipset, and that board in particular, does.

Compatibility list: Desktop Boards Compatibility with Intel Virtualization Technology (Intel VT)

I just thought I'd stop by here and run this by you guys to see what you think. The CPU I intend to use is either the i5-3470T or the i5-3570T.

Also - are there any special requirements for the PCI devices themselves?

Thanks!

[Bigfrank83]

Hi There,

(I think) the board you have mentioned does have Intel VT-d. I can also confirm that the ASROCK Pro 4 Motherboard supports Intel VT-d. This has the Z77 Chipset and is the board I am using. I am not sure if passing through on-board devices is a good idea within ESXi, from what I have read, but I may be wrong.

I have also been told by Gigabyte that all their UEFI Motherboards support VT-d.

I've also built a home project ESXi server with an i5 2500 along with the ASROCK board and 24GB RAM. If you need any advice on anything please feel free to PM me, I've learnt a lot since starting it last year and will be more than happy to assist.

Karl

[jellyfish]

Thanks a lot bud - I may just do that.  Why would passing through on board devices be a bad idea? Hanks for the heads up - I'll check it out.

[Bigfrank83]

No problem at all.

You cannot have two devices in two different contexts -- such as one used by VMkernel and one in pass-through -- using the same PCI slot. For example, the dual head NIC is dedicated to the VMkernel OR is available for pass-through. If you select one, the other is automatically selected as well. A dialog informs you why this occurred.

The above taken from here http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010789

If the above applies to that Dual on-board NIC then you will have to have an additional NIC installed to maintain connectivity to the host.

I went for the ASROCK Pro 4 merely because it had 3 x PCI slots, (and of course it had VT-d) and I have 3 PCI GB NICS Spare. So in theory (If I wanted to) I can use the on-board NIC for the Management network and the 3 add-in NICs in passthrough mode to particular VM's. Does that make sense?

What do you plan on using your host for?

[jellyfish]

I intended to passthrough one of the  NICs and a PCIe dual NIC Intel PRO/1000 PT expansion card - but in light of that I'm thinking passthrough both devices and use a separate PCIe x1 card for the management interface.

Am I right in thinking that the devices are managed by a bootstrapped Linux kernel? So a cheap Intel or Realtek card would do for the management interface? I'm assuming correctly that each PCI slot is treated separately?

The intention for my host? To run three instances - pfsense (NICs passed through), 2x Gentoo installations - each with their own hard drive passed through (not pfsense): can these devices be treated separately?

EDIT: I just noticed "Devices must be behind a PCI Switch that supports ACS" - I will have to do a bit of background reading so a reply may be a day or two.

Thanks for your patience.

[Bigfrank83]

Yes definitely use the add-in card for management. A cheap Intel or Realtek one will do the job. If that card goes down though you lose connectivity to the host as you won't be using Failover etc etc. Just a heads up for ya.

Each PCI slot is treated separately, yes.

You can certainly pass through the on-board SATA controller however I'm pretty certain that it can only be to 1 Virtual Machine and not shared between 2 or 3 etc etc.... your best bet is to have a look into RDM (Raw Device Mapping).

My suggestions to you -

1. Run ESXi from a USB stick

2. Use Add-in NIC for Management (This NIC will also supply network connectivity to your 2 Gentoo VM's as you will have passed through the other ports to an individual VM)

3. Pass through on-board Dual NIC to pfSense VM

4. Use RDM for HDD's. After all, you're gona need a datastore for the VM's and if you pass through the controller to a single VM, you have no ports for your datastore.

Karl

EDIT: Check this link - I've read it a few times before and its very informative.

Re: vt-d enabled Ivy Bridge motherboard recommendations

[jellyfish]

Bigfrank83 wrote:

1. Run ESXi from a USB stick

2. Use Add-in NIC for Management (This NIC will also supply network connectivity to your 2 Gentoo VM's as you will have passed through the other ports to an individual VM)

3. Pass through on-board Dual NIC to pfSense VM

4. Use RDM for HDD's. After all, you're gona need a datastore for the VM's and if you pass through the controller to a single VM, you have no ports for your datastore.

Sorry for the delay; done some background reading.

1. Why a USB stick? If I do could I use a lockable stick? (assuming no config changes)

2. I intend to just use a virtual switch for my two Gentoo installs - the passed through devices will be for pfsense. Will the combination between a virtual NIC and RDM be too much for the CPU on my NAS? (one of the Gentoo installs) I am attempting to write at gigabit speeds from time to time - mainly for backups.

4. I read up a little on RDM - Is  there a performance hit when using it; does it utilise a lot of CPU? I assume the 'Raw' in RDM means I have raw access to the disk and hence I can format any file system - but does the data reside inside any container.. or is it 'raw' on disk? Silly question really. :smileywink: I don't want system snapshots hosing any data.

Q: I noticed that PCI passthrough requires an ACS capable PCI switch - this has been included since the PCI 2.0 specification right? So it shouldn't be a problem?

"BTW, jellyfish, I have a DQ77MK at home, although it's running Debian 7, not ESXi :smileysilly:.  Let me know if I can collect any info to help with your purchasing decision..."

Yeah that would be helpful:

1) The BIOS - does it support Enhanced SpeedStep? i.e. frequency scaling. There is suppose to be a section in the CMOS on supported board.

2) Is there an option for enabling VT-d in the CMOS? Some affirmative confirmation for this would be great.

3) Do you run KVM at all? It supports VT-d - if it's not too much out of your way perhaps you could try it out... but perhaps that's asking too much. :smileyhappy:

Thanks for the helpful replies guys.

[Darius Davis]

You cannot have two devices in two different contexts -- such as one used by VMkernel and one in pass-through -- using the same PCI slot. For example, the dual head NIC is dedicated to the VMkernel OR is available for pass-through. If you select one, the other is automatically selected as well. A dialog informs you why this occurred.

That is true for integrated devices as well as add-on devices.  Specifically with respect to motherboard-integrated devices, though, there is an additional factor at play:  Motherboard manufacturers may consider the silicon on the motherboard to be a non-divisible entity, and make hardware and firmware design choices which assume, for instance, that all the motherboard hardware will be under the control of a single OS for purposes of power management, resource allocation, etc.  Those assumptions may be broken when any subset of the motherboard-integrated devices are taken separately and passed through to a VM, because we have no way to characterize how the passed-through devices might interact with other motherboard components.  Motherboard hardware or firmware could then start interfering in unpredictable ways with a device which is now supposedly under the control of the guest OS, or the guest OS could even end up interfering with other host hardware.  Any add-on card essentially guarantees hardware isolation by design; Integrated devices do not have any such inherent guarantee.

We've had plenty of reports of success with passing through various motherboard integrated devices, so my gut feeling is that the above is mostly a theoretical concern, but AFAIK this is not a configuration that we're likely to officially support in the general case.

Hope this helps!

BTW, jellyfish, I have a DQ77MK at home, although it's running Debian 7, not ESXi :smileysilly:.  Let me know if I can collect any info to help with your purchasing decision...

--

Darius