Configure Infiniband interface on virtual machine

[Bibo027]

Hi All,

I installed the Mellanox Infiniband driver on ESXi 4.0 U1.

Now I want to create a virtual machine that use the Infiniband interface.

I added hardware networking (VM IB Network) in configuration->hardware->networking.

Then I added new network adapter from VM IB Network on virtual machine. Which adapter type should I use in this case? VMXNET 2 (enhanced) or VMXNET 3

Is this procedure correct?

Regards,

Bibo

[William Lam]

If you want to allow a VM to directly access a physical device, you'll need to see if it supports VMDirectPath I/O pass-through, you can find more information in the ESX(i) documentation and this KB article should also help - http://kb.vmware.com/kb/1010789

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

Twitter: @lamw

Getting Started with the vMA (tips/tricks)

Getting Started with the vSphere SDK for Perl

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Community

If you find this information useful, please award points for "correct" or "helpful".

[Bibo027]

William,

It doesn´t support VMDirectPath I/O pass-through. I see the IB interfaces in configuration -> Network Adapters.

And, I don´t understand why theses interfaces are down because the infiniband network is up for others machines.

How could I put up these interfaces?

Best regards,

Bibo

[William Lam]

I see the IB interfaces in configuration -> Network Adapters.

You're mixing up two different questions.

A) To allow a VM to access any physical device, you need to setup VMDirect I/O Passthrough, assuming it supports it

B) What you're asking about now is physical host (ESX or ESXi), if you have issues getting the device online, I would recommend contacting your hardware vendor. If it's down, can you check your IB switches? make sure the cable is functional/etc.

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

Twitter: @lamw

Getting Started with the vMA (tips/tricks)

Getting Started with the vSphere SDK for Perl

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Community

If you find this information useful, please award points for "correct" or "helpful".

[mlxali]

Hi Bibo,

Please make sure that the subnet manager is active in the IB subnet.

Please follow the instruction in the Installation Guide.[http://mellanox.com/related-docs/prod_software/IB_OFED_for_VI_3_5_and%20vSphere_4_installation_guide_1_30.pdf]

[Bibo027]

Hi,

I used the command below and my interface is up.

$ vicfg-module --server 192.168.1.15 -s "port_type_default=1" mlx4_en

In VMDirectPath configuration I have this message "Host does not support

passthrough configuration".

How could I use the infinibandon virtual machine?

Regards,

Bibo

[mlxali]

You're host doesn't support pass-through.

This is not related to Mellanox drivers.

Please check vSphere help for pass-through requirement.

[Bibo027]

Hi,

The pass-through requirement is VT-d support but my CPU only support VT-x.

Is there another way to use infiniband on virtual machine?

Best regards,

Bibo

[mlxali]

Today VMs don't have direct access to IB HW.

You can use IPoIB (for networking services over IB) or SRP (for storage services over IB).

With the help of IPoIB/SRP you will be able to create virtual NICs and HBAs and assign them any VM.

[Bibo027]

Hi,

I am going to use the IPoIB. I added a NIC (IB) on VM but the IB driver doesn´t work.

What do I do wrong?

Best regards,

Bibo

[mlxali]

What "doesn´t work"?

Please follow the Installation Guide, it explains step by step how to configure a virtual nic over IPoIB uplink.

http://mellanox.com/related-docs/prod_software/IB_OFED_for_VI_3_5_and%20vSphere_4_installation_guide_1_30.pdf

And make sure that you have Subnet Manager running on the system.

[mivy]

I'm in a similar boat. I'm working on getting ipoib up on ESXi. my question is this:

Is it possible to run opensm within ESXi, or will I have to go the managed switch route. I'm using IB as a point to point line.

[mivy]

Nevermind. I answered my own question: No, the subnet manager has to run on a non-ESX host or I need a managed switch.

[champagnecharly]

deleted to reply to original post.

[champagnecharly]

Hello,

Sorry to revive.. however a few questions if I may re the success.. and a thought I had..So in theory a seperate machine is required to run open sm.. BUT what if opensm was run on a vm but passingthorugh a second ib card directly to the VM? ESX(i) and OPENSM VM would then have 2 different non conflicting devices.. Any thoughts?

Either way can I just clarify a quite possibly dumb question showing my ignorance now however I'd rather sound daft than go to extra unnecessary expense)

Does creating a SMA machine (e.g. opensm on ubuntu server). require as many ports as physical devices connected i.e the same way as a switch would present x connections? Or does it perform its service in a slightly more obscure way?

e.g. when working with 2 physical machines..

3 cards (4 ports) would be required.

esxi Server.. 1 card.. 1 port connected to SMA

Work station: 1 card.. 1 port connected to SMA

SMA machine: 1 card .. 2 ports.. 1 port to Server, 1 to workstation?

I look forward to reply..

Thanks