Running ESXi under Ubuntu

[rawsik]

Hi guys,

  could you help me please run ESXi 6 under Ubuntu 14.04.04  with kvm-qemu.

I have   Ubuntu 14.04.4 LTS

             ESX  6.0   vmware-201601001-3380124-iso

             Running hypervisor: QEMU 2.4.0

I try to run VM on EXSi  which running under KVM.  

So, ESXi was successfull installed,  the command in ps :

qemu-system-x86_64 -enable-kvm -name esx -S -machine pc-i440fx-2.4,accel=kvm,usb=off -cpu Nehalem,+invpcid,+vmx -m 8096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid b2c3b37b-9596-98ab-8e0d-f05df89d2920 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/esx.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/esx-20.img,if=none,id=drive-ide0-0-0,format=raw -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=26,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:96:74:1c,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 0.0.0.0:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4

But I always get error like :

Virtualized INtel VT-x/EPT is not supported on this platform. Continure without virtualized msg.intel.hvhwmmu? during starting VM on ESXi

cat /proc/cpuinfo  on 0 host machine
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid

what is wrong ?   what kind of flags I should add for run VM on  1 host machine ( esxi) ?

[Doug Covelli]

> what kind of flags I should add for run VM on  1 host machine ( esxi) ?

ESXi is the inner hypervisor so you need to configure QEMU to expose VT to ESXi.  Can you check with the qemu folks on how to do this?  ALso you may need to add  "vmx.allowNested" to the ESXi VMs .vmx file.  

[Doug Covelli]

Sorry the option should be "vmx.allowNested=TRUE"

[rawsik]

I added these options in  /etc/vmware/config

vhv.allow = "TRUE"

hv.assumeEnabled="TRUE"

vhv.enable = "TRUE"

vmx.allowNested = "TRUE"

But, it didn't help me.  ( I rebooted ESXi also)

It affected only for 32-bit VM, which failed with new error .

I read many articles which describes that people successfully run  ESXi under KVM.  But I can't reproduce it.  So, I try to understand what I doing wrong.

[Doug Covelli]

I don't think the problem is with how you are configuring ESX - you need to configure QEMU to expose VT to ESXi.  Can you check with the qemu folks on how to do this?

[rawsik]

for clarify:

cat /sys/module/kvm_intel/parameters/nested

Y

cat /sys/module/kvm/parameters/ignore_msrs

Y

cat /sys/module/kvm_intel/parameters/ept

Y

my xml:

<domain type='kvm' id='41'>

  <name>esx</name>

  <uuid>b2c3b37b-9596-98ab-8e0d-f05df89d2920</uuid>

  <memory unit='KiB'>8290304</memory>

  <currentMemory unit='KiB'>8290304</currentMemory>

  <vcpu placement='static'>4</vcpu>

  <resource>

    <partition>/machine</partition>

  </resource>

  <os>

    <type arch='x86_64' machine='pc-i440fx-2.4'>hvm</type>

    <boot dev='hd'/>

  </os>

  <features>

    <acpi/>

    <apic/>

    <pae/>

  </features>

  <cpu mode='custom' match='exact'>

    <model fallback='allow'>Nehalem</model>

    <vendor>Intel</vendor>

    <feature policy='require' name='vmx'/>

    <feature policy='require' name='lm'/>

    <feature policy='require' name='lahf_lm'/>

    <feature policy='require' name='nx'/>

    <feature policy='require' name='fxsr'/>

    <feature policy='require' name='invpcid'/>

  </cpu>

  <clock offset='utc'/>

  <on_poweroff>destroy</on_poweroff>

  <on_reboot>restart</on_reboot>

  <on_crash>restart</on_crash>

  <devices>

    <emulator>/usr/bin/kvm-spice</emulator>

    <disk type='file' device='disk'>

      <driver name='qemu' type='raw'/>

      <source file='/var/lib/libvirt/images/esx-20.img'/>

      <target dev='hda' bus='ide'/>

      <alias name='ide0-0-0'/>

      <address type='drive' controller='0' bus='0' target='0' unit='0'/>

    </disk>

    <disk type='block' device='cdrom'>

      <driver name='qemu' type='raw'/>

      <target dev='hdc' bus='ide'/>

      <readonly/>

      <alias name='ide0-1-0'/>

      <address type='drive' controller='0' bus='1' target='0' unit='0'/>

    </disk>

    <controller type='usb' index='0'>

      <alias name='usb0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>

    </controller>

    <controller type='pci' index='0' model='pci-root'>

      <alias name='pci.0'/>

    </controller>

    <controller type='ide' index='0'>

      <alias name='ide0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>

    </controller>

    <interface type='bridge'>

      <mac address='52:54:00:96:74:1c'/>

      <source bridge='virbr0'/>

      <target dev='vnet0'/>

      <model type='e1000'/>

      <alias name='net0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

    </interface>

    <serial type='pty'>

      <source path='/dev/pts/2'/>

      <target port='0'/>

      <alias name='serial0'/>

    </serial>

    <console type='pty' tty='/dev/pts/2'>

      <source path='/dev/pts/2'/>

      <target type='serial' port='0'/>

      <alias name='serial0'/>

    </console>

    <input type='mouse' bus='ps2'/>

    <input type='keyboard' bus='ps2'/>

    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>

      <listen type='address' address='0.0.0.0'/>

    </graphics>

    <video>

      <model type='cirrus' vram='9216' heads='1'/>

      <alias name='video0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

    </video>

    <memballoon model='virtio'>

      <alias name='balloon0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

    </memballoon>

  </devices>

  <seclabel type='none'/>

</domain>

If you run ESXi on KVM successful,  please  write your OS and KVM version, also please provide your XML config

[Doug Covelli]

The error :

Virtualized INtel VT-x/EPT is not supported on this platform. Continure without virtualized msg.intel.hvhwmmu? during starting VM on ESXi

indicates that you won't be able to use virtualized VT-X under ESXi but you still should be able to run VMs using VT-x and EPT if you remove the line vhv.enable = "TRUE".  You won't be able to run guest hypervisors like Hyper-V under the ESXi VM but normal guests that don't require VT-X should run fine.

[rawsik]

I'll try to ask folks from KVM.

my /etc/vmware/config now:

hv.assumeEnabled = "TRUE"

vmx.allowNested = "TRUE"

vhv.allow = "TRUE"

error which I got in GUI:

Failed to start the virtual machine.

This virtual machine is configured for 64-bit guest operating systems. However, 64-bit operation is not possible.

This host supports Intel VT-x, but the Intel VT-x implementation is incompatible with VMware ESX.

For more detailed information, see http://vmware.com/info?id=152.

from vmware.log

2016-05-17T13:31:46.976Z| vmx| I120: guestCPUID level 8000001e, 0: 0x00000000 0x00000000 0x00000000 0x00000000

2016-05-17T13:31:46.976Z| vmx| A115: ConfigDB: Setting monitor.phys_bits_used = "42"

2016-05-17T13:31:46.980Z| vmx| I120: Msg_Post: Error

2016-05-17T13:31:46.980Z| vmx| I120: [msg.cpuid.noLongmode2] This virtual machine is configured for 64-bit guest operating systems. However, 64-bit operation is not possible.

2016-05-17T13:31:46.980Z| vmx| I120+ This host supports Intel VT-x, but the Intel VT-x implementation is incompatible with VMware ESX.

2016-05-17T13:31:46.980Z| vmx| I120+ For more detailed information, see http://vmware.com/info?id=152.

2016-05-17T13:31:46.980Z| vmx| I120: ----------------------------------------

2016-05-17T13:31:46.984Z| vmx| I120: Vigor_MessageRevoke: message 'msg.cpuid.noLongmode2' (seq 2445) is revoked

2016-05-17T13:31:46.984Z| vmx| I120: Module CPUID power on failed.

2016-05-17T13:31:46.984Z| vmx| I120: VMX_PowerOn: ModuleTable_PowerOn = 0

[rawsik]

I tryed to use ubuntu 16.04 with new module version.

At the all VM runs on the ESX but without  VT-x ( VMX flags),  so I have performance issues.  ( more than 4 times )

   ( I chosed   answer.msg.cpuid.noVHVQuestion = "Yes"  )

I tryed different CPU option for KVM but none of them didn't delivery  VT-x into ESX.

for example,  every linux distributive has vmx, when i run it into KVM.   ( cat /proc/cpuinfo )

esxcfg-info |grep "HV Support"

         |----HV Support............................................3

            |----World Command Line.................................grep HV Support

So, I'm dissapointed with ability ESX use VMX from KVM.

If you have any advice, you are welcome

[Doug Covelli]

I'm sorry to hear you are still having trouble. 

> I tryed different CPU option for KVM but none of them didn't delivery  VT-x into ESX.

It is unfortunate that KVM makes it so hard to enable support for nested VMs.  I wish they could make it as easy as VMware does.

> So, I'm dissapointed with ability ESX use VMX from KVM.

I'm quite disappointed that KVM makes it so difficult to expose VT-x to ESX as well.

> If you have any advice, you are welcome

Unfortunately we can not control what features the underlying HW or Hypervisor chooses to expose to us.  My advice is to contact KVM.

[Krishnan Srinivasan]

It looks like you are running on a Nehalem CPU. Have you tried this on an Haswell ?