ESXi 6.7 fails to list Intel X540 (ixgben) Network Ports via a Thunderbolt3 to Thunderbolt2 Apple Adapter

[philaneous]

Environment

• Intel Hades Canyon​
  • 64GB (2 x 32GB DDR4 2400 MHz)
• ESXi 6.7 (15820472)
  • Boot Drive = 16GB USB 3.1 Flash Drive
• Promise SANLink2 10GbE (Thunderbolt2)​
  • Uses the Intel X540 AT2 Ethernet Controller​
  • Connected to the Intel NUC via an Apple Thunderbolt3 to Thunderbolt2 Adapter

Problem Statement

ESXi fails to list the Intel X540 (ixgben) as configurable network ports via a Thunderbolt3 to Thunderbolt2 Apple Adapter.

1. On ESXi, the Intel X540 PCI card does get detected and is properly listed under Hardware -> PCI Devices:
2. 
   [root@esxinuc:~] lspci | grep X540
   0000:40:00.0 Ethernet controller: Intel(R) Ethernet Controller 10 Gigabit X540-AT2
   0000:40:00.1 Ethernet controller: Intel(R) Ethernet Controller 10 Gigabit X540-AT2
   
3. You can clearly see that the ESXi detects the PCI card; however, there seems to be some addressing issue on the PCI side of things; as it fails to list the (2) available network ports.
4. In addition, I have an Apple Thunderbolt to Gigabit Ethernet Adapter (Broadcom Corporation NetXtreme BCM57762) connected via the same Thunderbolt chain and the network port get detected with the tg3 inbox driver. It appears that ESXi fails to address the Intel X540 with the ixgben driver when the TB3 to TB2 adapter is the intermediary between the host and card.

5. [root@esxinuc:~] esxcli network nic list

   Name    PCI Device    Driver  Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description

   ------  ------------  ------  ------------  -----------  -----  ------  -----------------  ----  --------------------------------------------------------

   vmnic0  0000:00:1f.6  ne1000  Up            Down             0  Half    d4:5d:df:07:4f:59  1500  Intel Corporation Ethernet Connection (2) I219-V

   vmnic1  0000:05:00.0  igbn    Up            Up            1000  Full    d4:5d:df:07:4f:5a  1500  Intel Corporation I210 Gigabit Network Connection

   vmnic2  0000:44:00.0  tg3     Up            Down             0  Half    38:c9:86:22:e4:b0  1500  Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet

6. The problem would have made more sense if the PCI card didn't get detected by ESXi; however, the fact that it is detected, ESXi should be able to assign the ixgben driver.
7. The problem isn't the SANLink2 10GbE (Intel X540) but an MMIO addressing issue amongst the Thunderbolt bus translation to ESXi when using a TB3 to TB2 adapter; hence, the ixgben driver never gets assigned.
8. Disclaimer – I understand that:
   (A) the Intel NUC is not officially part of the HCL (many people in the community use NUCs for home labs)
   (B) an Apple Thunderbolt3 to Thunderbolt2 adapter is not officially part of the HCL
   
9. Please note that if I boot in Windows 10 with the same HW (Intel NUC) and physical configuration (SL2 via TB3 to TB2), the Intel X540 contained in the Promise SANLink2 10GbE via the Thunderbolt bus works perfectly fine:
   
   
10. For those wondering if the Promise SANLink2 10GbE works natively using the ixgbe (ESXi 6.0) or the newer driver ixgben (ESXi 6.5, 6.7), when connecting the Promise SANLink2 10GbE to a Thunderbolt2 or Thunderbolt1 host running on ESXi, it works perfectly fine. ESXi automatically assigns the driver to the Intel Ethernet Controller X540.
    ESXi 6.7 with SL2 connected to a TB1 MacBook Pro
    
11. I've used the SL2 with multiple Thunderbolt1/2 based Macs that are running ESXi and they work great.
12. Since I had (1) laying around, I thought I would test using the TB3 to TB2 adapter before looking for TB3 based 10GbE adapters; as the ones listed in Virtually Ghetto.

The vmkwarning.log outputs PCI MMIO limit exceeded warnings:

0:00:00:05.485 cpu0:2097152)WARNING: PCI: 452: 0000:40:00.0: Failed to add BAR[2] (IO f=0x1 0xa020-0xa040) - out of resources on parent: 0000:3f:03.0

0:00:00:05.485 cpu0:2097152)WARNING: PCI: 475: 0000:40:00.0: Failed to add BAR[2] (IO f=0x1 0xa020-0xa040) status: Limit exceeded

0:00:00:05.485 cpu0:2097152)WARNING: PCI: 221: IO (0xa000 - 0xa01f) out of range

0:00:00:05.485 cpu0:2097152)WARNING: PCI: 452: 0000:40:00.1: Failed to add BAR[2] (IO f=0x1 0xa000-0xa020) - out of resources on parent: 0000:3f:03.0

0:00:00:05.485 cpu0:2097152)WARNING: PCI: 475: 0000:40:00.1: Failed to add BAR[2] (IO f=0x1 0xa000-0xa020) status: Limit exceeded

0:00:00:05.496 cpu0:2097152)WARNING: PCI: 678: 0000:40:00.0: Unable to free BAR[2] (IO f=0x1 0xa020-0xa040): Limit exceeded

0:00:00:05.496 cpu0:2097152)WARNING: PCI: 678: 0000:40:00.0: Unable to free BAR[4] (MEM64 f=0xc 0x2fd0200000-0x2fd0204000): Limit exceeded

0:00:00:05.496 cpu0:2097152)WARNING: PCI: 678: 0000:40:00.1: Unable to free BAR[2] (IO f=0x1 0xa000-0xa020): Limit exceeded

0:00:00:05.496 cpu0:2097152)WARNING: PCI: 678: 0000:40:00.1: Unable to free BAR[4] (MEM64 f=0xc 0x2fd0600000-0x2fd0604000): Limit exceeded

[root@esxinuc:~] lspci | grep 0000:3f:03.0

0000:3f:03.0 PCI bridge: Intel Corporation DSL5520 Thunderbolt 2 Bridge [Falcon Ridge 4C 2013]

- TB2 Bridge via Apple TB3-to-TB2 Adapter (Parent)

0:00:00:05.496 cpu0:2097152)WARNING: PCI: 678: 0000:40:00.0: Unable to free BAR[4] (MEM64 f=0xc 0x2fd0200000-0x2fd0204000): Limit exceeded

0:00:00:05.496 cpu0:2097152)WARNING: PCI: 678: 0000:40:00.1: Unable to free BAR[2] (IO f=0x1 0xa000-0xa020): Limit exceeded

- Promise SANLink2 10GbE (Intel Ethernet Controller 10 Gigabit X540-AT2)

It appears that I'm experiencing a PCI MMIO issue with the adapter and verified this by reviewing Memory Addressing Limits for ESXi 5.1.x, 5.5.x 6.x and 7.0 (2087943)

I couldn't figure out a way to change this on the Intel NUC with the latest BIOS. :smileysad:

[philaneous]

I never posted about my testing with newer TB3 based Mac machines while using the TB3-to-TB2 adapter, since previously, nothing would be detected via the Thunderbolt3 bus.

1. I recently discovered a posting from lamw whom is the blog owner of https://virtuallyghetto.com and works for VMware.
2. Some of the information he posts on his blog ultimately comes from internal knowledge; hence, without him providing these workarounds to the community publicly, we'd all be more frustrated on getting resolutions to our home-lab deployments. For that, I appreciate the insight he provides the community.
3. Recently, there was a posting on his blog that was shared via Slack https://vmwarecode.slack.com regarding a flag you set in the kernel in order for TB3 peripherals to be detected by the kernel: https://www.virtuallyghetto.com/2020/04/update-on-esxi-on-apple-mac-mini-2018-mac-pro-2019.html
4. Please note that the above is resolved by updating to ESXi 6.7 Patch 02 (ESXi670-202004002).
5. After applying the patch to my USB based ESXi install, I booted my MacBook Pro 15,1 machine and the Intel X540 was properly detected via the Promise SANLInk2 10GbE connected via TB3-to-TB2 adapter. (I wanted to test another TB3 based machine with ESXi as a sanity check and future reference.)
6. At this point, this HW configuration just doesn't work on the Intel Hades Canyon + ESXi and would love input from anyone at VMware to resolve this.
7. At this juncture, I elected to go with the (2) OWC Thunderbolt 3 10G Ethernet Adapter 10GBASE-T Adapters, which per the posting by lamw, it seems to properly detect it and listed per – https://www.virtuallyghetto.com/2019/04/new-thunderbolt-3-to-10gbe-options-for-esxi.html​ for the Intel NUC Hades Canyon. The only thing is that the blog fails to test dual port 10Gb NICs via TB as there aren't a lot of TB3 dual port 10GbE options. Luckily, the Intel NUC Hades Canyon is equipped with (2) TB3 ports.
8. The only TB3 dual 10GbE adapter that you can purchase in the market for a steep price of $499.99 – Sonnet Twin10G (Thunderbolt 3 Edition)​ which you may as well build your own enclosure using the ​​OWC Mercury 3.​ Essentially, some of us want to MPIO to our storage target.
9. I really wish the known advocates of the Intel NUC Hades Canyons, to test dual ported NIC options for the NUC. I've read forum postings that some have elected to go with the OWC Mercury 3 and put in a PCI card; however, I've read a few forum postings detailing their unsuccessful results.
   • VMware forum of a few people having similar issues with their Intel NUCs loading drivers via PCI cards housed via an OWC Mercury 3
   • fizzyade​​ gave up as the Intel NUC BIOS has no ability to change any PCI MMIO settings in the BIOS and installed Proxmox​
   • For these reasons above, I didn't want to spend the money and time testing which PCI cards (any of the VMware HCL 10GbE PCI cards) would work; moreover, just want to get the NUC online at this point with (2) 10GbE paths to my storage target.
10. With item #9 being stated, I came to the conclusion that building a PCI TB3 enclosure with a PCI dual 10Gb card would cost the same (if all purchased new) as buying (2) OWC TB3 10GbE adapters. Also, this adapter (uses the Aquantia chipset) has been tested in https://www.virtuallyghetto.com/2019/04/new-thunderbolt-3-to-10gbe-options-for-esxi.html​ and VMware now includes the Aquantia driver as an inbox/native driver.
11. As long as you're OK losing your TB ports, purchasing (2) OWC Thunderbolt 3 10G Ethernet Adapter 10GBASE-T Adapters is not a terrible solution; especially if your end goal is to MPIO to your storage target or simply need to meet your bandwidth requirements.
12. It seems that the TB firmware on the Intel NUC Hades Canyon is simply not optimized for VMware ESXi; moreover, ESXi is not optimized to work with all TB peripherals on an Intel NUC even if lspci properly lists the PCI card :smileyhappy:. (While I still understand that the Intel NUC Hades Canyon is not part of the ESXi HCL).

Below is an example of a TB3 MacBook Pro 15,1 using the TB3-to-TB2 adapter with the Promise SANLink2 10GbE working fine after the ESXi670-202004002 patch:
(Same HW configuration that fails on the NUC):

[root@esxinuc:/var/log] esxcli network nic list

Name    PCI Device    Driver  Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description

------  ------------  ------  ------------  -----------  -----  ------  -----------------  ----  ---------------------------------------------------------

vmnic0  0000:0a:00.0  ixgben  Up            Up           10000  Full    00:01:55:14:6e:8f  1500  Intel Corporation Ethernet Controller 10 Gigabit X540-AT2

vmnic1  0000:0a:00.1  ixgben  Up            Up           10000  Full    00:01:55:14:6e:90  1500  Intel Corporation Ethernet Controller 10 Gigabit X540-AT2

vmnic2  0000:0e:00.0  tg3     Up            Up            1000  Full    38:c9:86:22:e4:b0  1500  Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet

[root@esxinuc:/var/log] lspci | grep X540

0000:0a:00.0 Ethernet controller: Intel(R) Ethernet Controller 10 Gigabit X540-AT2 [vmnic0]

0000:0a:00.1 Ethernet controller: Intel(R) Ethernet Controller 10 Gigabit X540-AT2 [vmnic1]

[root@esxinuc:/var/log] esxcli hardware platform get

Platform Information

   UUID: 0x28 0x64 0xf3 0xd1 0x83 0xfe 0x57 0x7c 0x93 0x36 0x13 0x66 0xfc 0x9e 0xb6 0x42

   Product Name: MacBookPro15,1

   Vendor Name: Apple Inc.

   Serial Number: <omitted>

   Enclosure Serial Number: <omitted>

   BIOS Asset Tag:

   IPMI Supported: false

If my MacBook Pro 15,1 can do exactly what I'm trying to accomplish via ESXi; this should be achievable on the NUC as well. Off to a new adventure. :smileyhappy:

[fizzyade]

Wow, a blast from the past!  I'm fizzyade.

I did indeed give up, nothing I did or tried could get the thing to work in ESXi, I spent quite a bit of money on various different cards with all the same results before I realised the issue with the MMIO mapping.

I installed Proxmox and it all worked straight away, off the bat, without any tinkering.  Been running it since that post and haven't had any issues.

[jhiguera99]

I have the same issue than you, my setup is NUC7i7BNH with a SONNET Echo Express SEL I and a SFC5122 10gb card. I get the same MMIO / PCI BAR  error, even if the card is listed with lspci. I tested the same setup but booting with Ubuntu 20 and it works perfectly, the two 10gb ports show up and work.

 

So the issue is with ESX. It doesnt handle well the PCI cards installed in TB2/3 enclosures.

[Joe42]

In ESXi 6.7 the Sonnet TB3 10G Solo Adapter works flawless with the 1.0.2.0 Driver Bundle. However, the same Setup won't work anymore with ESXi 8.0. 

I tried different driver versions (even downgrading from 1.0.3.0 to the before-working 10.2.0).

The adapter is listed by lspci. But the firmware can't be initialized (FW kickoff failed).