VMware vSphere

 View Only
Expand all | Collapse all

VMware 7 not seeing LUNs

  • 1.  VMware 7 not seeing LUNs

    Posted Jan 11, 2023 10:31 AM

    Hi All have a weird one here and hoping some has seen this before or has a bright idea.

    Let me start at the begin to try and make sense of this issue and paint you a picture. I am running 3 x Lenovo SR635 servers stand alone, no VCenter to manage and all 3 are connected to a Lenovo DE4000H storage device. I recently upgraded our development server from VMware 6.7u3 to VMware 7.0U3 and all worked fine without any issues. I used the required VMware zip file from the Lenovo support site so that all necessary drives etc. are included.

    I then continued to upgrade our first production server and all hell has broken lose. At first the upgrade appeared to be fine but I was unable to access the GUI VSphere interface from the browser using the IP or DNS name of the server. Logged ticket with Lenovo with little resolution, decided to re-install the server using a USB key. Again I used the ISO image from Lenovo support hoping for a speedy recovery. The network has now recovered and I can access GUI, but now I am unable to see any of the under lying LUNs from the DE4000H so that I can re-create my datastores. I have reconfigured the static targets and inputted the iqn... details, rescanned and refresh but nothing. Error logs do not show any error either.

    I have followed a process found on the www about Network port bindings. I have removed this detail in the config and done a restart of the ESXi host but still I am unable to detect any LUN's. Does anyone have any idea why I am not able to see my LUN's using VMware 7.0U3?

    PS the dev server that upgraded successfully sees the same LUN's that I am trying to connect to on the same VMware version. The only difference between the 2 servers is the BMC firmware and the UEFI version that Lenovo had suggested I upgrade to solve the original network issue. 

    Any help or suggestions would be greatly appreciated.



  • 2.  RE: VMware 7 not seeing LUNs

    Posted Jan 11, 2023 11:27 AM

    Hello.
    From your description you are using an ISCSI connection.

    Check the driver version that the vnic is using and the firmware version of the ethernet adapter.

    # esxcli network nic list

    # esxcli network nic get -n vmnicX

    as an example

    e_espinel_0-1673436270230.png

    Compare with the development server that it is working fine.

    On the Storage side (DE4000H) it is also necessary to validate if the current firmware supports version 6.7U3.

     

     



  • 3.  RE: VMware 7 not seeing LUNs

    Posted Jan 11, 2023 01:07 PM

    Hi

    I have compared the 2 servers and yes the drivers appear to be different. Not sure why this is the case, when I have used the customized VMware file from the Lenovo support page to upgrade both. As far as checking the DE4000H firmware. I assumed that the dev server was working, that firmware was supported. I could not find a affinitive answer if my older SAN firmware is officially supported or not, but seemed to work regardless.

    Working Server:  vmnic0 and vmnic1                                                                   Broken Server vmnic0 and vmnic1

    Driver Info:                                                                                                           Driver Info:
    Bus Info: 0000:45:00:1                                                                                        Bus Info: 0000:41:00:1
    Driver: i40en                                                                                                        Driver: i40en
    Firmware Version: 6.01 0x8000363d 1.1824.0                                                    Firmware Version: 7.00 0x80005183 1.2203.0
    Version: 2.2.4.0                                                                                                    Version: 2.3.4.0

    DE4000H - Version 08.53.00.01.004-LEN  latest version on lenovo downloads seems to 08.74

    If I upgrade the DE4000 firmware, will I have a similar issue on the dev server ? What is the correct order to update?

    E.G. device driver, Server System Firmware, SAN firmware then VMWare ???

    Problem I have I cannot afford for a 2nd server to not be available, as I have used the capacity of the other 2 remaining servers to keep the VMs up and running.

     

     

     



  • 4.  RE: VMware 7 not seeing LUNs

    Posted Jan 11, 2023 04:19 PM

    Hello.
    On the working server the combination driver version 2.2.4.0 and Firmware 6.01 is working, but it would be difficult to get the Broken server to the same versions.

    You can try to upgrade the Brocken server to the following combination Firmware 8.70 and driver 2.3.4.0

     

     



  • 5.  RE: VMware 7 not seeing LUNs

    Posted Jan 11, 2023 04:03 PM

    Some thoughts:

    • check LUN presentation on the storage system, i.e. that the storage presents the LUNs to the new host iqn(s)?
    • make sure that MTU (in case you use Jumbo Frames) is set correctly on the iSCSI vSwitch(es), as well as on the port groups
    • verify connectivity between the iSCSI initiators and the targets using the vmkping utility, which allows you to specify the vmk, and the MTU size, e.g. vmkping -I vmk1 -d -s 8972 <target-IP>

    If you cannot find anything, please provide details about the iSCSI setup (vSwitches, port groups, IP subnets, port binding, etc.)

    André



  • 6.  RE: VMware 7 not seeing LUNs

    Posted Jan 12, 2023 10:50 AM

    Hi, thank you for your reply. I have check the below.

    1. Check LUN presentation on the storage system, i.e. that the storage presents the LUNs to the new host iqn(s)?
    Checked Storage device on both Controller A and Controller B. Both have the correct IP Address and the iqn number. Both show a link status of up.
    2. make sure that MTU (in case you use Jumbo Frames) is set correctly on the iSCSI vSwitch(es), as well as on the port groups
    Done, I have confirmed that both the vswitch and the adapters in question all have the same MTU size of 9000. As well as checking the MTU size on the ports of the storage device.
    3. Verify connectivity between the iSCSI initiators and the targets using the vmkping utility, which allows you to specify the vmk,
    and the MTU size, e.g. vmkping -I vmk1 -d -s 8972 <target-IP>
    Both vmk1 and vmk2 pings return successfully.

    vmkping -I vmk2 -d -s 8972 192.168.145.110
    PING 192.168.145.110 (192.168.145.110): 8972 data bytes
    8980 bytes from 192.168.145.110: icmp_seq=0 ttl=64 time=0.138 ms

    vmkping -I vmk2 -d -s 8972 192.168.145.102
    PING 192.168.145.102 (192.168.145.102): 8972 data bytes
    8980 bytes from 192.168.145.102: icmp_seq=0 ttl=64 time=1.404 ms

    4. If you cannot find anything, please provide details about the iSCSI setup (vSwitches, port groups, IP subnets, port binding, etc.)
    vswitch
    name: storage
    MTU: 9000
    uplink 1: vmnic0 - Up, 10000Mbps
    uplink 2: vmnic1 - Up, 10000Mbps
    Link Discovery
    Mode: Listen
    Protocol: Cisco discovery protocol (CDP)
    Security
    promiscuous: Reject
    MAC address change: Accept
    Forge Transmits: Accept
    NIC teaming
    Load Balancing: Route based on originating Port ID
    Network failover detection: Link status only
    Notify Switch: Yes
    Failback: Yes
    Failover Order: vmnic0, 10000Mbps,full duplex, Active
    vmnic1, 10000Mbps,full duplex, Active
    Traffic shaping: Disabled

    Port Groups
    Name          Virtual Switch Active Clients VLAN ID
    StorageA      Storage                           1          0
    StorageB       Storage                           1         0

    VMK1
    Port Group: StorageA
    MTU: 9000
    IPversion: IPv4 and IPv6          (Have tried to make it IPv4 only but never seems to keep setting)
    IPv4 settings
    configuration: Static
    Address: 192.168.135.110
    Subnet: 255.255.255.0
    Services: vMotion, Provisioning, Fault tolerance logging, Replication, NFC replication

    VMK2
    Port Group: StorageB
    MTU: 9000
    IPversion: IPv4 and IPv6            (Have tried to make it IPv4 only but never seems to keep setting)
    IPv4 settings
    configuration: Static
    Address: 192.168.145.110
    Subnet: 255.255.255.0
    Services: vMotion, Provisioning, Fault tolerance logging, Replication, NFC replication

    Port Bindings (Have tried removing this from vmhba64 but makes no difference)
    VMK0
    Port Group: Management Network
    MTU: 1500
    IP Version: IPv4 and IPv6 (Have tried to make it IPv4 only but never seems to keep setting)
    IPv4 settings
    configuration: Static
    Address: 172.16.xxx.xxx
    Subnet: 255.255.255.0
    Services: Management

    #esxcli network nic get -n vmnic0
    Advertised Auto Negotiation: true
    Advertised Link Modes: Auto, 10000BaseCR4/Full
    Auto Negotiation: false
    Cable Type: DA
    Current Message Level: 0
    Driver Info:
    Bus Info: 0000:41:00:0
    Driver: i40en
    Firmware Version: 7.00 0x80005183 1.2203.0
    Version: 2.3.4.0
    Link Detected: true
    Link Status: Up
    Name: vmnic0
    PHYAddress: 0
    Pause Autonegotiate: false
    Pause RX: false
    Pause TX: false
    Supported Ports: DA
    Supports Auto Negotiation: true
    Supports Pause: true
    Supports Wakeon: false
    Transceiver:
    Virtual Address: 00:50:56:52:13:a7
    Wakeon: None

    #esxcli network nic get -n vmnic1
    Advertised Auto Negotiation: true
    Advertised Link Modes: Auto, 10000BaseCR4/Full
    Auto Negotiation: false
    Cable Type: DA
    Current Message Level: 0
    Driver Info:
    Bus Info: 0000:41:00:1
    Driver: i40en
    Firmware Version: 7.00 0x80005183 1.2203.0
    Version: 2.3.4.0
    Link Detected: true
    Link Status: Up
    Name: vmnic1
    PHYAddress: 0
    Pause Autonegotiate: false
    Pause RX: false
    Pause TX: false
    Supported Ports: DA
    Supports Auto Negotiation: true
    Supports Pause: true
    Supports Wakeon: false
    Transceiver:
    Virtual Address: 00:50:56:5f:77:45
    Wakeon: None

    Still not seeing any devices from the storage unit on vmware under Storage->Devices.

    Any more thoughts of things to check.

    PS: I tried removing everything to do with the storage, vswitch, port groups, vmhba etc. and add it all back again, was grasping at straws.

    Now it will not save the 2nd static target under Storage->Adapters->vmhba64. No errors just not there after clicking save button and then checking again



  • 7.  RE: VMware 7 not seeing LUNs

    Posted Jan 12, 2023 04:48 PM

    Ok, let's go through the iSCSI related settings:

    With two VMKernel groups in different subnets,

    VMK1
    Port Group: StorageA
    MTU: 9000
    Address: 192.168.135.110

    VMK2
    Port Group: StorageB
    MTU: 9000
    Address: 192.168.145.110

    each VMkernel port group has to be associated with a single, dedicated uplink (vmnic), i.e. one vmnic is "Active", and all other vmnics have to be set to "Unused" (not "Standby ! ).
    Since the storage system is accessed using two different subnets, you must not configure Port Binding in the software iSCSI adapter, but only configure the dynamic/static targets.

    As a side note: Check your storage vendor's documentation for how to configure "Delayed Ack". Many vendors recommend to disable "Delayed Ack" for better performance.

    André



  • 8.  RE: VMware 7 not seeing LUNs

    Posted Jan 13, 2023 05:02 PM

    I have split the vmkernel devices to each have there own Vswitch and port group. See settings below.

    I have also rolled back to version 6.7 using the Lenovo customised ISO file and yet the problem persists.

    VSwitch (StorageA)
    MTU 9000
    Uplink 1 vmnic0 - Up, 10000 mbps
    Link discovery
    Mode Listen
    Protocol Cisco discovery protocol (CDP)
    Security
    Promiscuous mode Reject
    MAC address changes Reject
    Forged transmits Reject
    NIC teaming
    Load balancing Route based on originating port ID
    Network failover detection Link status only
    Notify switches Yes
    Failback Yes
    Failover order
    Name Speed Status
    vmnic0 10000 Mbps, full duplex Active
    Traffic shaping
    Status Disabled
    Average bandwidth 100000 kb/s
    Peak bandwidth 100000 kb/s
    Burst size 102400 KB
    Note
    Traffic shaping policy is applied to the traffic of each virtual network adapter attached to the virtual switch.

    VSwitch (StorageB)
    MTU 9000
    Uplink 1 vmnic1 - Up, 10000 mbps
    Link discovery
    Mode Listen
    Protocol Cisco discovery protocol (CDP)
    Security
    Promiscuous mode Reject
    MAC address changes Reject
    Forged transmits Reject
    NIC teaming
    Load balancing Route based on originating port ID
    Network failover detection Link status only
    Notify switches Yes
    Failback Yes
    Failover order
    Name Speed Status
    vmnic1 10000 Mbps, full duplex Active
    Traffic shaping
    Status Disabled
    Average bandwidth 100000 kb/s
    Peak bandwidth 100000 kb/s
    Burst size 102400 KB
    Note
    Traffic shaping policy is applied to the traffic of each virtual network adapter attached to the virtual switch.

    vmk1
    Port group StorageA
    MTU 9000
    IP version IPv4 and IPv6
    IPv4 settings
    Configuration Static
    Address 192.168.133.110
    Subnet mask 255.255.255.0
    IPv6 settings Click to expand
    TCP/IP stack Default TCP/IP stack
    Services None ticked

    vmk2
    Port group StorageB
    MTU 9000
    IP version IPv4 and IPv6
    IPv4 settings
    Configuration Static
    Address 192.168.143.110
    Subnet mask 255.255.255.0
    IPv6 settings Click to expand
    TCP/IP stack Default TCP/IP stack
    Services None ticked



  • 9.  RE: VMware 7 not seeing LUNs

    Posted Jan 12, 2023 07:34 PM

    Hello.
    I once had problems configuring ISCSI, but changing my browser solved it (personally I use Waterfox G.X or Waterfox Clasic, which are based on Firefox).

    Which model is your Lenovo SR635 7Y98 or 7Y99?

     

     



  • 10.  RE: VMware 7 not seeing LUNs

    Posted Jan 13, 2023 04:38 PM

    Hi

    Sorry for the delay, it has taken me a while to figure out how to do the upgrade. The 10GbE SFP adapters are now upgraded to a later version firmware.

    esxcli network nic get -n vmnic0

    Driver Info:
    Bus Info: 0000:41:00:0
    Driver: i40en
    Firmware Version: 8.40 0x8000b67e 1.3079.0
    Version: 1.16.3.0

    esxcli network nic get -n vmnic1

    Driver Info:
    Bus Info: 0000:41:00:1
    Driver: i40en
    Firmware Version: 8.40 0x8000b67e 1.3079.0
    Version: 1.16.3.0

    However I still am unable to detect any LUN's on the DE4000H.

     

    I should also mention we have re-installed version 6.7u3 from the Lenovo customised files. But get the same symptom. 

    I have tried Firefox 108.0.2 (64-bit), Chrome Version 109.0.5414.74 and Edge Version 108.0.1462.76

     



  • 11.  RE: VMware 7 not seeing LUNs

    Posted Jan 14, 2023 12:54 AM

    Hello.
    If your network card is ThinkSystem Intel X710-T2L 10GBASE-T 2-Port PCIe Ethernet Adapter (4XC7A80266) then the driver version you should use is VMware ESXi 6.7 i40en 1.17.2.0 NIC Driver

    For version 7.0, the driver must be  VMware ESXi 7.0 i40en 2.2.4.0 NIC Driver  or VMware ESXi7.0 i40en 2.3.4.0 NIC Driver.

    attached link where to download it

    https://customerconnect.vmware.com/downloads/details?downloadGroup=DT-ESXI67-INTEL-I40EN-11720&productId=742

    https://customerconnect.vmware.com/en/downloads/details?downloadGroup=DT-ESXI70-INTEL-I40EN-2240&productId=974

    https://customerconnect.vmware.com/en/downloads/details?downloadGroup=DT-ESXI70-INTEL-I40EN-2340&productId=974

    After making any changes you must run (at least 3 times) the rescan storage and HBA.
    If your ethernet card is not as indicated, please send the full details of your vnic ethernet card.

    If the problem continues, I offer you a free remote session to try to reconfigure the ISCSI on that ESXi host.
    my time zone is UTC -5 to my personal email: eespinel@gmail.com, please contact me to coordinate date and time.

     



  • 12.  RE: VMware 7 not seeing LUNs

    Posted Jan 16, 2023 12:28 PM

    Hi

    I have gone to the URL you have supplied and down loaded the driver file (Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip) for VMware 6.7.

    However when I use the command : esxcli software vib update -d /vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip  I get the below error:

    [MetadataDownloadError]
    Could not download from depot at zip:/vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip?index.xml, skipping (('zip:/vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip?index.xml', '', 'Error extracting index.xml from /vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip: "There is no item named \'index.xml\' in the archive"'))
    url = zip:/vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip?index.xml
    Please refer to the log file for more details.

     

    Checking the /var/log esxi.log file I see an entry

    [2023-01-16 12:00:55,433 root ERROR] update failed: [MetadataDownloadError]
    Could not download from depot at zip:/vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip?index.xml, skipping (('zip:/vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip?index.xml', '', 'Error extracting index.xml from /vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip: "There is no item named \'index.xml\' in the archive"'))
    url = zip:/vmfs/volumes/datastore1/Intel-i40en_1.17.2.0-1OEM.670.0.0.8169922-19985261.zip?index.xml
    Please refer to the log file for more details.

    I know the Lenovo installation likes you to have an XML file in the directory that is usually available with the file.

    I have gone to Lenovo support to try and fine the same driver https://datacentersupport.lenovo.com/gb/en/products/servers/thinksystem/sr635/7y99/7y99cto1ww/j301zd4n/downloads/driver-list/component?name=Networking%3A%20LAN%20%28Ethernet%29 file VMware drivers package release for Intel Ethernet, but get the same result.

    I have checked the readme file and it would appear I am doing the right thing, but yet it is not working.

    I have verified the checksum and get the correct output.

     



  • 13.  RE: VMware 7 not seeing LUNs

    Posted Jan 16, 2023 01:16 PM

    Although I don't think that it's a driver issue, note that the downloaded .zip archive contains the driver .zip that you need to extract.

    ANdré



  • 14.  RE: VMware 7 not seeing LUNs
    Best Answer

    Posted Jan 17, 2023 01:49 PM

    Hi

    Finally found the issue, just thought I would put the update here for future.

    Seems that during my clean up of all the network and storage components on the VMWare, caused me to get a new server initiator address, while adding the software ISCSI device back.

    Once I had noticed the initiator name for the software ISCSI (vmhba64) did not correspond between the VMWare and the DE4000H device, I had to deleted the host from the DE4000H setup from the clustered hosts, re-added it back again with the new initiator address. Did a rescan on the VMware once more and can now we can see the LUNs that has allowed me to recreate the datastores.

    Thanks all for your support and patients, you guys rock.

    Now to get back to my original task,  upgrade to VMWare 7, fingers crossed it works smoothly. Or I'll be back



  • 15.  RE: VMware 7 not seeing LUNs

    Posted Jan 16, 2023 06:43 PM

    Hello.
    From the zip file you must extract the .vib file ( INT_bootbank_i40en_1.17.2.0-1OEM.670.0.0.8169922.vib)

    then take it to the ESXi and run the command

    esxcli software vib update -v {VIBFILE}