VMware vSphere

 View Only

  • 1.  Failed to mount NFS datastore - '1234' already exists

    Posted Aug 12, 2021 11:21 PM

    Hi Guys,

    I have recently set up a homelab using ESXi 7u2 and am loving it. So far only one issue.

    I have some extra storage in my NAS that I'm hoping to use for VMs. I haven't had the time to try and work out iSCSI so for the moment I'm trying to use NFS. 

    I add my NFS datastore to ESXi no problems, can even make VMs on it and they run fine. A little slow on the write speed, but fine. 

    Then I reboot my ESXi host and the datastore is no longer present in the web panel, and any VMs that were using it are now broken/won't start. 

    Trying to add the datastore again results in the following error

    Failed to mount NFS datastore 1234 - The specified key, name, or identifier '1234' already exists

     

    I found another thread which prompted me to run some SSH commands:

    esxcli storage nfs list

    Which returns the following:

    [root@hostname:~] esxcli storage nfs list
Volume Name  Host          Share                    Accessible  Mounted  Read-Only   isPE  Hardware Acceleration
-----------  ------------  -----------------------  ----------  -------  ---------  -----  ---------------------
1234         192.168.2.18  /mnt/WDBlack_VMS/Blacks       false     true      false  false  Unknown
[root@hostname:~]

     

    If I run the command to remove the datastore I can then add it back and it works again until next reboot. 

    [root@hostname:~] esxcli storage nfs remove -v 1234

     

    Any ideas on how I can get this working? Thanks in advance. 



  • 2.  RE: Failed to mount NFS datastore - '1234' already exists

    Posted Aug 13, 2021 07:04 AM

    Ciao 

    Are there any errors or warnings in the vmkernel.log file on the ESXi host (they should start with WARNING: NFS :)?

    The network infrastructure used for the NFS part is not indicated but I expect it to be quite standard as it is a home lab, you could check the following ESXi host parameters and change them for example by increasing the HeartbeatMaxFailure number ...

    fabio1975_0-1628838206884.png

     



  • 3.  RE: Failed to mount NFS datastore - '1234' already exists

    Posted Aug 16, 2021 10:17 PM

    Hey Mate, sorry for the delay. 

     

    We have a lot of things in the logs. 

     

    2021-08-12T23:42:26.125Z cpu5:2100741 opID=aa19e531)NFS: 161: Command: (mount) Server: (192.168.2.18) IP: (192.168.2.18) Path: (/mnt/WDBlack_VMS/Blacks) Label: (1234) Options: (None)
2021-08-12T23:42:26.125Z cpu5:2100741 opID=aa19e531)StorageApdHandler: 966: APD Handle f0a122d0-1df3b7cb Created with lock[StorageApd-0x43183e402b40]
2021-08-12T23:42:26.125Z cpu5:2100741 opID=aa19e531)SunRPC: 1092: Destroying world 0x204930
2021-08-12T23:42:26.127Z cpu5:2100741 opID=aa19e531)SunRPC: 1092: Destroying world 0x204931
2021-08-12T23:42:26.127Z cpu5:2100741 opID=aa19e531)SunRPC: 1092: Destroying world 0x204932
2021-08-12T23:42:26.127Z cpu5:2100741 opID=aa19e531)NFS: 347: Restored connection to the server 192.168.2.18 mount point /mnt/WDBlack_VMS/Blacks, mounted as f0a122d0-1df3b7cb-0000-000000000000 ("1234")
2021-08-12T23:42:26.127Z cpu5:2100741 opID=aa19e531)NFS: 222: NFS mount 192.168.2.18:/mnt/WDBlack_VMS/Blacks status: Success
2021-08-12T23:42:26.128Z cpu5:2100741 opID=aa19e531)WARNING: NFS: 2554: Failed to get attributes (I/O error)
2021-08-12T23:42:26.128Z cpu5:2100741 opID=aa19e531)NFS: 2660: [Repeated 3 times] Failed to get object (0x453891f1b3b6) 36 f0a122d0 1df3b7cb 516c145b 5c0431de 22000a 0 f 0 0 0 0 0 :I/O error
2021-08-12T23:42:26.128Z cpu5:2100741 opID=aa19e531)NFS: 2665: Failed to get object (0x45389ed9b346) 36 f0a122d0 1df3b7cb 516c145b 5c0431de 22000a 0 f 0 0 0 0 0 :I/O error
2021-08-12T23:42:26.128Z cpu5:2100741 opID=aa19e531)WARNING: NFS: 2554: Failed to get attributes (I/O error)
2021-08-12T23:42:26.128Z cpu5:2100741 opID=aa19e531)WARNING: NFS: 2554: Failed to get attributes (I/O error)
2021-08-12T23:42:26.128Z cpu5:2100741 opID=aa19e531)NFS: 2660: [Repeated 1 times] Failed to get object (0x45389ed9b346) 36 f0a122d0 1df3b7cb 516c145b 5c0431de 22000a 0 f 0 0 0 0 0 :I/O error
2021-08-12T23:42:26.128Z cpu5:2100741 opID=aa19e531)NFS: 2665: Failed to get object (0x45389ed9b3b6) 36 f0a122d0 1df3b7cb 516c145b 5c0431de 22000a 0 f 0 0 0 0 0 :I/O error
2021-08-12T23:42:26.128Z cpu5:2100741 opID=aa19e531)WARNING: NFS: 2554: Failed to get attributes (I/O error)
2021-08-12T23:42:26.129Z cpu5:2100741 opID=aa19e531)WARNING: NFS: 2554: Failed to get attributes (I/O error)
2021-08-12T23:42:26.129Z cpu5:2100741 opID=aa19e531)WARNING: NFS: 2554: Failed to get attributes (I/O error)
2021-08-12T23:42:26.130Z cpu5:2100741 opID=aa19e531)WARNING: NFS: 2554: Failed to get attributes (I/O error)
2021-08-12T23:42:26.130Z cpu5:2100741 opID=aa19e531)NFS: 2660: [Repeated 3 times] Failed to get object (0x45389ed9b3b6) 36 f0a122d0 1df3b7cb 516c145b 5c0431de 22000a 0 f 0 0 0 0 0 :I/O error
2021-08-12T23:42:26.130Z cpu5:2100741 opID=aa19e531)NFS: 2665: Failed to get object (0x45389ed9b346) 36 f0a122d0 1df3b7cb 516c145b 5c0431de 22000a 0 f 0 0 0 0 0 :I/O error
2021-08-12T23:42:26.130Z cpu5:2100741 opID=aa19e531)WARNING: NFS: 2554: Failed to get attributes (I/O error)
2021-08-12T23:42:26.131Z cpu5:2100741 opID=aa19e531)WARNING: NFS: 2554: Failed to get attributes (I/O error)
2021-08-12T23:42:26.131Z cpu5:2100741 opID=aa19e531)NFS: 2660: [Repeated 1 times] Failed to get object (0x45389ed9b346) 36 f0a122d0 1df3b7cb 516c145b 5c0431de 22000a 0 f 0 0 0 0 0 :I/O error
2021-08-12T23:42:26.131Z cpu5:2100741 opID=aa19e531)NFS: 2665: Failed to get object (0x45389ed9b3b6) 36 f0a122d0 1df3b7cb 516c145b 5c0431de 22000a 0 f 0 0 0 0 0 :I/O error
2021-08-12T23:42:26.131Z cpu5:2100741 opID=aa19e531)WARNING: NFS: 2554: Failed to get attributes (I/O error)
2021-08-12T23:42:26.131Z cpu5:2100741 opID=aa19e531)SunRPC: 1092: Destroying world 0x204934
2021-08-12T23:42:26.131Z cpu5:2100741 opID=aa19e531)SunRPC: 1092: Destroying world 0x204935
2021-08-12T23:42:26.132Z cpu5:2100741 opID=aa19e531)SunRPC: 1092: Destroying world 0x204933
2021-08-12T23:42:26.132Z cpu5:2100741 opID=aa19e531)StorageApdHandler: 1051: Freeing APD handle 0x43183e402b40 [f0a122d0-1df3b7cb]
2021-08-12T23:42:26.132Z cpu5:2100741 opID=aa19e531)StorageApdHandler: 1135: APD Handle freed!

     

     

    And all of those values are already set to what yours were. Have doubles all of them except MaxFailures. That won't let me. 

    Failed - A specified parameter was not correct:

    • Operation failed, diagnostics report: Unable to complete Sysinfo operation. Please see the VMkernel log file for more details.: Bad parameter

     

    2021-08-16T22:13:34.789Z cpu7:2100744 opID=c1a23fc)World: 11986: VC opID esxui-9e9a-a95c maps to vmkernel opID c1a23fc
2021-08-16T22:13:34.789Z cpu7:2100744 opID=c1a23fc)Config: 716: "HeartbeatDelta" = 10, Old Value: 5, (Status: 0x0)
2021-08-16T22:13:40.525Z cpu3:2100738 opID=be65adfd)World: 11986: VC opID esxui-df78-a968 maps to vmkernel opID be65adfd
2021-08-16T22:13:40.525Z cpu3:2100738 opID=be65adfd)Config: 716: "HeartbeatFrequency" = 24, Old Value: 12, (Status: 0x0)
2021-08-16T22:13:59.116Z cpu6:2100288 opID=4388e1c3)World: 11986: VC opID esxui-eefd-a981 maps to vmkernel opID 4388e1c3
2021-08-16T22:13:59.116Z cpu6:2100288 opID=4388e1c3)Config: 716: "HeartbeatTimeout" = 10, Old Value: 5, (Status: 0x0)

     



  • 4.  RE: Failed to mount NFS datastore - '1234' already exists

    Posted Aug 17, 2021 12:50 PM

    Ciao 

    I have remembered a similar issue with a wrong configuration of MTU settings on VMKernel adapter used for the NFS connection.
    The first check in this VMware KB
    https://kb.vmware.com/s/article/1003967



  • 5.  RE: Failed to mount NFS datastore - '1234' already exists

    Posted Aug 20, 2021 12:04 PM

    Hello, 

    I started going through all the steps as the page suggests and got stuck at number 3. 

     

    [root@:~] nc 192.168.2.18 2049
[root@:~]

     

    It just times out? No error in the shell. 

    A quick google about ports that freenas uses (the nfs host) and someone suggested I run a rpcinfo in the freenas shell, which told me that nfs was using that port (correct). 

    lolapples_0-1629412079366.png

    There's no internal firewall so I can't think of a reason this isn't working. 

     

     wrote:

    Ciao 

    I have remembered a similar issue with a wrong configuration of MTU settings on VMKernel adapter used for the NFS connection.
    The first check in this VMware KB
    https://kb.vmware.com/s/article/1003967

    In the help page MTU is just the size of the packet they send to ping. Is that what you mean? 



  • 6.  RE: Failed to mount NFS datastore - '1234' already exists

    Posted Aug 22, 2021 09:30 PM

    Ciao 

    The test on the NFS port must give you positive result, to test it you must also add the -z parameter, so the command you need to use is
    nc -z 192.168.2.18 2049


    and an OK must be returned to you as an answer

    fabio1975_0-1629667744396.png

     


    If it still doesn't respond, check that you have the NFS service running on the ESXi host

    fabio1975_1-1629667767049.png

     

    I was talking about MTU (Jumbo frame) because in the past something similar to yours has happened to me and I have seen that it has happened to others as well.
    So you can try the commands that are indicated or go to check the VMKernel card used for the NFS that MTU has set (same thing you have to do for the network devices)