Can't add NSF datastore

[Jessicasdd]

I really don't know what I am doing wrong. I thought I followed everything but it's not working and it's left me feeling clueless. 

I need to add a  LenovoEMC™ px4-300d as a datastore but I am getting an error:

Operation failed, diagnostics report: Unable to complete Sysinfo operation. Please see the VMkernel log file for more details.: Sysinfo error: TimeoutSee VMkernel log for details.

I don't even know what details to provide, so any help would be greatly appreciated. Tell me what information you need from me and I will provide it.

Thanks,

Jessica

ESXi version:

6.7.0

   

[pcgeek2009]

Can you provide the output from here (please remove any customer specific information):

/var/log/vmkernel.log

The errors may help troubleshoot the issue. 

[vmotiontheinfo]

As recommended by  , vmkernel.log file would really help in understanding what is going on.

Also, I hope you are following this vmware kb to mount nfs on esxi. Kindly check if the nfs mount points have been exported properly and this esxi/cluster has the root access to the mounts or check for the access control on nfs side?

[Jessicasdd]

That is the document that I was following.

And yes I know we are way behind and need to upgrade our VMWare version.

Thanks,

Jessica

[vmotiontheinfo]

Also, it should not be related but v6.7 is out of support, even out of technical guidance too. You should get this upgraded asap.

 

Kindly note that as per this Hardware Compatibility Guide Link, it supports only NFS v4.1.

 

Also, check for the FW ports which are required for NFS to work

[Andre Nordhuus]

Do you have several NFS shares on LenovoEMC™ px4-300d? How is the NFS share that you want to use as a datastore setup? Show us a screenshot of the settings.

A good test of a NFS share is to use a linux machine to see if you are able to mount and browse the share.

[Jessicasdd]

I do not have a Linux machine to be able to test anything. Attached are the screen shots of the NSF setup.

Thanks,

Jessica

[Andre Nordhuus]

Add your Esxi host ip address here:

And please show us how you mount it from the Esxi host.

 

[Jessicasdd]

Just put the IP address in that box? Nothing else?

In ESXi I go to Storage - Datastores tab and click New Datastore - Mount NFS Datastore - Next

I fill in the boxes:

Name - I've tried the NAS' name on the network - NRD-BU1 and I've tried just giving it a name - EMC

NFS Server - 10.0.0.16 - ESXi IP is 10.0.0.11

NFS Share -  /nfs/VM

NFS Version = NFS4

Username - Username

Password - Password

Click Next and Finish.

Thanks,

Jessica

[pcgeek2009]

Yes, just the IP. This says that the host has permission to access the NFS share. You may also try using the IP rather than the name when trying to connect from the ESXi host. The documentation says "For the server name, you can enter an IP address, a DNS name, or an NFS UUID". I have found at times the IP works better. Address any potential DNS issues. Also, have you tried pulling the logs from the path in my previous post and uploading them? That may give us a better idea of the issue.

[Jessicasdd]

I thought I added the log but it's not there. I don't know what happened. I just tried to attach the file that was generated and it says I can't attach it because it is a .tgz file. I don't even know what that is. So maybe that's what happened and I didn't notice it.

Anyway, I am wondering if I am missing a step. Do I need to set up a separate VMKernel NIC for this to work properly?

Thanks,

Jessica

[Jessicasdd]

Ok I found this in the log...

WARNING: NFS41: NFS41ExidNFSProcess:2054: Server doesn't support the NFS 4.1 protocol

And I found another post that said the solution was upgrading the distro. I'm not sure what that means. What do I need to upgrade to get this to work?

Thanks,

Jessica

[pcgeek2009]

Distro would be the version of a Linux server you are running. In your case, you are trying to mount ESXi, so a distro does not apply. I found what I think is the post you are taking about and they mention their Linux box only supported 4.0, not 4.1. It looks like your on only on 6.7.0, no higher versions like  u1, u2, etc. I have been trying to see if 6.7.0 supports 4.1, or just 4.0. It does look like the storage may need to be on firmware version 4.1 or higher. Wondering if it does not fully support 4.1. 

 

[Jessicasdd]

In the VMWare compatibility guide it says this...

"Attention: Storage partners using ESX 4.0 or later may recommend VMW_PSP_RR for path failover policy for certain storage array models. If desired, contact the storage array manufacturer for recommendation and instruction to set VMW_PSP_RR appropriately."

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=san&productid=36586&releases_filter=369&deviceCategory=san&details=1&partner=51&releases=369&isSVA=0&page=8&display_interval=10&sortColumn=Partner&sortOrder=Asc

Thanks,

Jessica

[Jessicasdd]

LOL I didn't see the image you included until after I posted that. 

[Jessicasdd]

Is there maybe an NSF rule that I can create on the NAS that says to use 4.0?

[pcgeek2009]

I am not that familiar with the px4-300d to be able to answer that question. I found a manual, but it did not indicate a way to force it down to 4.0. From looking at your previous screenshots, if you click "NFS" are there any options listed? Is an upgrade possible for your VMware environment?

[Jessicasdd]

I don't know. I can't even get the files to copy from the current datastore (synology DS3612xs) to the local disk. I'm in panic mode at this point. Just hoping the synology stays up long enough to get everything off of it.

I have 2 copies of VMWare running. Maybe if I can upgrade one copy and move the vm's to it and then upgrade the other. What do I need to do that?

Thanks,

Jessica

[pcgeek2009]

Do you mean you have just two ESXi hosts or two vCenter environments?

[Jessicasdd]

I'm sorry... 2 ESXi hosts.

Thanks,

Jessica

[Jessicasdd]

I am trying to download one of the files for one of the VMs to a local disk. Anything to save it at this point. I don't know if I'll be able to download the file for the file server. It's 4TB going to a 20TB disk but I'm afraid it's not going to stay up long enough. The download I am trying now keeps stopping saying it lost internet connection. If I could attach the USB drive to the synology and move the files that way it would probably be easier but I haven't been able to find a way to do that.

Thanks,

Jessica

[pcgeek2009]

Here is a link to how to upgrade from 6.7 to 7 when you do not have vCenter. Maybe it will help:

Upgrading ESXi 6.7 to 7.0 without vCenter | StarWind Blog (starwindsoftware.com)

 

[ThompsG]

Hi ,

 

I'm starting to suspect that the px4-300d might not support NFS4.1. The manual does seem to indicate this in that the one I have which says "Your px4-300d Network Storage uses NFS 4, which improves security and performance". If this is correct, then VMware doesn't support NFS 4 but rather 4.1 as others have mentioned.

 

You can confirm it is a protocol mismatch by doing the following from a SSH session to the ESXi host and running:

 

cat /var/log/vmkernel.log | grep -i "nfs41"

 

 

In this you will likely see a message like: Server doesn't support the NFS 4.1 protocol

 

If the above is true, then all might not be lost depending on the px-300d as it might support the previous NFS protocols with NFS 4 being the maximum supported. To confirm this, you have two options:

 

1. When creating the New Datastore, select NFS 3 rather than 4.1
2. Mount via the SSH sessions with esxcli storage nfs add --host=<NAS IP> --volume-name=<datastore name> --share=<NFS sharename>

 

The command line seems to default to NFS 3 (or at least against the Synology I have it does).

 

Let us know if this works or has added extra confusion?

 

Kind regards.

 

[Jessicasdd]

When I try to add it leaving the selection on NFS3 I get this error:

NFS: 193: NFS mount 10.0.0.16:/VM failed: The mount request was denied by the NFS server. Check that the export exists and that the client is permitted to mount it.

When I try to add it selecting NSF4 I get this error:

WARNING: NFS41: NFS41ExidNFSProcess:2054: Server doesn't support the NFS 4.1 protocol

The synology is failing and I can't get this attached to move the files. I'm about to start copying everything to an external 22TB just to save the files.

Thanks,

Jessica

[ThompsG]

Cool - that is better. This is now telling us that it could connect however doesn't have permission. Much better than before

 

Couple of things to consider as well and sorry for the assumptions. Firstly, is the NFS share name you are using exactly the same as on the NAS? NFS is case-sensitive so "Images" is not the same as "images". Looking at your screenshots I would try "/nfs/VM" for the NFS share name as the NAS interface lists two with one specifying NFS 4.

 

Secondly, can you run this to confirm what IP addresses you have on your host?

 

 

 

esxcli network ip interface ipv4 get

 

 

 

Don't need to post the result here, just want to confirm that you have an IP on the same subnet as the storage or rather which IP might be used to connect to the NAS.  

 

Then, on the NAS you need to add this IP to the NFS Rule (another poster had a screenshot of the section). I'm not exactly sure of the format because the manual is a bit vague but I would try to be as broad as possible to begin with. Not best practice but just to get it running, i.e. if your network is 10.0.0.0 then do 10.0.0.0/255.255.255.0 to cover your bases. Once working you can restrict it better.

 

Hope this makes sense?

[Jessicasdd]

I don't know where to run that code you provided. 

I know that the synology is connected through a switch and they are all on 10.0.0##. I attached the other NAS to this same switch.

I changed the IP rule to what you suggested but it still fails instantly with the (NFS: 193: NFS mount 10.0.0.16:/VM failed: The mount request was denied by the NFS server. Check that the export exists and that the client is permitted to mount it.) error. 

Thanks,

Jessica

[ThompsG]

I see that you are trying 10.0.0.16:/VM - this is the NFS 4 share. Please change to /nfs/VM for the NFS share you are attempting to connect to (this should be the NFS 3 endpoint).

 

The command I provided was to be run from a SSH session on the host. Try the above first and then reply

[Jessicasdd]

It's still kicks out with the same error. I just wanna cry!

Thanks,

Jessica

[ThompsG]

Don't give up - I'm sure we can find the issue

 

Can you post a screenshot of the NFS Rules you have made? Also, what creating a new one looks like?

[Jessicasdd]

[ThompsG]

Lets try two dumb things:

 

1. Create a host entry with *
2. Change your current one from "\" to "/"

[Jessicasdd]

THAT WORKED!!!!! OMG Now I am crying, I'm so happy!

Thank you soooooooo much!

Jessica

[ThompsG]

<blush> you are more than welcome.

 

Which one worked? The wildcard or the forward slash "/"?

[Jessicasdd]

The forward slash. Of course had I noticed it when I first entered it, but I'm glad you did. Now to get the files moved. The synology keeps dropping the connection so it's going to be a process but I'm hopeful. 

Thanks,

Jessica