Cannot Download .vmdk files from VM - "Storage expected but message Got: ERROR"

[ITVM]

Hi

I have some VMs, that i cannot download the .vmdk using Datastore Browser. I get "Storage expected but message Got: ERROR"

This only happen with some VMs, not with all my VMs.

It seems that the VM is locked. I have check if there is any ISO in the CD/DVD, there is none.

I have restart one of the servers and try to backup again, still no luck, same error.

All this VMs, are using the same Storage

Recently I have upgrade this infrastructure to vShpere, all this VMs have the VMware tools and Virtual Hardware updated

Any ideas what is locking this VMs preventing them to download?

I only notice this, when i start to use Veeam to backup the VMs

Thank You

LP

[osha]

if you are in a hurry, you can login to the console of the ESX and just scp the files from /vmfs/volumes/datastore-name/file-name to your machine.

If you need additional details, tell me.

[ITVM]

Hi

Sorry but SCP do not work also.

[marcelo_soares]

2 questions:

- When you copy the files using datastore browser, are the VMs powered off?

- What is the veeam error you are receiving? Is the same one?

Marcelo Soares

VMWare Certified Professional 310/410

Technical Support Engineer

Globant Argentina

Consider awarding points for "helpful" and/or "correct" answers.

[ITVM]

Hi marcelo

Yes I am downloading with the VMs power on. I have tried with the VM powen off, all i can download the files. But I cannot backup this with power off

In the Veeam is similar

<23> Info (Client) Service error: File does not exist or locked. VMFS path: [[http://MYSTORAGE] VMServer/VMServer.vmx|http://MYSTORAGE] VMServer/VMServer.vmx].

<23> Info (Client) Service error: Please, try to download specified file using connection to the ESX server where the VM registered.

<23> Info (Client) Service error: --tr:DiskLib error: .File error -- File open failed: Generic error.

<23> Info (Client) Service error: --tr:Failed to start file downloading. VMFS path: [[http://MYSTORAGE] VMServer/VMServer.vmx|http://MYSTORAGE] VMServer/VMServer.vmx].

<23> Info (Client) Service error: --tr:NFC storage connection is unavailable. Storage: . Storage display name: .

<23> Info (Client) Service error: Failed to create NFC download stream. NFC path: .

<23> Info (Client) Service error: --tr:Unable to open source file .

<23> Info (Client) Service error: --tr:Failed to copy local file. Source: . Target: .

<23> Info (Client) Service error: --tr:Unable to get content of the file. File path: .

<18> Info (Server) Service error: End of file

Thank You

LP

[ITVM]

Hi

I have notice that this only happen with the VMs that are in one of the host. All the VMs that I have in the other host, I can download the .vmdk files with the VM power on. And the Storage is the same.

I check all the settings and see no difference between the two hosts.

LP

[marcelo_soares]

Check mainly DNS/hostname settings and routing.

Marcelo Soares

VMWare Certified Professional 310/410

Technical Support Engineer

Globant Argentina

Consider awarding points for "helpful" and/or "correct" answers.

[ITVM]

Marcelo,

Since HA and DRS are enable, I think this is not an issue. Because then the HA agent will not work.

But I will check anyway.

LP

[marcelo_soares]

I think we are getting something weird here. You cannot download the VMDK while it is in use. You must first take an snapshot of the VM an then take the VMDK file - which will not be locked because of the snapshot. The veeam software are trying to download the the files also with or without snapshotting?

I really can't understand why you are able to dowload powered on VM VMDK's from other hosts.

Marcelo Soares

VMWare Certified Professional 310/410

Technical Support Engineer

Globant Argentina

Consider awarding points for "helpful" and/or "correct" answers.

[ITVM]

Hi

Yes you are correct.

I have tested using snapshot created temporally by veeam, or a snapshot created for this tests.

LP

[marcelo_soares]

If you have the opportunity of doing a "tail -f /var/log/vmware/hostd.log" while generating the error it would be good.

Marcelo Soares

VMWare Certified Professional 310/410

Technical Support Engineer

Globant Argentina

Consider awarding points for "helpful" and/or "correct" answers.

[ITVM]

Hi

For a better understanding the I add a txt file with the tail log

The file that I try to download was VMSQL01-flat.vmdk from the VMSQL01 server

But looking at the log I cannot see anything wrong.

LP

[marcelo_soares]

Will take a look. Other weird thing: you should not be able to see the flat files from the browse datastore, unless they are not together with their respective vmdk header. You are sure this files belong to a healthy VM?

Marcelo Soares

VMWare Certified Professional 310/410

Technical Support Engineer

Globant Argentina

Consider awarding points for "helpful" and/or "correct" answers.

[ITVM]

Hi

Nope when I try to download I only see the VMSQL01.vmdk , not the FLAT, when I start to copy, then shows the VMSQL01-flat.vmdk trying to copy.

In the destination folder I have after the error the VMSQL01.vmdk and VMSQL01-flat.vmdk both are zero bytes.

LP

[marcelo_soares]

Nothing on logs... maybe at the vi client logs you will find something. I don't know anymore where to look for.

Marcelo Soares

VMWare Certified Professional 310/410

Technical Support Engineer

Globant Argentina

Consider awarding points for "helpful" and/or "correct" answers.

[ITVM]

Hi

Ok after some tests I have check that when i try to download from host02 i get this error, but VMs from host01 dont have this error. So I migrate 2 VMs from host02 to host01(just migrate host to host, not Storage) and try to donwload the files... i was 100% success

So my problem is in the host02, but where... :smileysad:

I will migrate all the VMs from host02 to host01 and remove this host from the cluster, restart the host and add again to the cluster to see if i have any changes.

LP

[ITVM]

Hi

Removing the host from the cluster did not correct the problem

I have checked the FC Swtich and double check the zone etc., to see if there is any difference between the hosts, nothing wrong there.

I also have checked the Storage Admin and check the Groups, and host connections to the LUN, no difference found and i think no problem here also.

So the problem must be on the Host02(VMware configuration, or adapters) or hardware problem. I cannot see any other problem

Any one as any idea where the problem can be found?

Thank You

LP

[marcelo_soares]

Look, the next step is to check for some hardware errors. Paste the vmkernel output here during the error.

Marcelo Soares

VMWare Certified Professional 310/410

Technical Support Engineer

Globant Argentina

Consider awarding points for "helpful" and/or "correct" answers.

[ITVM]

Hi Marcelo,

I think I have found the problem when checking the vmkernel.

I have a storage path that is dead, and maybe here is the problem. But I think is a path that is not used, and is not needed, but maybe the problem is here.

Let me say that this start to happen when adding a new VMware host to the cluster, and create new FC connections etc.

I will add a file with the vmkernl log, and 4 images from my host1 and host2 storage adapters.

Do I need to remove this dead path?

Thank You

LP

[ITVM]

Next 2 images

LP

[marcelo_soares]

If you can, you can try removing the path. As I said, I never had a problem like this.

After fixing in the storage side, rescan everything and try again.

Marcelo Soares

VMWare Certified Professional 310/410

Technical Support Engineer

Globant Argentina

Consider awarding points for "helpful" and/or "correct" answers.

[ITVM]

Hi

Removing the path in the VI Client I think is not possible.

I think that we can remove by using the vSphere Command-Line Interface, but I never used this before. I have the vSphere Management Assistant installed.

LP

[marcelo_soares]

To remove a path, you must hide it from ESX and rescan the storage adapters. You maybe are having some issues on your storage side, because the problem seems to be located on the target 0 of it. The other ESX hosts can see all the 3 paths?

Marcelo Soares

VMWare Certified Professional 310/410

Technical Support Engineer

Globant Argentina

Consider awarding points for "helpful" and/or "correct" answers.

[ITVM]

Hi

Yes they can. As you can see the screenshots that i add above, there is the path from host01(that is ok), and screenshots from host02(that have this problem)

In my Storage I have 2 Raid Group. The first Raid Group have 3 LUNs(0, 1, 3). The LUN 1 and 2 are the local disk for host01 and host02(they boot from the Storage), and the LUN 3 is the LUN that holds all this VMs

Now i have upgraded the Storage with more 10 disks(5Tb). This is the 2 Raid Group, that have 3 Luns(0, 1, 3). This LUNs are for the VMs, and for the new host that as exists. This new host don't need LUN to boot from, have is own local disks.

Both Raid Group have the host, 1,2,3 added. Each host have 2 FC connections to the storage

I move all VMs from this host to the host01, and then remove all Storage connections to this host02, and I rescan and restart this host02.

If this will not correct the problem, then or FC connections are not correct, or is a Hardware problem with this FC

LP

[ITVM]

Hi Marcelo,

After I try to remove this Storage from this host02, I forget that this holds the esxconsole from the upgraded 3.4 to 4.0.

So I have two options, move the exsconsole(VMware do not recommend this), or reinstall the VMware host.

So I move all the VMs and remove all Storage connectios to this host02, and reinstall VMware.

After this I reconnected the LUNs to this host, and all dead path are gone.

I try download files after a snapshot, and all is ok. Veeam can Backup all the VMs from this host without any problem.

I think this dead path was disrupting the connections between the host and the Storage. All is ok now

Marcelo, thank you for all the help on this.

LP