VMware vSphere

 View Only
Back to discussions
Expand all | Collapse all

.vmware_ha folder on non-heartbeat datastores

  • 1.  .vmware_ha folder on non-heartbeat datastores

    Posted Feb 08, 2013 02:34 PM

    I'm noticing the .vmware_ha folder structure being created on datastores that are not configured as preferred HA heartbeat datastores.

    Since this is a test environment I end up adding and removing datastores often. It appears every time I add a datastore it gets this folder. This in turn results in the error "vSphere HA agent on host 'x' failed to quiesce file activity on datastore" message every time I try to dismount the datastore.

    Is there a reason this folder is being created on non-perferred datastores? Is there a way to prevent this? What is the correct procedure for removing it so I can cleanly remove my datastore?

    Thanks!



  • 2.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted Feb 08, 2013 02:44 PM

    There is much more than the heartbeat datastores going on. So it is normal you will find it on every datastore in an HA enabled cluster.

    The folder contains lists for example which host has powered on which VM etc.

    Just take a look into it :smileyhappy:

    Regards,

    Mario



  • 3.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted Feb 08, 2013 02:46 PM

    Thanks vMario. I know what the folder is for. My question is, what is the correct way to get HA to not use this datastore so that I can dismount it without errors.



  • 4.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted Feb 08, 2013 02:45 PM

    Note: Discussion successfully moved from VMware ESXi 5 to Availability: HA & FT

    every time I try to dismount the datastore.

    What steps (or how) are you performing this?



  • 5.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted Feb 08, 2013 02:51 PM

    vmroyale, I see this error both when right-clicking on the datastore and selecting dismount in the vSphere client as well as using the SDK for some of my automation when calling host.getHostStorageSystem().unmountVmfsVolume(uuid).

    I've seen the recommended storage removal steps:

  • Unregister all objects from the datastore including VMs and Templates
  • Ensure that no 3rd party tools are accessing the datastore
  • Ensure that no vSphere features, such as Storage I/O Control or Storage DRS, are using the device
  • Detach the device from the ESX host; this will also initiate an unmount operation
  • Physically unpresent the LUN from the ESX host using the appropriate array tools
  • Rescan the SAN

    • I've also seen recommendations to check the cluster HA settings and remove the datastore from the preferred list. In this cause though, the cluster HA settings do say to only use preferred datastores for datastore heartbeating and the datastore in question is not in the list as one of the preferred datastores.



  • 6.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted Feb 08, 2013 03:54 PM

    The .vSphere-HA directory is used by HA for things other than datstore heartbeating, so its presence is expected on all datastores. This is normal. Check out depping 's blog entry http://www.yellow-bricks.com/2012/11/23/what-is-that-poweron-file-in-my-vsphere-ha-folder/ for one of the uses of this directory.

    http://kb.vmware.com/kb/1033634 has possible solutions for this error, but definitely follow the correct procedure for removing LUNs/storage devices you listed earlier as well.

    I guess you could always disable HA and then remove the datastores, even though that is more work...


  • 7.  RE: .vmware_ha folder on non-heartbeat datastores

    Broadcom Employee
    Posted Feb 10, 2013 12:29 PM

    HA uses this folder for various other reasons, for instance to track which virtual machines were powered on / off. So you cannot force HA not to use it. If this causes trouble you will need to disable HA before you take the unmount action.



  • 8.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted Feb 11, 2013 02:21 PM

    Thank you Duncan! Just to be sure I understand fully, the only way to get rid of the "failed to quiesce file activity on datastore" error message would be to disable HA completely, dismount the datastore, then reenable HA, correct?  Thanks!



  • 9.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted Feb 11, 2013 02:23 PM

    HA should quiesce IO to the datastore without being disabled so seems like there is a bug here. Appreciate if you could file an SR so we can take a look at the logs and figure out what went wrong in your environment.

    Thanks,

    Elisha



  • 10.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted Feb 11, 2013 07:24 PM

    We are getting the same message when trying to unmount a NFS export on a NAS that has failed. Hopefully there is a fix for this.

    Thanks.



  • 11.  RE: .vmware_ha folder on non-heartbeat datastores

    Broadcom Employee
    Posted Feb 18, 2013 09:13 AM

    CorporIT wrote:

    We are getting the same message when trying to unmount a NFS export on a NAS that has failed. Hopefully there is a fix for this.

    Thanks.

    As Elisha mentions above, please file a bug if you have errors like these. That way engineering can research it and potentially solve it. I mount/unmount datastores regularly and haven't hit this yet...



  • 12.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted Feb 26, 2013 12:02 AM

    I was having this same issue and I found the following post:

    http://adminotes.blogspot.com/2012/11/tips-vmware-cluster-remove-datastore.html

    I had been trying to remove the datastore via "Inventory->Datastores and Datastore clusters" (popped error for both hosts to which the datastore is attached) and also through "Inventory->Hosts and Clusters->Configuraiton->Storage" for the individual hosts.  I tried the unmounting from the "summary" tab per the article above, and it worked.  The interesting part was that I tried using the summary tab for the second host, and it failed with the error again. I then went to the Configuration->Storage page and tried removing it from there, and it worked.  Weird

    (Sorry for not logging a support incident, vmware)



  • 13.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted May 29, 2013 08:13 AM

    same problem here. vSphere 5.1 and... ...SRM.

    So my LUN replicated on the spare datacenter has .vSphere_HA folder with data from the primary datacenter.

    I don't find any configuration in HA to change things.

    as you can see in the following picture there's only two datastore selected.

    but all not selected but monted on the host of my cluster.

    Is this a bug ?



  • 14.  RE: .vmware_ha folder on non-heartbeat datastores

    Broadcom Employee
    Posted May 30, 2013 10:39 AM

    This is standard behavior by HA and as design. As mentioned above the activity should be quiesced, so you shouldn't have any problems.

    Can I ask you to file a bug with the exact issues you are experiencing and post the SR number here so I can have a look and point the right engineers at this problem?

    thanks,



  • 15.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted Jun 21, 2013 05:46 PM

    Hi Duncan,

    I have opened an SR for this as we have also seen this multiple times since upgrading to vCenter 5.1.  There are screen shots and VMware support bundles attached with this ticket.  Resolution to this issue will be much appreciated.  This issue is highly reproducible since uprgading from vCenter 5.0 -> vCenter 5.1 and ESXi 5.1U1 and 5.0U2.

    Thanks

    Support Request Confirmation Number: 13339664806

    Support Request Status: Open (Inbound message received)

    Date and Time Created: 2013-06-21 09:23

    Target Response Time: 2013-06-24 09:23 (GMT-08:00) Pacific Standard Time

    Configuration:

    Two ESXI hosts in a VMware cluster with HA enabled

    vCenter 5.1

    ESXi 5.1U1, 5.0U2

    Summary:

    Unmounting an inactive NFS datastore results in Vmware returning an error: ‘failed to quiesce file activity on datastore’

    • This behavior has been seen since upgrading to vCenter 5.1.
    • This behavior was not seen in the same configuration under vCenter 5.0

    While attempting to unmount an inactive datastore in the “Configutation -> storage” pane the following error is thrown.  Subsequent attempts can be successful,

    ERROR:

    Call "HostDatastoreSystem.RemoveDatastore" for object "datastoreSystem-76" on vCenter Server "VC-129-50-27.labx.simplivt.local" failed.

    Remove datastore

    ds1

    The vSphere HA agent on host '10.129.3.110' failed to quiesce file activity

    on datastore '/vmfs/volumes/c533f62f-7afa7af7'. To proceed with the

    operation to unmount or remove a datastore, ensure that the datastore is

    accessible, the host is reachable and its vSphere HA agent is running.

    LABX\Administrator

    6/21/2013 11:52:21 AM

    6/21/2013 11:52:21 AM

    6/21/2013 11:52:22 AM



  • 16.  RE: .vmware_ha folder on non-heartbeat datastores

    Broadcom Employee
    Posted Jun 24, 2013 07:13 AM

    I have tried to reproduce the issue by the way, so far not successful.



  • 17.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted Jun 24, 2013 12:05 PM

    Thanks, every time I have seen this behavior it is attempting unmount an NFS datastore that has just gone inactive under vCenter 5.1.  It seems to happen quite frequently and has been seen in multiple configurations.



  • 18.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted Jun 25, 2013 08:09 PM

    Updated SR ticket:  Known issue with related catch all KB article.

    I am surprised that this has been uncovered with more users adopting 5.1.  We will use "Reconfigure for vSphere HA" or perform multiple attempts when encountering this behavior.

    Thanks

    ** Please do not change the subject line of this email if you wish to respond. **

    Hello Darryl,

    As discussed over call this is a known behavior in vCenter Server 4.1.x, 5.0.x and 5.1.x.

    I am sending you the Kb article

    vSphere HA and FT Error Messages (1033634)

    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033634

    Since this is a known issue hopefully We are planning to get rid of this issue with the upcoming version of V center server we are still working on it.

    I am going to close the case on temporary note In case you need help please reply to this email within twenty one days from now and I will schedule another call for you.



  • 19.  RE: .vmware_ha folder on non-heartbeat datastores

    Broadcom Employee
    Posted Jun 26, 2013 07:10 AM

    I've have reported this to engineering, they will take a look at it ddio. May I suggest making a dump of your log files after experiencing the issue (support will request this soon I am guessing anyway)



  • 20.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted Jun 26, 2013 01:47 PM

    Thank you,

    It sounds like this SR will be closed as a known issue based on the KB referenced above.  I did provide screenshots along with system logs and uploaded them to the SR, I followed the outlined procedure for collecting diagnostic information needed for technical support.



  • 21.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted Jun 26, 2013 01:50 PM

    Bonjour,

    Je suis actuellement absent du bureau avec un accès limité à ma messagerie.

    Je serais de retour le 28/06/2013. En cas d'urgence merci de contacter François BLAES (francois.blaes@orange.com)

    Cordialement,

    Yannig BEYET



  • 22.  RE: .vmware_ha folder on non-heartbeat datastores

    Broadcom Employee
    Posted Jul 01, 2013 04:53 PM

    Regarding the problem of unmounting a datastore, there is a known issue in VC 5.1 that was fixed in VC 5.1U1. In 5.1, in certain circumstances, during the datastore unmount operation, VC is unable to log into the HA agent on the host on which the unmount is being done, and hence the HA agent's use of the datastore cannot be quiesced.

    You can determine if you are encountering this issue by searching the VC log files. Find the task corresponding to the unmount request, and see if the follow error message is logged during the task's execution:


    2012-09-28T11:24:08.707Z [7F7728EC5700 error 'DAS'] [VpxdDas::SetDatastoreDisabledForHACallback] Failed to disable datastore /vmfs/volumes/505dc9ea-2f199983-764a-001b7858bddc on host [vim.HostSystem:host-30,10.112.28.11]: N3Csi5Fault16NotAuthenticated9ExceptionE(csi.fault.NotAuthenticated)


    While we are on the subject, I'll also mention that there is another know issue in VC 5.0 that was fixed in VC5.0U1 (the fix is in VC 5.1 too). This issue related to unmounting a force mounted VMFS datastore. You can determine whether you are hitting this error by again checking the VC log files. If you see an error message such as the following with VC 5.0, then you may be hitting this problem. A work around, like above, is to disable HA while you unmount the datastore.

    2011-11-29T07:20:17.108-08:00 [04528 info 'Default' opID=19B77743-00000A40] [VpxLRO] -- ERROR task-396 -- host-384 -- vim.host.StorageSystem.unmountForceMountedVmfsVolume: vim.fault.PlatformConfigFault:



  • 23.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted Jul 03, 2013 01:49 PM

    Thank you,  I was contacted by support for the SR referenced above and they recommended VC 5.1U1 also.  I have successfully retested with 5.1U1a

    Performed ~ 40 iterations of unmount inactive datastore operations without hitting the error in question.

    vCenter Server Version 5.1.0 Build 1123961

    Client Version: 5.1.0 Build 1064113



  • 24.  RE: .vmware_ha folder on non-heartbeat datastores

    Broadcom Employee
    Posted Jul 04, 2013 07:00 PM

    ddio wrote:

    Thank you,  I was contacted by support for the SR referenced above and they recommended VC 5.1U1 also.  I have successfully retested with 5.1U1a

    Performed ~ 40 iterations of unmount inactive datastore operations without hitting the error in question.

    vCenter Server Version 5.1.0 Build 1123961

    Client Version: 5.1.0 Build 1064113

    Thanks for the feedback, just for your info... kfarkas is the lead engineer on vSphere HA... nice right to have passionate and customer caring engineers :smileyhappy:



  • 25.  RE: .vmware_ha folder on non-heartbeat datastores

    Posted Jul 09, 2013 12:36 PM

    It certainly is!  Thank you very much.