vSAN1

 View Only
  • 1.  Container volume disk consolidation

    Posted Jun 01, 2021 02:13 PM

    Hello,

    I have a problem with my container volumes, they are running from delta files, some of them have 16 delta stages and I have no idea how to consolidate these disks because I tried everything what I found on the internet. I tried, creating snapshot with delete all snapshots after that with and without quiescence checkbox. I tried migration from vsanDatastore to NFS datastore where I tried both options with snapshots but delta files are still there and nothing changed. The VM sometimes tell me warning about consolidation needed so I run it and it finished successfully but my disks are still in delta files and I still cannot change disk size. My datastore has a lot of free space. I was looking for some locks but the terminal tool tell me that there is no lock.

    Delta files was probably created by my Kasten backup tool which is now working without problems, I just need to consolidate old snapshots.

    Environment:

    Kubernetes 1.19 with CSI driver v2.2.0 on VMs in 15 HW version. ESX 7.0.1 16850804, vcenter 7.0.1 17005016. vSAN stretched cluster with 2 locations and 6 hosts.

    Which information I should share for better view to my issue?

    Thank you! 

     



  • 2.  RE: Container volume disk consolidation

    Posted Jun 02, 2021 09:01 AM

    But I found that there is a lot of .lck it is probably lock is`n it? But look at this output:

     



  • 3.  RE: Container volume disk consolidation

    Posted Jun 02, 2021 09:30 AM

    And I can see that the lck files is holding my k8s-node1 VM.

     



  • 4.  RE: Container volume disk consolidation

    Posted Jun 02, 2021 09:34 AM

    , That is normal.

    With vSAN Objects you have to actually query the .lck itself for locker/free info e.g.:

    or a loop:

     



  • 5.  RE: Container volume disk consolidation

    Posted Jun 02, 2021 09:50 AM

    Hi, ,

    Thank you! You are right, I can see that it is my esx2.brm host and it is k8s-node1 VM running on it. Do you have some idea how to free up these locks? Will be power off of VM enough or is there some other way without VM restart?

     

     

    Thank you!



  • 6.  RE: Container volume disk consolidation

    Posted Jun 02, 2021 07:39 PM

    , Again, it is completely normal for the host that a VM is running on is locking the vmdks - this is a basic protection mechanism.


    However, this doesn't mean that there isn't something else running on that host (e.g. backup-proxy VM) or physical backup server commands via the ESXi host also locking the files - one very basic check I do with such VMs is (if possible) migrate it to another host and see do some locks on some files still say the original host is locking them.


    Can you check if there is anything other than the expected files/handles accessing these in case they are still hot-added to backup proxy or some such (run on node esx2.brm or wherever it resides now):

     

    Powering off the VM is unlikely to stop whatever is locking it, if it is a physical backup appliance then rebooting the appliance may break the lock but sometimes (e.g. if there is some command previously issue by backup appliance (or host command resultant of this)) the host itself will need to be rebooted.



  • 7.  RE: Container volume disk consolidation

    Posted Jun 03, 2021 07:17 AM

    Ok, thanks, these lck files were only single clue which I had. I tried grep lsof output and did not find any other process which is holding my disks so I assume that my problem is not cause by locks. And it is returning me to beginning of my problem where I have no idea why is esx not consolidating my disks during Consolidating triggered by Consolidation option nor by Delete All Snapshots.

    The single option which I did not try is to shutdown cluster and do snapshots with deleting all snapshots. I am gonna try it.

     

    govc output of one of my broken disk:

     



  • 8.  RE: Container volume disk consolidation

    Posted Jun 04, 2021 06:15 AM

    I tried creating and deleting all snapshots from my vms when it was powered off and it did not help. Snapshots removing just ends well but delta files are still there. And I did not find any errors. 



  • 9.  RE: Container volume disk consolidation

    Posted Jun 04, 2021 08:36 AM

    I have no idea why is my consolidation process and all snapshots deleting process ending sucessfully but my delta files are still there without touch. I have no error about it but deltas are still same.