VMware vSphere

 View Only
  • 1.  Failed to cleanup registration key on volume

    Posted Jun 08, 2023 12:36 PM

    Hi guys,

    We have volumes that recently went unmount from some of ours hosts for unknown reasons. No configuration have been changed in the mean time.

    If I connect the Vcenter and go to datastore / configure / Connectivity and Multipathing, I can see that:

    (Mount option fails)

    NLdenkavit_0-1686227195631.png

    If I get a look into /var/run/log/vmkernel.log, I can see that at the moment I try to mount the volume, these lines appear:

    2023-06-08T12:31:02.479Z Wa(180) vmkwarning: cpu5:2097880)WARNING: HBX: 2720: Failed to cleanup registration key on volume645cee36-XXX-425d-84160c4b53ea: Failure
    2023-06-08T12:31:02.479Z Wa(180) vmkwarning: cpu5:2097880)WARNING: Vol3: 4323: Error closing the volume: . Eviction fails: Failure
    2023-06-08T12:31:56.770Z In(182) vmkernel: cpu22:2098351)NMP: nmp_ThrottleLogForDevice:3864: Cmd 0x89 (0x45b982cbf400, 2099889) to dev "naa.60002ac000000000000000040002a2cf" on path "vmhba4:C0:T0:L0" Failed:
    2023-06-08T12:31:56.770Z In(182) vmkernel: cpu22:2098351)NMP: nmp_ThrottleLogForDevice:3869: H:0x0 D:0x2 P:0x0 Valid sense data: 0x7 0x27 0xc7. Act:NONE. cmdId.initiator=0x430a096fda80 CmdSN 0x4
    2023-06-08T12:31:56.770Z In(182) vmkernel: cpu22:2098351)ScsiDeviceIO: 4605: Cmd(0x45b982cbf400) 0x89, CmdSN 0x4 from world 2099889 to dev "naa.60002ac000000000000000040002a2cf" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x7 0x27 0xc7
    2023-06-08T12:31:56.770Z In(182) vmkernel: cpu34:2099889 opID=3ed8e991)World: 12231: VC opID lhyvklur-272849-auto-5uj7-h5:70103817-e5-ce-b9fa maps to vmkernel opID 3ed8e991
    2023-06-08T12:31:56.770Z In(182) vmkernel: cpu34:2099889 opID=3ed8e991)LVM: 7015: Forcing APD unregistration of devID 646c9b55-XXXX-cc7e-84160c4ba8a4 in state 1.
    2023-06-08T12:31:56.770Z In(182) vmkernel: cpu34:2099889 opID=3ed8e991)LVM: 16106: Failed to open device naa.60002ac000000000000000040002a2cf:1 : Read only
    2023-06-08T12:32:02.476Z Wa(180) vmkwarning: cpu5:2097880)WARNING: HBX: 2720: Failed to cleanup registration key on volume645cee36-XXX-425d-84160c4b53ea: Failure
    2023-06-08T12:32:02.476Z Wa(180) vmkwarning: cpu5:2097880)WARNING: Vol3: 4323: Error closing the volume: . Eviction fails: Failure

    Do you have any idea how to fix it?



  • 2.  RE: Failed to cleanup registration key on volume

    Posted Jul 17, 2023 02:39 PM

    Is this on a Dell Server? I am having a similar issue on the RAID 1 BOSS card with a Dell R760.
    My happens about every minute.

    Now the volume is not unmounted and it never gets unmounted  but it still shows an error like yours.

    2023-07-17 09_35_49-esx7.illinoiseyecenter.com - PuTTY.png

     

     



  • 3.  RE: Failed to cleanup registration key on volume

    Posted Jul 20, 2023 08:06 PM

    Hi,

    as per SCSI return code D 0x2 and the sense code 0x7 (Data Protect) and advanced sense code 0x27 0xc7 that device seems to be write protected.

    Unfortunetaly the advanced sense code isn't fully listed here, so you should ask the storage vendor what it exactly means.

    If I should bet it seems be related to SPACE ALLOCATION FAILED WRITE PROTECT.



  • 4.  RE: Failed to cleanup registration key on volume

    Posted Jul 24, 2023 08:29 PM

    Hello  ,

    The SCSI sense code means that the LUNs have entered into Data Protect mode. This means that the LUNs are in Read Only state. The SCSI response did not give the exact reason for that but it is definately an issue from SAN side which is not allowing any WRITE operations on the LUN.

    Please involve the SAN Vendor and check that why the WRITES are not allowed.

     



  • 5.  RE: Failed to cleanup registration key on volume

    Posted Jul 26, 2023 12:38 PM

    This is not on a SAN. The error is coming from the local NVMe PE8010 disk on a Dell PowerEdge with BOSS-N1 storage.



  • 6.  RE: Failed to cleanup registration key on volume

    Posted Jul 28, 2023 04:48 PM

    Hello  ,

    If these errors are coming from local disk then perform the hardware diagnostics on the local disks. For HPE (use iLO) for Dell use (iDRAC) to perform the extensive diagnostics on the hardware and you will get the result of issue.

    The issue can range from faulty disks, faulty controllers to a simple Driver Firmware upgrade on controllers. 



  • 7.  RE: Failed to cleanup registration key on volume

    Posted Aug 09, 2023 08:24 PM

    This issue has been fixed on my Dell R760 with BOSS-N1 Monolithic with RAID1 NVMe drives.
    So the issue was when ESXi gets installed on a drive greater than 142GB it will automatically create a VMFS datastore.

    ESXi System Storage Changes | VMware

    Once I deleted the VMFS volume on the BOSS storage the error stopped. Since my servers are not in production I worked on a way to install ESX without creating the VMFS volume. 
    I setup a simple kickstart configuration (ks.cfg) to stop the creation of the VMFS.
    vmaccepteula
    install --firstdisk=NVMe,local --novmfsondisk
    rootpw myp@ssw0rd
    reboot

    Hope this helps someone else if they have the same problem.