vSAN1

 View Only
Expand all | Collapse all

VMware ESXi 5.5.0 hanging while booting on services

  • 1.  VMware ESXi 5.5.0 hanging while booting on services

    Posted Apr 13, 2022 02:03 PM

    Hello,

    My ESXi 5.5.0 hung for more than 1 hour when booting. It hanged on "Starting up Services" "Running vsantraced start":

    1649857060131 (2).jpg

    When I press alt+F1 I see this:

    1649857060112.jpg

    When I navigate into the Alt+? I also see this :

    1649857060121 (2).jpg

     

    When the ESXi is started, all is ok but it hung for a long time always at the booting and I don't understand the logs.

    I tried to make a :

     

    services.sh restart

     

     and when it arrived to :

     

    Begin vsantraced stopped
    Begin vsantracedUrgen stopped
    Persisting traces to /locker/vsantraces

     

    It stopped for a while and all these messages comes on the ssh screen for a long time :

     

    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./▒': Input/output error
    rm: can't remove '/locker/vsantraces/.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$.': Directory not empty
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./.$./▒': Input/output error
    [...]
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./.$./.$./▒': Input/output error
    rm: can't remove '/locker/vsantraces/.$./.$./.$./.$./.$./.$.': Directory not empty
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./.$./▒': Input/output error
    rm: can't remove '/locker/vsantraces/.$./.$./.$./.$./.$.': Directory not empty
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./.$./▒': Input/output error
    rm: can't remove '/locker/vsantraces/.$./.$./.$./.$.': Directory not empty
    rm: can't stat '/locker/vsantraces/.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./.$./▒': Input/output error
    rm: can't remove '/locker/vsantraces/.$./.$./.$.': Directory not empty
    rm: can't stat '/locker/vsantraces/.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./.$./▒': Input/output error
    rm: can't remove '/locker/vsantraces/.$./.$.': Directory not empty
    rm: can't stat '/locker/vsantraces/.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./▒': Input/output error
    rm: can't stat '/locker/vsantraces/.$./▒': Input/output error
    rm: can't remove '/locker/vsantraces/.$.': Directory not empty
    rm: can't stat '/locker/vsantraces/▒': Input/output error
    rm: can't stat '/locker/vsantraces/▒': Input/output error
    rm: can't stat '/locker/vsantraces/▒': Input/output error
    rm: can't stat '/locker/vsantraces/▒': Input/output error
    rm: can't stat '/locker/vsantraces/▒': Input/output error
    rm: can't stat '/locker/vsantraces/▒': Input/output error
    rm: can't stat '/locker/vsantraces/▒': Input/output error
    rm: can't remove '/locker/vsantraces': Directory not empty
    mkdir: can't create directory '/locker/vsantraces': File exists
    Running vsantraced restart

     

    It's a pyramid-like, it reduces the "/.$" over time.

     

    Do you know what's going on? What can I do for troubleshooting this issue? And how to fix this long hanged at the boot?

    Can you help me, please?

     

    Thanks in advance...



  • 2.  RE: VMware ESXi 5.5.0 hanging while booting on services

    Posted Apr 27, 2022 02:01 PM

    Can someone guide me to discover or troubleshoot my issue, please?



  • 3.  RE: VMware ESXi 5.5.0 hanging while booting on services

    Posted Apr 27, 2022 02:22 PM

    I've reported your thread to moderators, they might move it to the vSAN area based on the information you've provided.

     



  • 4.  RE: VMware ESXi 5.5.0 hanging while booting on services

    Posted Apr 27, 2022 03:28 PM

    Do you use vSAN ?
    Are the NFS-connections still valid / reachable ?

    Do you think sabotage is an option ?
    Maybe someone poisoned the vsantraces directory so that it slows down the host ?
    Can you navigate with putty to

    /locker/vsantraces/

    and list files ?

    Ulli



  • 5.  RE: VMware ESXi 5.5.0 hanging while booting on services

    Posted Apr 28, 2022 02:59 PM

    Hello,

     

    I think I don't use vSAN, just use the simple datastore and don't have any cluster.

    I see one Storage was unreachable, so I delete it because it was old.

     

    I don't think there is sabotage here.

    But at 

    /locker/vsantraces/

     I see this with a simple "ls" :

    Actulog_0-1651157829783.png

    I'm aware that display is not normal/conventional.

     

    Regards.



  • 6.  RE: VMware ESXi 5.5.0 hanging while booting on services
    Best Answer

    Broadcom Employee
    Posted Apr 28, 2022 03:08 PM

    I think this still works on ESXi 5.5 but its been a long time since I used ESXi 5.5  

    connect to the ESXi 5.5 host and stop the VSAN trace service:
    /etc/init.d/vsantraced stop

    If you are NOT using VSAN, you can delete the folder/files  

    to permanently turn off the VSAN trace service issue the command

    chkconfig vsantraced off

    This should prevent it from restarting on bootup

    Hope this helps, 

     

     



  • 7.  RE: VMware ESXi 5.5.0 hanging while booting on services

    Posted Apr 28, 2022 03:27 PM

    Hello,

     

    For information, I reboot the Esxi after deleting the NFS storage unreachable, but it is still stuck at the vsantraced service.

    Thanks paudieo for the advice, I'll try it later when the ESXi was started.

    I hope the vSAN was not used, idk where to see if it is used or not.

     

    Regards.



  • 8.  RE: VMware ESXi 5.5.0 hanging while booting on services

    Broadcom Employee
    Posted Apr 28, 2022 03:36 PM

    vSAN is enabled on a vSphere Cluster from vCenter, and there will be a Datastore called vsanDatastore

    This KB may help to check if you have vCenter https://kb.vmware.com/s/article/2058322 

     



  • 9.  RE: VMware ESXi 5.5.0 hanging while booting on services

    Posted Apr 29, 2022 01:27 PM

    Hi, 

    I don't use vSAN also, thanks for the kb.

     

    I do the command

    /etc/init.d/vsantraced stop

    One hour later, the service shutdown (with the pyramid like: )

    rm: can't stat '/locker/vsantraces/.$./.$./.$./▒': Input/output error
    rm: can't remove '/locker/vsantraces/.$./.$./.$.': Directory not empty

     

    But after that, I disable the vsan service with

    chkconfig vsantraced off

     

    After this, I reboot the ESXi and it starts normally!!

    But when I ls in /locker/vsantraces/ I see this output again, strange...

    Actulog_0-1651238655107.png

    I hope the problem doesn't come back again.

     

    Thanks a lot, paudieo for the tips.

    Regards.



  • 10.  RE: VMware ESXi 5.5.0 hanging while booting on services

    Posted Apr 29, 2022 01:33 PM

    Can you go to

    /locker/vsantraces/.$./.$.

     with WinSCP ? - if yes try to delete the tree - this can take hours !

    Ulli

     



  • 11.  RE: VMware ESXi 5.5.0 hanging while booting on services

    Posted May 03, 2022 07:47 AM

    Hi,

     

    I try to delete it several times with WinSCP without success.

    After several hours/days the scp throw an error to delete the directories.

     

    I also try to delete it in ssh with

    rm -rf ./.$.

     

     But I got the pyramid-like the first post.

     

    I try to repair the vmfs with voma with

    voma -m vmfs -f check -d /vmfs/devices/disks/naa.xxxx:x

     

    But

    esxcli storage vmfs extent list

     

    return nothing and different vm1.x:x returns an error 

    ~ # voma -m vmfs -f check -d /vmfs/devices/disks/vml.0000000000766d68626133323a303a30:1
    Checking if device is actively used by other hosts
             ERROR: Failed to reserve device. Function not implemented
    Aborting VOMA

     

     

    I'm lost here because I think there is a problem because of these directories and the service stop of Vsan was just a patch and I want to resolve the issue correctly.

     

    Do you have any idea to delete these directories or repair the vmfs partition?

     

    Regards.



  • 12.  RE: VMware ESXi 5.5.0 hanging while booting on services

    Posted May 03, 2022 07:46 PM

    voma that comes with 5.5 is more or less a bluff - repair functionality was added years later.
    By the way - dont run voma against  vml.*. vml devices are just links .... so dont be surprised if a command working on vml* does not work.

    To delete a huge number of directories you may need to create for-loop.
    I am running some experiments on that right now ...

    Ulli

     



  • 13.  RE: VMware ESXi 5.5.0 hanging while booting on services

    Posted Apr 29, 2022 01:02 PM

    I/O errors on a VMFS-volume can slow down the system to a crawl.
    If you have important VMs on the same datastore I would suggest to copy them to another datastore.

    Ulli

     



  • 14.  RE: VMware ESXi 5.5.0 hanging while booting on services

    Posted May 16, 2022 10:06 PM

    Yeah time to upgrade