Solved! Unable to boot Esxi 6 Server after reboot, Need help

[Janmail]

Hi guys, i'm in desperate need of help.

Yesterday I restarted my vmware esxi 6 server, one of the vm's was acting strange, so I first shut down all vm's, then went to maintenance mode, then used reboot option to reboot the server.

Server didnot reboot, instead it locked itself. what I mean is that it became unresponsive on vshpere client, then I went to the console / monitor and was unable to login using F2.

then I rebooted it using power button.

After that it didnot boot again. I get an error at the boot :

/lpfc.v00: file not found.

Fatal error: 15 (Not found).

I tried restarting few times, but nothing changes.

This screen appears as soon bios is finished all checks, so I dont get any options / boot screens etc... this is first thing that comes on screen.

Please advise how to go from here. I'm out of options.

many many thanks in advance.

[a_p_]

Welcome to the Community,

I'm not sure, t it looks like an issue with the system/boot disk. Is ESXi installed on an USB device, or on HDD/SSD?

André

[Janmail]

Thank you Andre.

no USB, it's a basic HP server with scsi hdd raid 5 config.

installed all default way. nothing special. and hdd's are working fine, all green and synced.

also al basic tests are OK, memory test, hdd test etc...

No changes were made, no updates / upgrades or any bios changes. the only thing I did is reboot it, and when it didtno reboot by soft I did it by pressing the power button. that's all.

this is / was normal working for long time.

[a_p_]

Not sure what happened!?

Does ESXi present a page with a 5 sec countdown before the error occurs?

If yes, please press Shift-R to see whether it presents an option to revert to a previous version. It may be an option - other than trying to reinstall ESXi (with preserving the VMFS datastore) - to revert to make ESXi work again.

André

[Janmail]

No, i'm not getting any chance or any options / menu, 1-2 seconds after the bios checks that red lettered screen with error appears.

I saw indeed the option to reinstal it without losing the data store, but then I'll lose the settings of the host, I have to find a way to try to preserve / keep the settings other way I'll be restoring it for a long time :smileyhappy:

[b.w]

try to update with the installer-cd (customized or standard)

if the installer detects an valid esxi-installation it will offer you 3 options:

- install

- update and overwrite datastore

- update and preserve datastore

choose the option "update and preserve datastore" which will keep your settings and leaves the vms untouched. during update the bootloader will be repaired - that helped me years ago.

if the installer only gives you the choice to install then immediately stop because no valid harddrive was detected and your are going to destroy anything

[Janmail]

Going to try this now.

[Janmail]

No unfortunately this didnot work, the esxi starts from usb, then scans the discs and shows main datastore, if I press install or update, then it says unable to keep, only possible new install with deleting everything.

I have tried to start with ubuntu desktop. in demo mode, then it shows 2x partitions 262 mb, each. one of them older, looks like 5.5 esxi, and second is esxi 6, I dont even remember updateing it, looks like from last year.

in ubuntu it does not show the main datastore partition, only in gparted app it show as unknown, and is unable to mount it.

I did find the file that it's not able to find at boot, at the first (older 262mb ) partition, I copied it over to the new partition, and restarted the system, it went little further with boot, and now it's sayin missing : /scsi_be2.v00.  file not found.

i checked for this file in both partitions, they are  not there, not in older and not in newer. stuck here now.

I wander will I be able to get acces to the data store if I install new esxi on a usb stick, cause that's my last hope to save the vm's that are in the data store.

[continuum]

Can you use the ubuntu livecd to read the CD for your current ESXi version ?

If yes - you could inject the missing files by copying from CD to the damaged / incomplete bootbank.

Installing another ESXi to a USB-stick should work.

You can also try to copy out the VMs from Ubuntu when you install vmfs6-tools

[Janmail]

I have now installed esxci on usb and started, I can see the datastore, will be moving all vm's from there first to make sure nothing is lost.

then will try to move the missing files from usb to old esxi partition, maybe that will help starting it.

[Janmail]

Ok, so here's what happened.

After my last post I went on with ubuntu desktop demo mode, and checked all partitions and files.

1. i found out that the datastore was 100% OK, this was a relief.

2. I restarted the server and installed esxi on a usb. and started it up,

3. The datastore was recognized, so I started to backup all the vm directories to an external drive. Because I didnot want to risk it also getting corrupted. this took few days, cause few of vm's were quiet big.

4. while restarting the server suddenly I noted there is a new disk, in the boot list. sometimes it showed and sometimes it didnot. this is how I found out that there was an SD Card installed on the Motherboard, and the Esxi (the original crashed one) was installed on it.

5. I started the ubuntu desktop demo, went to the sd card, second partition, and found there a file called state.tgz  opened it and found out that this file contains all the information about the host, host settings, vm machines etc... including the license.

6. I copied it to a usb, then I went to check why this SD card didnot start properly, I tried putting there the files that were shown in the boot error. but that didnot help, because some of the files in the error didnot exist on sd card but also didnot exist in the usb with good working esxi, so after many many tryouts I left the idea of repairing it. and moved on to new SD card.

7. I tried for like 2 hours to install a new SD card with esxi on it, that didnot work, then I tried installing esxi from usb to Esxi, same problem, after installation starts, at 5% it says unable to write etc... While I know the sd card is 100%, it's new.

8. I removed the SD card. connected a usb with esxi on it and installed fresh esxi on it and booted the server. then after server booted and all was working, of course I was missing all the settings including Vlan / networks etc...

9. booted with Ubuntu again, copied the state.tgz file from backup sub to the working esxi usb. and started up. Host was completely restored to original condition. All settings, all Vm's all 100% working now.

so I guess then, the reason was that my originally installed SD card of 8GB from sandisk got corrupted, that's why the server was unable to start.

I also found out that there is a file state.tgz which contains all settings of the host and vm's. wich I didnot read anywhere, and didnot know before, using esxi's for more then 10 years :smileyhappy:

so for now this is solved.

thanks to everyone trying to help. I hope you guys never experiance this, it's a big stress... but we came out dry this time.