Remount boot filesystem on a running system.

[Kemperkipie]

Recently my ESXi 5.5 server showed an error:

Lost connectivity to the device mpx.vmhba32:C0:T0:L0 backing the boot filesystem /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0. As a result, host configuration changes will not be saved to persistent storage.

To give a little background information:

The server is a HP ProLiant ML350 G6 with 2 x Xeon E5504 and 72GB RAM. I have a P410i Raid controller with 8 disks (7 in RAID-5 with 1 spare) and a Areca ARC-1220 with 8 disks in RAID-5. De RAID-array on the P410i is used for Virtual Machines while the array on the Areca is used for bigger storage. I had ESXi 5.5 (downloaded an image from the HP official site - HP-ESXi-5.5.0-Update3-iso-550.9.3.26) installed on a USB-stick (Sandisk Extreme USB 3.0 16GB) placed in the internal USB-slot on the motherboard.

No one has been near the server at the moment the problem occured. So I'm sure it's not because the someone bumped the USB or something.

Of course I've been searching around to see if someone has the exact same problem or knows how to properly fix it. So far I haven't found something I could use. I do know now that once booted, ESXi is completely loaded into the RAM, so I could simply plug out the USB and inspect it. I have inspected the USB and found no problems when I put it in my desktop. To be sure I backed-up the entire USB using Paragon Backup and Recovery. After plugging it back in the server, the error keeps showing. When I start SSH and search the devices, "mpx.vmhba32:C0:T0:L0" does show, but I have no idea what I can do with it to make sure the current running configuration gets backed-up again to this USB and make sure that when the server will be restarted, it will load my configuration.

I hope someone can help or tell me what I'm doing wrong,

Thanks

[Davoud Teimouri]

Get backup from your ESXi configurations and install esxi on another USB drive and restore esxi configurations. Maybe current USB drive has problem.

[Kemperkipie]

When accessing the USB on my desktop, everything works fine. So I still find it strange.. However, even if it was the USB, how would I install ESXi on a new USB, without stopping the server. And when I back-up configuration (I'd have to google how, first) does this back-up everything, including drivers? I have one device which drivers has to be manually installed.

[Kemperkipie]

Eventually I had enough time to shut down all VM's and try some stuff out.

For the ones interested, this were the steps I took to restore my ESXi.

1. Set ESXi into maintainance mode
2. Download VMware vSphere PowerCLI
3. Run as administrator
4. Connect to ESXi (Connect-VIServer -server 0.0.0.0 -user user -pass pass) - where 0.0.0.0 = ip from ESXi, user = login for ESXi and pass = password for this login.
5. Back-up the current running-configuration (Get-VMHost 0.0.0.0 | Get-VMHostFirmware -BackupConfiguration -DestinationPath 'D:\temp\') - where 0.0.0.0 = ip from ESXi, 'D:\temp\' =  destination folder for the backup (be sure this folder exists on the PC you are executing it from)
6. Shutdown your ESXi host

Now at this point it turned out there was nothing wrong with my USB, for some reason ESXi didn't use it anymore until I rebooted the system, of course the configuration it loaded was outdated, so I restored the configuration to the back-up I just made. It could also be that your USB (or any boot drive for that matter) is defect and you'd have to completely reinstall ESXi. However, after a clean install you can also just restore the backup configuration. For restoring the backup I used the following steps:

1. Boot you ESXi host (make sure it is installed)
2. Connect to ESXi (Connect-VIServer -server 0.0.0.0 -user user -pass pass) - where 0.0.0.0 = ip from ESXi, user = login for ESXi and pass = password for this login.
3. Restore to the backup configuration (Get-VMHost 0.0.0.0 | Set-VMHostFirmware -Restore -Force -SourcePath 'D:\temp\configBundle-0.0.0.0.tgz') - where 0.0.0.0 = ip from ESXi, 'D:\temp\configBundle-0.0.0.0.tgz' =  the backed up configuration file (be sure this file exists on the PC you are executing it from)
4. Wait a few seconds, if it doesn't automatically reboot, do it manually
5. The configuration should now be loaded again, and the error of the missing boot drive should be gone.

I hope this helps someone else who has the same problem