bootbank cannot be found at path /bootbank

[bvanarse]

Hi

In vCenter for one of the ESXi server I am getting  error as "bootbank cannot be found at path /bootbank"

Reboot is temporary solution for this.

Out of two devices there is none USB device at boot time

vmhba32 usb-storage link-n/a usb.vmhba32 () USB

And below is the error details :-

SCSI Level: 2 Is Pseudo: false Status: dead timeout

Is RDM Capable: false Is Removable: true

Can some one please suggest permanent solution if any.

Thanks &  Regards

[bspagna89]

Hi,

I've seen an issue where bootbank errors were showing up on hosts after upgrading to 6.0.. This was particular to HP and there's a KB article referencing it. Before we proceed, I think we need to know a little bit more about your environment..

1. What kind of servers/model?

2. Were there any changes done? upgrades/firmware/etc..?

3. Are you booting (or trying to) from a SD card or USB stick? Do you have a method of checking the health of that device? (HP and Dell both have a way to check in DRAC/iLO).

4. Were there any changes done to the BIOS of the server?

Let me know..

-Brian

[RajeevVCP4]

• If you accidentally mask a boot device, the /bootbank partition on your ESXi system might become unavailable. The /bootbank partition contains core hypervisor code and is critical for the functioning of the ESXi

Please provide out put for these commands.

• Use the esxcfg-scsidevs -a command to determine which adapter is used for the boot device.
• Run the following command to obtain a list of all claim rules for the boot device adapter. The following example uses vmhba1 adapter:
• esxcli corestorage claimrule list|grep vmhba1

[Dee006]

Hi,

Welcome to VMware Communities,

Are you using your USB drive to boot the ESXi host?If yes try to change the USB drive and enable/disable the USB after host reboot.

Reference Link:

Important information about the new ESXi 6.5 USB driver vmkusb, and the legacy USB drivers (2147650) | VMware KB

Unable to connect to host with vSphere Client or vCenter - My VMware blog

[BenLiebowitz]

I have to agree with bspagna89​, this sounds like the same issue we had with our HP Blades. 

Please confirm what hardware you're running on and what version of ESXi so we can help troubleshoot your issue. :smileyhappy:

[philanimk]

Hi BenLiebowitz​ and bspagna89​

I am seeing the same alarm in my environment.

Overview

We have 3x HPE BL 660c blades with an HPE 3Par (StorServ 7200) providing storage. Blades one and two of these are Gen 8, blade 3 is Gen9. We are running ESXi 5.5 in our datacenter. I ran esxcfg-scsidevs -a as suggested in this thread and below is the output from the three blades:

Blade1:

====

~ # esxcfg-scsidevs -a

vmhba0  lpfc              link-up   fc.50060b0000c26201:50060b0000c26200    (0:33:0.2) ServerEngines Corporation Emulex OneConnect OCe11100 10GbE, FCoE UCNA

vmhba1  lpfc              link-up   fc.50060b0000c26203:50060b0000c26202    (0:33:0.3) ServerEngines Corporation Emulex OneConnect OCe11100 10GbE, FCoE UCNA

~ # exit

====

Blade 2:

====

~ # esxcfg-scsidevs -a

vmhba0  lpfc              link-up   fc.50060b0000c26205:50060b0000c26204    (0:33:0.2) ServerEngines Corporation Emulex OneConnect OCe11100 10GbE, FCoE UCNA

vmhba1  lpfc              link-up   fc.50060b0000c26207:50060b0000c26206    (0:33:0.3) ServerEngines Corporation Emulex OneConnect OCe11100 10GbE, FCoE UCNA

vmhba2  hpsa              link-n/a  sas.5001438028499260                    (0:34:0.0) Hewlett-Packard Company Smart Array P220i

~ # exit

====

Blade 3:

====

~ # esxcfg-scsidevs -a

vmhba0  hpsa              link-n/a  sas.50014380339fd040                    (0:34:0.0) Hewlett-Packard Company Smart Array P220i

vmhba32 bnx2fc            link-up   fcoe.50060b0000c2623d:50060b0000c2623c  (0:33:0.2) Broadcom Corporation QLogic 57810 10 Gigabit Ethernet Adapter

vmhba33 bnx2fc            link-up   fcoe.50060b0000c2623f:50060b0000c2623e  (0:33:0.3) Broadcom Corporation QLogic 57810 10 Gigabit Ethernet Adapter

~ # exit

====

What changed

The StorServ is having additional hard drives installed and HPE recommended we upgrade the OS on the StorServ prior to this. One of the requirements of the OS upgrade was a change in Host Persona Configuration; previously this was set to "Generic". On the StorServ the following steps were carried out for Blades 1 and 2:

1. Under the Host edit menu, Unexport all virtual volumes
2. Remove the Paths from the host
3. Set the Host OS to "ESX 4.x/5/x"
4. Add the paths
5. Export virtual volumes to the hypervisor

Since the above changes were made, we are now seeing there are alarms from VMware:

====

[VMware vCenter - Alarm alarm.HostErrorAlarm] Issue detected on w.x.y.z in datacenter: Bootbank cannot be found at path '/bootbank' (2017-05-15T07:32:52.348Z cpu57:319614292)

====

Please let me know if you need more information and thank you for answering our questions.

/philani

[bspagna89]

Hi,

This sounds a lot like the issue we were running into with our HP G8/G9 Blades. Have a look at this KB below .. So far upgrading the iLO has resolved this issue for us...

HP Gen 8/9 lose access to device backing boot filesystem (2144283) | VMware KB

Let me know.

-Brian

[philanimk]

Hi bspagna89​,

Thanks for this. I'll go through everything and let you know of results once done.

/philani

[philanimk]

Hi bspagna89​,

I've upgraded to iLO 2.4 and I'm still getting notifications from VMware. Could it have to do with the installation mounts being volumes which are resident on the 3Par (not SD cards or memory sticks)?

[pbalderos]

Has anyone seen this issue on Dell power edge servers? I don't think its limited to HP as we are having the same issue. Any resolution?

[Kent Wicker]

Yes, I am having this issue on Dell M520, M620 and M630 blades.  This just seemed to start and has happened to about 6 systems.  I am running ESXi 5.5 and most of the 6 systems have all the firmware up to date.  One of the M630's is brand new but after a reboot has not done it again.

[planewalker]

December 28, at 4 am. One of BL460g8 with the installed version 3116895 "froze" with the same error. Firmware 2.40 Dec 02 2015

[TEST1981]

gotta same issue in here.

ILO 2.55

BIOS P89 v2.0
we have one other exact configuration blade in cluster on BIOS P89 v 2,52 and there is no issue.

[planewalker]

The similar found in a logs vcenter

Target: 10.0.5.96

Stateless event alarm

Alarm Definition:

([Event alarm expression: Host error] OR [Event alarm expression: Host warning])

Event details:

Issue detected on 10.0.5.96 in ALF-DATACENTER: Bootbank cannot be found at path '/bootbank'

(2017-12-27T19:03:01.257Z cpu24:206300420)

[lagasss]

Same problem here. Dell R430 running Esxi 6.0 Dell image on an SD card.

[SajjadAxia]

Same problem here as well on ESXI 6.5 with SD card boot on a HP DL120 G9... All the latest bios / fw have been applied.

[jiridvorak1]

Today during restart

"P[OST Error: 324-HPE Dual microSD Device Error - Both microSD cards have failed. Action: Replace the HPE Dual microSD device."

And system prevents to boot - HP DL360 G9

Seems this is known issue of HP internal USB of bad series which need to be replaced

Check

https://support.hpe.com/hpsc/doc/public/display?sp4ts.oid=7796454&docLocale=en_US&docId=emr_na-c05369827

[jiridvorak1]

So this is what need to be replaced on every HP DL9.....otherwise you are going to big issue (as me today)

[SajjadAxia]

How can you even check if you have that part without physically opening the server? iLO only says Embedded Flash/SD-CARD

[jiridvorak1]

to be honest,i think that opening can is only way to get SN to get patched from HP.According all information it seems,that all 001/002 are eligble for change.So to prevent issue i woudl recommend make it asap.

[darth_cisco]

Yep im running into the exact same issue now. I was getting error 30 when trying to remediate hosts and taking a look at the esxupdate.log I can see stuff like "HostImage: INFO: Installer <class 'vmware.esximage.Installer.BootBankInstaller.BootBankInstaller'> was not initiated - reason: bootbank is invalid: /bootbank is not a directory!" so taking a look then at the vmkernel.log I saw a lot of " Bootbank cannot be found at path '/bootbank'" I am still messing around trying to see if I can clear this up somehow, or if there is a way to see what is causing it.

Hosts:

Apple MacPro 6,1 64GB

ESXi 6.5U1 Build 5969303

[darth_cisco]

Running "vmkfstools -P /altbootbank" on a host that was failing to remediate and I get "Could not open /altbootbank Error: Unable to access device, please check your connection to the device." and all the ones that had no issues come up with:

vfat-0.04 (Raw Major Version: 0) file system spanning 1 partitions.

File system label (if any):

Mode: private

Capacity 261853184 (63929 file blocks * 4096), 107085824 (26144 blocks) avail, max supported file size 0

UUID: 60bb93e2-bae9cfea-08ce-a7ad0878d60f

Partitions spanned (on "disks"):

  t10.SanDisk00Ultra_Fit000000000000004C530001240407120381:5

Is Native Snapshot Capable: NO

[TEST1981]

Got same issue - all redirects and folders on these bootbank werenot accesible.

So I was not able even to upgrade/install VM tools,which were stored there.

Resolution were only to make shortcut to another folder on datastore like workaround or reinstall hypervisor on new sd card

[RainGigel]

Hi Everyone,

I have the same issue, I'm using:

BladeSystem c7000

ProLiant BL460c Gen9

Firmware - 2.54 Jun 15 2017

Event details:

Issue detected on 10.10.57.3 in Chassis-1: Bootbank cannot be found at path '/bootbank'

(2018-02-21T09:02:16.646Z cpu55:7946611)

I didn't do anything yet, I check the Blade and everything seems to work fine.

How worried should I be?

Thanks.

[TEST1981]

Can you check in sphere client if your SD card are live or dead ?

[GervandeKolk]

Hello there,

We are getting the same error in a ESXI 6.5 environment.

Server is a HP DL380 Gen9 with a 32 Gb SD card HP brand.

This morning our VM Terminal server decides to reboot suddenly.

I am going to check the hardware through iLo.

Regards,

Ger

[rdugaue]

It's been awhile since a post to this thread, but I can't find any further information. Has anyone found exactly what this error is about? I have several Dell R610s (Esxi 6.0 U2) and this just started on one dell about 3 months ago (Only reported ONCE) and then another reported last night.

Going into vSphere though shows the USB (8gb sandisks) as attached, while a previous post above showed it as "dead". So the drive is still attached. What further diags can I run to check this out. Is there an easy procedure to backup an existing config and restore it to an offline spare to have ready? Besides this error, there are no hardware problems reported in ilo or vsphere on any of my hosts.

[Misha1975]

I've got the same issue on some Dell R630 servers, after updating them to esxi 6.5 build 10884925.

Opened a case with VMware support, they've suggested to disable the vmkusb driver described in VMware Knowledge Base​ but this didn't solve the problem.

Going to contact them again and will reply when having more answers.