ESXi lost configuration on reboot

[wolfwolf]

Hi,

Recently we've installed ESXi 4.1 Update 1 embedded version on the internal SD card (SanDisk 2GB) for a few Dell PowerEdge M610 blade servers using the ESXi recovery CD from Dell, also available for download from the VMware site:

http://downloads.vmware.com/d/details/dell_esxi_recoverycd/ZHcqYip3ZWJkKmV3

Yesterday we rebooted the servers to make sure everything would come back up.

To our surprise when ESXi come back up on each server, the whole configuration was gone: IP address, server name, password, everything was lost, the configuration was back as if we just installed ESXi.

Any ideas why that happened?

We did a proper reboot for ESXi: one of the servers was rebooted using vCenter, the others were rebooted using the ESXi DCUI using the F11 key.

Before the reboot, one of the ESXi server was up with its configuration for a few days and was managed through vCenter, the others were installed an hour or so before the reboot and weren't connected to vCenter yet: no matter how long they were up, all of them lost the configuration after the reboot.

And we've been unable to reproduce the issue so far, we've reconfigured ESXi on a couple of those servers today and rebooted them from the ESX DCUI: each of them came back online just fine this time, the configuration is still there.

We didn't reconfigure all of them, let me know if there is anything we can check on those still untouched after the reboot to tell how come they lost the configuration.

Thanks.

[krishnaprasad]

Hello,

   I do see a Dell document which talks about the compatibility of USB keys supported on Dell Power Edge Servers. ttp://support.dell.com/support/edocs/software/eslvmwre/VS/docs/compat/Com_Mat.pdf

As per this document, the SanDisk 2GB card is not supported on an M610 system. Worth trying with a Kingston 2GB / 1GB SD card? 

This doesnt explain why the failure is seen. But just a pointer if you are using an unsupported configuration.

Might be useful if the logs can be captured at the time of system reboot when the failure is occurred.

Thanks,

Krishnaprasad

[wolfwolf]

krishnaprasad wrote:

   I do see a Dell document which talks about the compatibility of USB keys supported on Dell Power Edge Servers. ttp://support.dell.com/support/edocs/software/eslvmwre/VS/docs/compat/Com_Mat.pdf

As per this document, the SanDisk 2GB card is not supported on an M610 system. Worth trying with a Kingston 2GB / 1GB SD card? 

This doesnt explain why the failure is seen. But just a pointer if you are using an unsupported configuration.

Might be useful if the logs can be captured at the time of system reboot when the failure is occurred.

We've already called Dell and for some reason they couldn't find the Kingston 2GB SD card matching the part number (738M1) from that PDF document, anyway we've ordered from them what appears to be the same item, just with a different part number, we should be getting it soon.

Meanwhile, any ideas on what we could do to further investigate the issue, in case it's not related to the SD card compatibility?

We still have access to the systems that lost the ESXi configuration on reboot.

Thanks.

[krishnaprasad]

it looks like the /bootbank was not updated in your faulty systems. /bootbank contains all the .tgz files where the state of the ESXi is stored.  

There is a script in ESXi called backup.sh which takes care of updating the state of the ESXi ( like network re-configuration if any etc ) in /bootbank. The backup.sh normally runs in every hour i think. This should have ideally taken care of updating required configuration files modified as part of updating netwrk info, password etc ( User configurable settings in ESXi ). But In your case, it looks like the backup script may have failed to update the state of the SD card with the new configurations. can you run backup.sh manually on those faulty system and see the time it takes with the sandisk SD card?

So replacing with the supported SD model will be ideal and see if you can reproduce the issue?

Thanks,

Krishnaprasad

[wolfwolf]

On every system we've replaced the SanDisk 2GB SD card with a Kingston 2GB SD card ordered from Dell, that should be supported.

And we can still reproduce the issue on all systems.

It appears that the configuration is only lost on the first reboot: after the first reboot, if you reconfigure the system and reboot it again, the configuration will stick just fine. But we're still not very confident putting these systems into production, until we figure out what's causing the problem.

Also, the issue appears to only affect Dell ESXi ISO, we cannot reproduce it with VMware ESXi ISO (but we need the Dell version, since the VMware version doesn't include the drivers for the Intel 10Gbit NIC).

Any ideas on what to try next to figure out the issue?

Thanks.

[krishnaprasad]

Interesting. can you check if running the command /sbin/auto-backup.sh solves the problem?

i.e. right after first boot, do the necessary configuration changes like password set, change IP etc. Then before reboot, run the above command and see if it makes any difference? also if you dont mind, post the output of the command from your system.

http://deinoscloud.wordpress.com/2010/02/17/esxi-automatic-backups/

[antonvn]

Colleagues,

We hit exactly the same issue with M610, ESXi4U1 (Dell patched) and 2Gb SanDisk cards last week.

Depending on luck, afer 1-3 configuration attempts followed by a reboot, system "memorizes" its settings (IP, password etc). We tried to push configuration save with /sbin/auto-backup.sh 0 /bootbank/ (run few times, then sync; sync; reboot). What became clear is sometimes /bootbank/ gets re-initialized, and we see /altbootbank/ partition as well. Also we tried to install ESXi onto a system 20Gb SATA SDD drive, with the same result.

Once settings are "remembered" after a reboot, all consecutive reboots did not flush the configuration.

We still not happy to put affected systems to production.

Anton.

[Dave_Mishchenko]

So you're following this process: 1) boot Dell ISO 2) install 3) reboot 4) configure esxi 5) reboot 6) loss config changes As suggested I would manually run the backup before step 4 or 5 and also check that bootbank is properly mounted at that point.

[wolfwolf]

Dave Mishchenko wrote:

So you're following this process: 1) boot Dell ISO 2) install 3) reboot 4) configure esxi 5) reboot 6) loss config changes As suggested I would manually run the backup before step 4 or 5 and also check that bootbank is properly mounted at that point.

Yes, that's exactly the process we followed:

1) Boot Dell ESXi installable edition ISO

2) Install ESXi

3) When ESXi installation is completed, press enter to reboot

4) Configure ESXi: network, etc.

5) Reboot from ESXi DCUI with F12 key (let's call this first reboot from now on)

6) ESXi comes back up and all configuration appears to be lost

As per your suggestion, we also tried to run the /sbin/auto-backup.sh script between step 4 and step 5, it worked fine, but it didn't help with the problem.

Also, if after step 6, we reconfigure ESXi again and reboot, then the configuration stays in place: the issue appears to happen just for the first reboot (after the first time ESXi is booted on the system).

We've been able to reproduce the issue even by installing the Dell ESXi installable edition in a VM (on VMware Fusion).

If you want to give it a try, here is the Dell ESXi installable edition ISO:

http://support.dell.com/support/downloads/download.aspx?c=us&cs=555&l=en&s=biz&releaseid=R297945&SystemID=PWE_M610&servicetag=&os=EX41&osl=en&deviceid=25794&devlib=0&typecnt=0&vercnt=2&catid=-1&impid=-1&formatcnt=0&libid=47&typeid=-1&dateid=-1&formatid=-1&source=-1&fileid=447309

http://ftp.us.dell.com/esg%20solutions/VMware-VMvisor-Installer-4.1.0.update1-348481.x86_64-Dell_Customized_A01.iso

The issue doesn't happen with the VMware ESXi installable edition ISO, there is definitely something wrong with the Dell ISO.

We did some comparison between between ESXi from Dell installed on a VM and ESXi from VMware installed on another VM and we found some differences:

- On VMware ESXi, the initial /bootbank and /altbootbank partition remain the same after first reboot. On Dell ESXi, the initial /bootbank partition becomes the /altbootbank partition and the /altbootbank partition becomes the /bootbank partition after the first reboot (we verified this by creating a test file with a different name in each partition before the reboot).

- When ESXi is started for the first time after the installation, on VMware ESXi the /altbootbank partition is almost empty, it only contains the boot.cfg file, on Dell ESXi the /altbootbank partition appears to have all the files also in the to the /bootbank partition (even if running a recursive diff shows differences between two file sets)

- On Dell ESXi, when ESXi is started for the first time, it appears that the modified date of files in the /altbootbank partition are more recent than the files in the /bootbank partition, except the for the /bootbank/local.tgz file, that's also the file which gets updated if we manually run the /sbin/auto-backup.sh script, like if the /altbootbank partition was actually the active partition, except for the config file. Also, in the process above, if after step 5, we press Shift+R when ESXi is loading, ESXi comes back online with the correct configuration!

Any ideas on what's wrong with Dell ISO? How come the partitions gets switches at the first reboot?

How does ESXi pick up the partition that becomes active?

Thanks.

[DSTAVERT]

See what happens with a recovery boot. Try Shift + R when the loading Hypervisor screen appears. That should give you the ability to go to the previous update/version.

[wolfwolf]

David Stavert wrote:

See what happens with a recovery boot. Try Shift + R when the loading Hypervisor screen appears. That should give you the ability to go to the previous update/version.

As per my previous message, in the process above, if after step 5, we press Shift+R when ESXi is loading, ESXi comes back online with the correct configuration!

I'm still not sure what's broken with Dell ISO and if's safe to assume that the issue only affects the first reboot.

Thanks.

[DSTAVERT]

Missed that part. I woiuld go through and make configuration changes and restart to see whether you have a viable install.

[Dave_Mishchenko]

Boot.cfg is read in both Hypervisor1 and Hypervisor2 to determine which has the variable updated = 2.  That get's mounted as bootbank.  Boot.cfg in altbookbank has a value of update = 1. It may be that the Dell image includes some scripts, etc that they just want run once and that their cleanup process to deal with those scripts is not properly dealing with the bootbanks.

[DSTAVERT]

Just out of curiosity did you do an MD5 checksum on the downloaded iso?

[wolfwolf]

Yes, the MD5 sum is correct.

And we can reproduce the issue installing Dell ESXi on our M610 servers and also in a VM on VMware Fusion.

[DSTAVERT]

I can also confirm that same behavior using your Dell download link. Shift + R does boot the correct bootbank and a subsequent reboot does come up in a configured state.

Does this server have the dual SD card slots?

[Dave_Mishchenko]

oem.tgz on Hypervisor1 contains a tweak for MD36xxi support in the form of a VIB which means that /altbootbank being set as the active boot partition for the next reboot is correct.  They should have included a reboot command in  the customization. Since the host doesn't reboot  your changes get written to /bootbank either when you reboot or when the auto backup process runs (1 min past the hour).

[krishnaprasad]

That's exactly the case Dave.

In this case, 'updated' variable in /bootbank becomes 1 and same in /altbootbank becomes '2'. hence in the next bootup the bootup will uncompress files from /altbootbank which dont carry the changes like password change, IP changes etc.

As you mentioned, Dell carries an init script for configuring their MD36xxi array and during clean up of the VIB ( which contains the script ) via esxupdate, this scenario happens. 

I also see an esxupdate command, 'esxupdate clearpending' executed in a VMware script 130.-x.x... Does this removes the locks or sync the VIB database in SYNC?

So other than rebooting the server to make the changes effective ( removal of the VIB ), is there any other way to remove the VIB from the init script but still the updated variable can be kept as '1' for altbootbank and '2' for bootbank? 

BTW, wolfwolf , you can Call Dell tech support so that they can change the ISO images.

[wolfwolf]

Dave Mishchenko wrote:

oem.tgz on Hypervisor1 contains a tweak for MD36xxi support in the form of a VIB which means that /altbootbank being set as the active boot partition for the next reboot is correct.  They should have included a reboot command in  the customization. Since the host doesn't reboot  your changes get written to /bootbank either when you reboot or when the auto backup process runs (1 min past the hour).

Thanks for the explanation.

At this point, is it safe to go with the Dell ISO (doing a manual first reboot, after the initial boot, before applying any configuration) or is it better to go with the stock VMware ISO to avoid any risks of losing the configuration at future ESXi reboots/updates?

Besides the Intel 10G NIC driver missing from the VMware ISO, what are the advantages of the Dell ISO?

It looks like you can download the Intel 10G NIC driver from VMware site and apply it to the stock ESXi installation from VMware.

If you do so, will the driver be carried over when applying future ESXi updates to the host?

About the Dell ISO bug, what would be the best way to report it to Dell? Any specific email address?

Thanks.

[krishnaprasad]

I think it's okay to still use the Dell ISO image with an additional reboot after first bootup. The configuration loss will happen only on for the first boot.

To know what is extra in Dell Customized images, i think you can refer Dell support website. They have documented what's additional available in Dell Customized image. This is how i could see the information. Might be helpful to you.

http://support.dell.com/support/index.aspx?c=us&l=en&s=biz  --> Select  “Drivers & Downloads” section --> Choose the System model that you have by going to “Select Model” tab  --> Select “Servers, Storage, networking” tab from ‘Select your product family’ section --> Click on ‘PowerEdge Server’ --> Select your product Model ( Eg/- R910 ) --> Confirm

See the attached screenshots where i could see the info.

For fixing the issue in Dell ISO, You can call Dell Tech support . you may hve got tech support numbers / mail id when you purchased servers from them?

[krishnaprasad]

forgot to mention that "Additional Information" section carries the information on what's extra in Dell Customized images. Refer the screenshot ( attached ) as well for 4.1 Update1

[Dave_Mishchenko]

I'd go with the dell iso as well.   The issue you've had with the ISO will be a one time thing.

[wolfwolf]

It looks like the only driver from the Dell ISO missing from the VMware ISO is the Intel 10G NIC ixgbe driver and you can download version 3.x of in from the VMware site, while the Dell ISO contains older version 2.x. Not sure what are the practical advantages of the newer version from VMware site, if any.

Is it still better to go with the Dell ISO?

In case, we decide to go with the VMware ISO and add the ixgbe driver to it using the vihostupdate utility, will the driver remain in place when we apply future ESXi updates to the hosts?

Thanks.

[krishnaprasad]

I dont think it's just one driver update in the Dell image. if you see the screenshots that i had attached, the 'additional information' section talks about different driver updates/inclusions. For example  bnx2 driver version available in Dell Customized image is later version compared to VMware ISO.  and 'qlge' driver included in the image is not part of VMware image by default. Need to really see what happens on an upgrade.

[wolfwolf]

Yes, the Dell ISO contains other customizations, but since we're only going to use the 10G Intel NIC with ESXi, the other drivers shouldn't matter.

And for the 10G Intel NIC the driver available as a separate download on the VMware site (version 3.x) is actually newer than the one available from the Dell ISO (version 2.x).

But I'm still not sure installing that driver manually on a stock VMware installation is a good choice, e.g. if future ESXi updates will retain it.

And I'm not sure how to test that either, since there are no updates available for ESXi 4.1 Update 1 yet.

Anyone tested this in the past?

Thanks.

[DSTAVERT]

I would go with the Dell ISO. A later driver version may not be the most appropriate version or the tested and supported version from Dell's persective. Before considering using the VMware downloaded driver I would check with Dell support. I would also suspect that the Dell version of the ISO would include Dell specific CIM providers.

[krishnaprasad]

The OM providers are not included in Dell Customized image. it can be separately downloadable and installed on the ESX/ESXi images.  You can see this information from the 'additional information' section screenshot that i had poster earlier ( From Dell Support website ) .

I think you can even use the drivers posted @ VMware website since it might be tested by VMware or the driver vendor before posting it in VMware website. I second Dave and DSTAVERT for continued usage of Dell ISO since the issue will be present only on the first bootup.

[krishnaprasad]

This issue can be even reproducible with VMware ISO.

1. Let's say We have VMware ESXi 4.1 Update1 ISO installed on a system.

2. Boot into the USB key/LUN where the image is installed.

3. Make some changes to the configuration ( Eg/- Change the password of ESXi host )

4. Install an offline bundle/VIB  ( eg/- any Driver package available @ VMware.com ) using vihostupdate/esxupdate command sets.

5. Make the configuration changes again in the system ( eg/- Create a vSwitch, Register a Virtual Machine to the host etc )

6. Reboot the server

7. Boot into ESXi.

You can see that the changes made like vSwitch creation, Registering VM to the host are lost where as password change is persistent.

i.e. whatever changes made AFTER execution of esxupdate command is found be to lost.

[krishnaprasad]

The reason of the failure is because when a package is installed/uninstalled on ESXi, the changes are made to the alternate bootbank ( where updated=2 in boot.cfg so that this becomes /bootbank in next bootup ). The changes made between the updates (package install/Uinstall) and reboot are written into /bootbank only. So it is not available during next bootup.

[krishnaprasad]

it looks like Dell released the updated ISO image which carries the fix for the issue discussed in this thread. See the screenshot attached.

[devilz666]

This is seriously %$$^$^^'ed up

Who do we send the bill to at Dell for the time lost in this stuffup?

Take a few vmware admins per hour charges and time lost and im sure this will cost them a wee bit

I had this server fully configed, joined to luns/sans/multipathed/networking etc etc

Then a nice reboot a few hours later all gone! Later i found this stretch of comments on the problem...

If they are going to release preconfiged mission critical software can they at least do some testing of it first?

Dont vmware have any say in testing of their software thats been modified and released?

Imagine the hassel if a poweroff/on wasnt tried until a later date! (of course you always do a poweron/off test but im sure most admins wouldnt imagine the entire config gone on a single reboot!)

[krishnaprasad]

You can see the problem in VMware native image as well. Here are the steps.

1. Install VMware ESXi 4.1 Update1 image

2. Boot into ESXi for the first time.

3. Change a configuration setting ( let's say set password )

4. Install a package in ESXi ( for example an async driver package available in VMware website )

5. After successful installation of the package, set some other configuration ( eg/- Set the IP to a static 192.168.1.5 IP )

6. Reboot the server

After reboot, you can see that the IP is set to the old one where as password changes is retained.

This is actually a design w.r.t VMware ESXi. i.e. when ever there is a package gets installed, it updates altbootbank. So what ever configuration changes that you do after installing a package and before reboot WILL BE WRITTEN ONLY to /bootbank but NOT to /altbootbank... The second boot will take /altbootbank as the current bootbank and hence you see that the configuration is lost.

But actually the configuration is not LOST. it's still there in /bootbank.

This scenario came across in Dell Customized ISO images since they have a package installed during first boot to support their storage array. and hence you see a failure in Dell image alone.

Hope it's clear.

Thanks,

Krishnaprasad

[jjgunn]

Great. So until this is fixed, I've been going to console, configuring networking, password, etc. rebooting, (lose everything) then set it up again, reboot again and it's ok.

Thanks I think I got the installation down pat now. :smileylaugh:

[jbarbee]

I just got this error on my install on a Dell 2950 server, using the Dell ISO of ESXi v4.1.  I assumed it had been rebooted before now (it is a new installation), as I have been working on this system over a week now.  But perhaps I didn't.  When I did, I too came up with the blank configuration. 

So using the "Shift-R" at the Hypervisor boot screen, it apparently overwrote the latest "blank" config, with the only other configurations I had located in "Hypervisor2".  This restored everything back to pre-reboot status.

I did a quick list of the any "state.tgz" in /vmfs/volumes/Hypervisor?/ and only 1 and 2 are updated.  1 is newer than 2, and 2 is the pre-reboot config.

Now that I need to know I need to reboot after initial config after a fresh installation I will, but for those, like me that got bit, the "Shift-R" could help you out as it did me.

Thanks for the thread.

[vonsch]

Has anyone validated that this issue does not persist after the first time that the server is rebooted?

[Dave_Mishchenko]

When I tested this with the original Dell ISO image it was a one time thing.

Dave
VMware Communities User Moderator

Free ESXi Essentials training / eBook offer

Now available - VMware ESXi: Planning, Implementation, and Security

Also available - vSphere Quick Start Guide

[jjgunn]

The latest Dell ISO (Version A03) I have didn't experience the one time lost config bug :smileyhappy:

http://support.dell.com/support/downloads/download.aspx?c=us&cs=555&l=en&s=biz&releaseid=R301608&SystemID=PWE_R710&servicetag=&os=EX41&osl=en&deviceid=25794&devlib=0&typecnt=0&vercnt=5&catid=-1&impid=-1&formatcnt=0&libid=47&typeid=-1&dateid=-1&formatid=-1&source=-1&fileid=453777

Noticed this is version A04. I'll assume A04 is working since the A03 version I'm using no longer has the bug.

Thanks for fixing this Dell.

[SDBinSF]

ESXi 4.1u1 (Dell customized ISO) lost configuration on reboot.

I experienced something very much like this, but using much simpler components:

I downloaded the Dell customized ESXi 4.1u1  VMware-VMvisor-Installer-4.1.0.update1-348481.x86_64-Dell_Customized.iso

I downloaded that iso image on April 29th from the VMware site.  (After the update from Dell?)

I installed ESXi on a Dell 2950, configured the basics, including the free license, and proceeded to install a CentOS 5.6 server, with VMware Tools etc.  A colleague and I put about 3 days worth of effort into this.  Then I shut down the CentOS server; put the host into maintenance mode, shut it down, unplugged it and moved it to a different room.  When I booted the host (physical server) in its new location, it came up with no IP address (0.0.0.0), no root password, and no virtual servers in its inventory.  As another user said: that is seriously #!$@d up. 

Luckily, no reformatting had been done and the server image was still on the disk, so all we had to do was add the virtual server image back into the inventory.  (In the host configuration tab, click storage in the hardware navigation pane, and then right click the storage device to "browse datastore" and add to inventory.)  Our datacenter manager bailed me out of this situation, though he had never seen or heard of this problem. 

My situation had nothing to do with USB cloning or anything else fancy; just simple installation from a DVD, direct attached storage, and plain vanilla configuration.

Having done nothing but add the lost configurations, the machine can now reboot and keep its configuration.   In 10 plus years of light sysadmin work (not my main job), I have never seen anything like this.

[DSTAVERT]

Sorry your introduction had a rocky start.

As for not seeing anything like this: the world of technology is full of similar tales. Stick around long enough and you will have much to laugh or cry about.

[robfish]

My problem is slightly different but I have not found a better thread.

I am new to ESXi. I have inherited a Dell server set up by a colleague who is no longer with us.

After rebooting it seems to revert to the state before my changes (password and new VM's)

(i.e. the changes made previously by my colleague are there so it it not the first reboot problem experienced by others.)

I can SSH to the machine (I tried connecting a screen and keyboard but the keyboard did not seem to work)

and I think the problem may be something to do with mounting the bootbank partitions.

~ # ls -l | grep vmfs
lrwxrwxrwx    1 root     root                 49 Sep 27 05:15 altbootbank -> /vmfs/volumes/65092bef-de8a06b5-22db-2bbbc32dc3d2
lrwxrwxrwx    1 root     root                 49 Sep 27 05:15 bootbank -> /vmfs/volumes/ff060de6-cecc88e5-4d14-8726d7ed0132
lrwxrwxrwx    1 root     root                 49 Sep 27 05:15 store -> /vmfs/volumes/3c3693e8-f77a642a-1910-5c6bdcb26d3a
drwxr-xr-x    1 root     root                512 Sep 27 05:14 vmfs
~ #
~ # esxcfg-rescan vmhba1
~ #
~ # ls -l /bootbank/
ls: /bootbank/: No such file or directory
~ #

Any suggestions?

[jjgunn]

If possible, rebuild the machine using the link in my post from June 16th and burn the A04 version of the ISO from Dell. I haven't had any issues with my fresh builds.

[robfish]

Thanks for this suggestion. I had thought about it.

Will all of the current VM's be retained and I would just need to add them to the inventory again?

[jjgunn]

Yes, as long as the VM's are located on external storage. If they're located on local storage, they would be wiped out unless you save them first.

[bulletprooffool]

Morning,

This is definteily something we have seen before and in almost all cases was related to bootbank not backing up as it should (genreally a faulty filesystem)

I have a blog post for this with a fix and a more complete explanation of what is happening:

http://www.get-virtual.info/2011/02/02/esx-host-losing-settings-at-reboot-checking-system-partitions-of-esx-host/

[Seemannger]

If is passthrough configured for USB-Device disabled this, because SD-Cards are bound via USB

[Terry3]

Gentlemen, I am having this issue and I would wager a guess this is related to Dell Openmanage software installation, or the actual ESXi ISO used for setup.

I installed ESXi 4.1 348481 using the ISO from VMware website. Then configured Networking, NTP, storage, SNMP and as a last step, installed Dell Openmanage OM-SrvAdmin-Dell-Web-6.5.0-2247.VIB-ESX41i_A01.zip. After this is done I rebooted the host, lost my configuration, then I ended up at this forum post.

I haven't had a thorough chance to test this, but it's either the VMware ISO or Dell OM 6.5.0 along with first reboot. Subsequent reboots don't clear the configuration. If I get a chance I will post back with results.