How can I make a VM boot from its second disk?

[pwsteele]

I have a custom CentOS installer image that I use to perform an automated hands-free install. When used on real hardware, this raw image is normally copied to a USB stick and then the server is booted from the USB drive. The logic embedded in the installer image then performs an automated install of a custom CentOS onto the server's hard drives. When the install process completes, there is logic that sets the USB drive as not bootable (parted /dev/sda set 1 boot off) and then performs a reboot. Because the USB drive is no longer marked as bootable, the BIOS automatically cycles through to the next bootable drive and boots from there. The USB stick can then be removed at the user's leisure later (or be left in if the user want to leave it there).

I am attempting a similar workflow using a VMware based VM created with vSphere (ESXi 6). I convert the raw image into a vSphere compatible disk (using the CentOS qemu-img tool). I then create a vSphere VM with disk 1 (sda) defined to point to this boot image and disk 2 (sdb) defined as a new disk taken from one of my datastores.

The VM boots fine with this configuration, booting off the boot image I've defined as drive 1. The hands-free install then proceeds without issue, up until the automated reboot phase. The install logic correctly sets disk 1 (sda) as not bootable and then reboots the VM. Instead of booting the new OS that has just been installed on disk 2 (sdb), the VM just sits there trying to find a bootable device. It's smart enough to recognzie that sda is not marked as bootable, but it doesn't seem to work like real hardware and automatically scan for the first bootable disk device.

The only option I've found is to power off the VM then remove disk 1 from VM definition (which makes disk 2 become disk 1), or alternatively change the drive order and move disk 1 to the disk 2 position and disk 2 to the disk 1 position. Both of these "solutions" get old pretty fast when a lot of these installs are done, especially considering this install process is designed to be hands-free. Is there a way to configure a vSphere VM when booted to automatically scan available disk drives looking for the first bootable disk similar to what happens with real hardware?

[continuum]

That sounds very familiar ....
A few weeks ago I had a similar project:

Task: provide boot media running Ubuntu 64
should start from CD,USB-stick or ISO, in a VM or elsewhere
then it automatically installs a new Ubuntu with a VMware Viewclient to any eventually existing local harddisk.
When the installer finishes it reboots and on next start it already boots from local disk.
The system works on all notebooks we tried, with or without EFI-firmware and on all possible VMs. EFI-firmware or MBR
The result now is a iso-file using grub2.
If I run this in a VM it also has to change boot-order on second boot - that is acchieved by ejecting the boot-media after successful install.
This is a function that works predictable inside a VM or on real hardware.
If it helps I can go into the details

Ulli

[pwsteele]

In my case our boot media is a bootable disk image, not an iso. It's prepared as a vmdk file for the VMware environment and a scsi disk device is created that maps to this vmdk file. In the BIOS it is included in the list of available hard drives, not as one of the removable disks. As such, it cannot be "ejected". If I attempt to do so, using the CentOS eject command, that's exactly what I'm told:

# eject --scsi /dev/sda

eject: /dev/sda: is not hot-pluggable device

So I don't think your solution will work for us. On real hardware the solution was simply to mark the disk as not bootable. VMware doesn't handle this.

[continuum]

I would not expect that this works:
disk1 - scsi0:0

disk2 - scsi0:1

but this options may work:
disk1 - ide0:0
disk2 - scsi0:0

or

disk1 - sata0:0

disk2 - scsi0:0

By the way .... a diskimage create with dd of /dev/sda is already a bootable vmdk - no need to convert it at all.
All you need is the additional vmdk-descriptor textfile thats point to sda.dd

[douglasarcidino]

To his point, you can make your install media the second disk in the boot order and the disk being installed to as the first.

[pwsteele]

I tried that but VMware wouldn't skip over the first disk to the second disk, even though the first disk was a brand new disk image and clearly not a bootable disk. I tried making my bootable disk an IDE disk instead of a SCSI disk but that did not work either--I could find no combination disk types/disk order that worked with VMware. It seems to see the first disk, recognizes it is not bootable, then tries to do a PXE boot instead. It does not scan ahead to the second disk.

[pwsteele]

I tried this same thing with KVM and Virtual Box and in both of those I could define my installer disk image to be the second (last) disk in the VM. When I booted the VM I could hit F12 to get the boot menu and select the second disk to boot from instead of the (empty) first disk. When the install was complete and the system rebooted, it automatically booted from the newly installed first disk of the VM, ignoring the original installer disk.

I've tried the same approach with VMware but the boot menu doesn't let you select a specific disk to boot from. The menu just lists "+Hard drive", and if you select this it boots from the first hard drive defined. The "+" in front of the name makes me think perhaps this is a way to expand this list so you can pick a specific disk, but I could find no key combination to do this.

I'm beginning to think there is no solution to this problem, which surprises me. I'd think this would be obvious feature for a virtual machine to support.

[douglasarcidino]

PXE or ISO based booting may be more ideal.

[pwsteele]

That option isn't really available to us. We have an existing installation framework that's geared towards a customized disk image. It works great everywhere except VMware. To change to an ISO or PXE approach would be costly (in terms of dev time).

Peter

[pwsteele]

By the way .... a diskimage create with dd of /dev/sda is already a bootable vmdk - no need to convert it at all.

All you need is the additional vmdk-descriptor textfile thats point to sda.dd

You are right, the qemu-img command that I use ultimately doesn't do anything to the original raw image, it just creates a text wrapper that points to the original image. The text wrapper looks something like this:

# Disk DescriptorFile

version=1

CID=7fbf5db7

parentCID=ffffffff

isNativeSnapshot="no"

createType="monolithicFlat"

# Extent description

RW 2222080 FLAT "hf-flat.vmdk" 0

# The Disk Data Base

#DDB

ddb.adapterType = "lsilogic"

ddb.encoding = "UTF-8"

ddb.geometry.cylinders = "138"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.virtualHWVersion = "4"

Even though the original image is copied directly, the qemu-img command is a convenient way to create the text wrapper. If I was to create it manually, what's the minimum I can get away with?