vSphere 8 post deployment Kickstart issue

[fehret]

Hi there,

Mentioned first on William's blog here: https://williamlam.com/2022/09/homelab-considerations-for-vsphere-8.html 

I had trouble deploying vSphere 8 with PXE in the first place but now I'm facing another issue: the second part of my kickstart is apparently not taken into consideration (mentioned as "Stage 02 - Post installation")

This configuration is still working with 7.X

This is an extract (I've more commands for the network part, but as the host doesn't have the second vswitch nor being in maintenance mode, I think this complete second section of the ks is ignored.

 

##### Stage 01 - Pre installation:

### Accept the VMware End User License Agreement
vmaccepteula

### Set the root password for the DCUI and Tech Support Mode
rootpw mysecretpass

### The install media (priority: local / remote / USB)
install --firstdisk=local --overwritevmfs --ignoreprereqwarnings --ignoreprereqerrors --forceunsupportedinstall

### Set the keyboard layout
keyboard "Swiss German"

### Set the network to DHCP on the first network adapter
network --bootproto=dhcp --device=vmnic0

### Reboot ESXi Host
reboot --noeject

##### Stage 02 - Post installation:

### Open busybox and launch commands
%firstboot --interpreter=busybox

### Enable maintaince mode
esxcli system maintenanceMode set -e true

### Set Search Domain
esxcli network ip dns search add --domain=mydomain.local

## Add second vSwitch & portgroup
esxcli network vswitch standard add --vswitch-name=vSwitch1
esxcli network vswitch standard portgroup add -v vSwitch1 -p "VSAN Network"

----- some more network settings----

### Disable IPv6 support (reboot is required)
esxcli network ip set --ipv6-enabled=false

## register with vcenter
esxcli network firewall ruleset set -e true -r httpClient
wget --no-check-certificate -O vcenter80.py https://webserverip/VMware/vcenter80.py
/bin/python vcenter80.py

### Reboot
esxcli system shutdown reboot -d 15 -r "rebooting after ESXi 8.0 host configuration"

 

Don't know if anything changed from the syntax point of view but where can I start looking ?

(I looked the esxi_install.log but there is a lot in there)

Thanks in advance for your help! 

 

[Daniel Casota]

Hi,

have you double checked the storage driver (.vib) ? What is the matching driver for “local” in ESXi7.0 or after a successful 8.0 setup without scripting? Is vmnic0 recognized?

[fehret]

Hi,

The thing is: all the first steps are successfully executed : 

• ESXi is installed and boots
• the root password & keyboard layout are correct 
• vmnic0 is properly configured as expected and I can access the console vis https after the reboot

But of course, I dont get:

• my second vSwitch (vSwitch1)
• IPv6 disabled
• the python script executed (with the vCenter join, etc.)
• a second reboot at the end

I would expect that a driver problem would stop the installation right at the beginning no?

[Daniel Casota]

Here some ideas.

1) As it seems that the goal is a vSAN setup, have a look to https://github.com/lamw/usb-to-sddc/blob/master/KS.CFG. See section %pre --interpreter=busybox

2) Do not reboot the esxi host and have a look to the logs.

 

edited:
If it worked on ESXi 7, and you do not include usb devices, the following hint is obsolete.

%pre --interpreter=busybox
# stop USB Arbitrator service to access USB device in ESXi Shell
/etc/init.d/usbarbitrator stop

 

[fehret]

Hi,

It's not necesarily to use VSAN, but it makes the nested host "VSAN ready".

I've tried to remove the boot, but I can't access the logs : main Windows ask for reboot and the web console is not yet available.

You only have access to some logs after the first reboot.

Just to say that I don't know how to move forward

[Daniel Casota]

As it is a PXE provisioning environment for a nested ESXi host which works on ESXi7.0 but not anymore on ESXi 8.0, do you use syslinux 3.86 ? 

edited:
I would check first if it is the same for booting from media and using pxe boot.

[Jangari]

Is Secure Boot enabled on that host?

If yes, %firstboot section is skipped in kickstart.

Installation and Upgrade Scripts Used for ESXi Installation
https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-esxi-installation/GUID-51BD0186-50BF-4D0D-8410-79F165918B16.html#firstboot-21

> Important:
> The %firstboot script does not run, if secure boot is enabled on the ESXi host.

[Daniel Casota]

nice catch , dumb me, didn’t see that, outch.

[fehret]

 Hero of the day!

Thanks a lot, you nailed it (thanks for the documentation too, this is also what I was looking for - probably didn't use the right keywords)

I did recreate the VMs from scratch when I had trouble installing ESXi 8 without knowing that by activating "all the fancy stuff", I would create other issues.

So now, feature request to support secure boot?

 

 I really appreciated the time you spent, thanks to you too. I'll study your script to learn a little more.

[Daniel Casota]

In Switzerland we call it "Baustelle" (building site).

Disabling/enabling Secure Boot feels wrong, but there is no sort of service 'let's encrypt' to only allow the execution of secure boot signed community packages. There is no ~light certification.

There is no Redhat8 like syntax and support e.g. for a liveimg parameter. A spanned workflow including firmware updates, well, Redhat introduced a workflow called 'Booting a beta system with Uefi Secure Boot' and for bootable, immutable, versioned filesystem trees there would be an ostree parameter.

There is a tiny tool Ventoy which is able to run when Secure Boot is enabled and start ISO files (ESXi, too). There is an ongoing discussion e.g. with the developer of Rufus, etc. about what-ifs. There were some fixes and refixes to provide an open-source powered solution for the topic there.

edited: Maybe there are vendors providing KMS-based solutions.

 

[fehret]

In Switzerland we call it "Baustelle" (building site). --> Where ? I work in Basel!

I can keep safe boot off for nested hosts I think but I will surelly turn them on on production servers.

[acancro]

> Important:
> The %firstboot script does not run, if secure boot is enabled on the ESXi host.

OMG.  I've been fighting a problem for days, and this little nugget of information got things working.  Thank you!