What I found that worked for me.
I removed IPv4 from the esx hosts, with just IPv6 all the hosts used the same address family for the vmk family.
Made sure that the vcenter appliance has working IPv6, somehow it keeps getting disabled on the vcenter 6.7 appliance, using the console I enabled IPv6, restart management, add prefix, restart management. Works.
Now you can upgrade the hosts using the update manager, the ISO is local, so that works, but the other updates are not reachable over IPv6 only :smileysad: so are not applicable.
After upgrading the entire cluster over IPv6 with DRS shuffling the VMs around I added the v4 addresses back to each host because otherwise I can't perform other updates.