How to debug problem with vCenter Converter Standalone?

[The Immortal]

Hello!

I'm trying to P2V an old server with Fedora 23 to ESXi 8.0.0. 
As known VMware vCenter Converter Standalone 6.6.0 should successfully support RHEL 7+ and Fedora 23 is related to RHEL 7.3.

I set the following parameters in C:\Program Data\VMware\VMware vCenter Converter Standalone\converter-worker.xml:

• useSourcePasswordInHelperVm to true (default is false)
• powerOffHelperVm should be set to false (default is true)

I set for Helper VM Netwrok settings from the same network as source machine.

But during converstion with 1% percent I got:

Error: The destination virtual machine did not boot up.

And the created VM on ESXi shows:
Operating System not found

You can find details in the attached screeshots:

Also all convertions logs are attached.

Could someone please help me to debug the problem please? And also I don't understand what exactly Operating System not found? It fails on 1% not even try to copy anything. So what's the deal then?.. =/

Thank you very much!

Attachment(s)

ConverterDiagnostics20250323235841.zip   52 KB 1 version

[Ivan Ivanov]

Hello,

Thanks for your feedback. First, to provide some background how Linux conversion works. When the job is submitted, Converter is creating the destination machine, then it remotely attaches a live Linux ISO image to it as a CD-ROM device and boots it from the ISO. Inside that image is a Converter component (the so-called Helper Service) which autostarts, Converter Server connects to the helper and instructs it to proceed with the conversion (i. e. source details, volumes to copy and so on). The Helper service takes over and does the actual data cloning from the source, writing the data to its locally attached virtual disks (i. e. those are the destination VM disks). After all data is cloned, Converter Helper is performing "reconfiguration" on the destination (rebuilding initrd, re-installing grub, updating fstab, networking, etc) and powers off the destination VM (unless the powerOffHelperVm setting that you mentioned earlier i changed to "false" - in this case the helper VM is left powered-on for troubleshooting purposes).

In your case it seems the remotely attached ISO did not work, which means the target VM was not able to boot from the helper ISO and the Helper Service did not start, which caused the conversion to fail as it cannot proceed without the Helper Service running (and this is the "Operating system not found" message that you see). Most likely this is caused by network issues. In order for the remotely attached ISO to work, there should be connectivity between the Converter Server machine and the destination ESX host on port 902 (note that it should be the ESX server on which the destination is created directly, but not vCenter server).

As a workaround you can try to copy the helper ISO to the destination ESX server datastore and configure Converter how to use it. This will switch from remotely-attaching an ISO to the destination VM, to attaching a local ISO image (living on the ESX datastore), which circumvents the Converter-to-ESX connectivity requirement on port 902. There are detailed steps of this process outlined at https://knowledge.broadcom.com/external/article/313474/when-converter-server-is-deployed-far-fr.html. Do not forget to restart the VMware Converter Worker service after editing the converter-worker.xml file, otherwise the changes would not be applied.

And lastly - yes, Fedora is close to RHEL, but it is not RHEL, so there is a chance some of the reconfiguration steps to fail and in this case they should be performed manually.

Hope this helps.

[The Immortal]

Hello Ivan,

Thank you very much for detailing description of the convertion proccess!

I understand the solution with locally attaching the ISO image (converter-helper-vm.iso) on the ESXi side, but before doing that, I'd like to debug why the problem occurs in the remote scenario.

I connected to the ESXi server via SSH and checked the availability of port 902 on the Converter Server machine:

[root@localhost:~] nc -zv 192.168.32.99 902
Connection to 192.168.32.99 902 port [tcp/authd] succeeded!

As you can see port 902 is available from ESXi server.

Also I checked port 902 from the Converter Server machine to the ESXi server:

>telnet 192.168.32.98 902

220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC , VMXARGS supported, NFCSSL supported/t, SHA256 supported

Everything looks good here as well.

What else can I check?

Thanks!

[Ivan Ivanov]

So in your case the networking looks OK. I can also see at the screenshots you have shared that the CD-ROM device is set to a remote device, which means at least the "initial" connection succeeded:

Not sure if any details would be visible if you open the CD-ROM device section for additional details.

Unfortunately I don't have much other good suggestions for troubleshooting. When attaching the remote ISO to the target VM, Converter is running a separate tool (shipped with the product) which is called remoteDeviceConnect.exe. It establishes the initial connection, connects the ISO as a remote device and waits until the ISO is detached after the conversion completes (that is, it is serving read/write requests to the ISO images, or actually READ requests as CD-ROM devices do not support WRITE). What I have seen over the years is that a network glitch interrupts this connection and the ISO is partially uploaded to the ESX server, but unfortunately we do not have a good logging in that area, as this remoteDeviceConnect.exe is a component we (Converter) are consuming directly from another team and do not have much control over it.

I would suggest the following steps to try to understand in more details what is going on:
   - check whether the remoteDeviceConnect.exe process is running on the Converter Server machine while the target OS is booting from the ISO
   - check if there are any details in the CD-ROM device configuration on the ESX server (expand the CD-ROM section)
   - check for some errors related to remoteDeviceConnect.exe in vmware-converter-worker.log file
   - you can try to find the remote device connection request on the ESX server side logs (the initial connection should go through the authd service (VMware Authentication Daemon - as you have noted earlier) and there should be a log file similar to authd in /var/run/log on the ESX server). In addition to the ESX server global logs, you can also check the VM log itself (as the VM is actually working with emulated devices - both local and remote), it should be located on the datastore inside the same directory where the target VM is created in a file e. g. vmware.log. You can also check for any errors related to the CD-ROM device (as we can see it was initially connected as a remote device, i. e. the initial connection was successful). Unfortunately I cannot provide more guidance on what to look for on the ESX host, I would search for a remote device connection request and whether it was established successfully, rejected or terminated at some point (prematurely).
   - check the boot order in the firmware configuration of the target VM and confirm it is configured to boot from a CD-ROM device. If this is not the case - you can reconfigure it and reboot it. Until the "wait" timeout in Converter expires you can reboot it multiple times and do experiments. If you are able to boot it successfully (either remotely or by a local ISO image on the datastore) and the Helper service starts - Converter Server will connect to it and the conversion process will resume.

In any case I would give a try of the locally copied ISO image and using a locally attached device instead of a remote one. If the boot order of the target VM is the issue however, it would not matter whether the ISO image is locally or remotely attached to the VM.

Regards,
Ivan

[ChrisLeblanc]

Hi,

not sure if it helps but here is something i've experienced converting linux systems in this method.

The other side creates a temp helper VM that boots up and will use a temp DHCP address (may need to confirm this service is available).  When I did my conversion, the other side didn't have DHCP as per an odd company policy..  my workaround was to find an ip in that range not in use and set the converter to use that.. it was one of the last steps.

look at the console of the destination. it might give you an idea if its getting an IP. its been awhile but I think you can interact with the helper system. ifconfig should show you if it has an address

Can't think of anything else except maybe some kind of firewall rule.