Are you running VirtualCenter at all? If so can you try selecting that as your destination then picking the ESX server. Can you webaccess into the ESX server without problem? Converter 3.0.1 now uses port 443, you might also try using Converter 3.0.0 which does not use the web port it uses 902 instead. Otherwise you can try re-generating the SSL cert for the ESX server using the process below, if you restart mgmt-vmware and have auto vm startups enable there is a bug that will shutdown your VM's. This is only on ESX 3.0.1 and is fixed with the 5/15 patches.
3.0.1 mgmt-vmware/auto startup bug fix - http://www.vmware.com/support/vi3/doc/esx-7557441-patch.html
1. Change directories to /etc/vmware/ssl
2. Create backups of any existing certificates by executing the
following commands:
mv rui.crt orig.rui.crt
mv rui.key orig.rui.key
3. Enter the following command to restart the vmware-hostd
process: service mgmt-vmware restart
4. Confirm that the ESX Server host generated new certificates by
executing the following command comparing the time stamps of
the new certificate files with orig.rui.crt and orig.rui.key:
ls -lha.