ESXi remote host, vCenter and updates

[fehret]

Dear all,

I'm struggling with one remote host (6.5 U1) and my vCenter (6.7 U1 appliance) updates.

The remote host is rented in a data center and serves as a "backup plan" for ore services (DCs, Exchange DAG)

It is available through a IP in Internet.

When I had no problem connecting that host to my vCenter, I have difficulties making update manager to work.

Looking at logs a little bit further, I've seen that the host tries to reach the vCenter by looking at DNS name.

Nothing strange there, but as the host is not really on the network, it won't find it.

Some more info :

* All VMs are on a isolated vSwitch, except the firewall that have 2 NICs.

* The FW VM makes a VPN tunnel connection back to the data center, using a second IP on Internet.

I though about putting a manual entry in host file and authorize the host IP to connect back to the vCenter through my firewall with NAT, but maybe there is a cleaner solution.

How would you solve that to get that host patched (and not transforming patching in a nightmare !) ?

Thanks in advance and best regards

ESXi host logs (esxupdate) :

2018-12-27T09:33:54Z esxupdate: 10819274: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetTimeout']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2018-12-27T09:33:54Z esxupdate: 10819274: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetRetries']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2018-12-27T09:33:54Z esxupdate: 10819274: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetRateLimit']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2018-12-27T09:33:54Z esxupdate: 10819274: esxupdate: INFO: --- Command: scan Args: ['scan'] Options: {'cleancache': None, 'viburls': None, 'retry': 5, 'loglevel': None, 'hamode': True, 'timeout': 30.0, 'meta': ['http://vcenter.domain.local:9084/vum/repository/hostupdate/vmw/vmw-ESXi-6.5.0-metadata.zip'], 'cachesize': None, 'nosigcheck': None, 'maintenancemode': None, 'proxyurl': None}

2018-12-27T09:33:54Z esxupdate: 10819274: BootBankInstaller.pyc: INFO: Unrecognized value "title=Loading VMware ESXi" in boot.cfg

2018-12-27T09:33:54Z esxupdate: 10819274: BootBankInstaller.pyc: INFO: Unrecognized value "title=Loading VMware ESXi" in boot.cfg

2018-12-27T09:33:54Z esxupdate: 10819274: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/bootOption', '-rp']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2018-12-27T09:33:54Z esxupdate: 10819274: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/bootOption', '-ro']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2018-12-27T09:33:55Z esxupdate: 10819274: HostImage: INFO: Installers initiated are {'boot': <vmware.esximage.Installer.BootBankInstaller.BootBankInstaller object at 0xb9b4c0dd8>, 'live': <vmware.esximage.Installer.LiveImageInstaller.LiveImageInstaller object at 0xb9b32b4a8>, 'locker': <vmware.esximage.Installer.LockerInstaller.LockerInstaller object at 0xb9b32b780>}

2018-12-27T09:33:55Z esxupdate: 10819274: downloader: DEBUG: Downloading http://vcenter.domain.local:9084/vum/repository/hostupdate/vmw/vmw-ESXi-6.5.0-metadata.zip to /tmp/tmpj3_uj5ss...

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR: An esxupdate error exception was caught:

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR: Traceback (most recent call last):

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/urlgrabber/grabber.py", line 1495, in _do_perform

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR: pycurl.error: (6, "Couldn't resolve host 'vcenter.domain.local'")

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR: During handling of the above exception, another exception occurred:

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR: Traceback (most recent call last):

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Downloader.py", line 199, in _getfromurl

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/urlgrabber/grabber.py", line 1182, in urlgrab

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/urlgrabber/grabber.py", line 1036, in _run_callback

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/urlgrabber/grabber.py", line 1030, in _do_raise

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/urlgrabber/grabber.py", line 1178, in urlgrab

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/urlgrabber/grabber.py", line 1097, in _retry

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/urlgrabber/grabber.py", line 1070, in _retry

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/urlgrabber/grabber.py", line 1163, in retryfunc

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/urlgrabber/grabber.py", line 1265, in __init__

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/urlgrabber/grabber.py", line 1602, in _do_open

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/urlgrabber/grabber.py", line 1740, in _do_grab

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/urlgrabber/grabber.py", line 1736, in _do_grab

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/urlgrabber/grabber.py", line 1588, in _do_perform

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR: urlgrabber.grabber.URLGrabError: [Errno 14] curl#6 - "Couldn't resolve host 'vcenter.domain.local'"

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR: During handling of the above exception, another exception occurred:

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR: Traceback (most recent call last):

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Transaction.py", line 83, in DownloadMetadatas

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Downloader.py", line 289, in Get

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Downloader.py", line 202, in _getfromurl

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR: vmware.esximage.Downloader.DownloaderError: ('http://vcenter.domain.local:9084/vum/repository/hostupdate/vmw/vmw-ESXi-6.5.0-metadata.zip', '/tmp/tmpj3_uj5ss', '[Errno 14] curl#6 - "Couldn\'t resolve host \'vcenter.domain.local\'"')

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR: During handling of the above exception, another exception occurred:

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR: Traceback (most recent call last):

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/usr/sbin/esxupdate", line 239, in main

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:     cmd.Run()

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esx5update/Cmdline.py", line 105, in Run

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR:   File "/build/mts/release/bora-5969303/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Transaction.py", line 85, in DownloadMetadatas

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: ERROR: vmware.esximage.Errors.MetadataDownloadError: ('http://vcenter.domain.local:9084/vum/repository/hostupdate/vmw/vmw-ESXi-6.5.0-metadata.zip', None, '(\'http://vcenter.domain.local:9084/vum/repository/hostupdate/vmw/vmw-ESXi-6.5.0-metadata.zip\', \'/tmp/tmpj3_uj5ss\', \'[Errno 14] curl#6 - "Couldn\\\'t resolve host \\\'vcenter.domain.local\\\'"\')')

2018-12-27T09:33:55Z esxupdate: 10819274: esxupdate: DEBUG: <<<

[IRIX201110141]

Well....

you can add an entry to the remote ESXi local /etc/hosts file to solve the DNS issue. After that you have to configure the FWs and let the remote ESXi setting up a connection to your vCenter on tcp/9084.

Another method can be to patch the host manually by using the last *.zip or just let im connect to vmware.com by updating against a selected profile (this goes very quickly!). After that you can use VUM to check the compliance status of the patched hosts.

Regards,

Joerg

[fehret]

Thanks Joerg,

I'll try the first option as I thought because the second one with manual updates will still not give me a compliance status in vCenter... So not very convenient.

I think (I hope !!!) I can manage to get that to work, I'll try to document the process and give a feedback here.

But as I suspected as well, there is of course no miracle. Many thanks for your quick reply !

[daphnissov]

It is available through a IP in Internet.

This is a very bad and dangerous idea. ESXi or vCenter should never be placed directly on the public Internet but always behind some secured connection like VPN.

[fehret]

Design is not the best but it is so for a cost effective point of view... but you know what is worse ?

A non-patched server on the Internet ! ;-)

If local ESX firewall is properly configured (white list of IPs for remote access, etc...), we should be fine no ?

[daphnissov]

ESXi is not hardened enough nor designed to be subject to Internet attacks. So you say this is for cost cutting reasons. What's cheaper, paying for proper infrastructure design and implementation, or having your business go down? How about data breach and randsomware?

[fehret]

Who spoke about business ? :smileywink:

It's part of my lab environment...

(And yes I know it's not the best and I know it's overkill... :smileysilly:)

PS : The host entry with appropriate firewall rules works. Thanks you both for all advice !

[IRIX201110141]

I'll try the first option as I thought because the second one with manual updates will still not give me a compliance status in vCenter... So not very convenient.

This is not right. If you press the rescan button within VUM it check the esxi patch/software status and compare it against the current Baselines. It works not in that way that VUM compare the patch status against its own history database.  So... manual patching, rescan gives you a valid compliance status in VUM.

Regards,

Joerg

[fehret]

In fact not, after a reboot of vCenter, compliance host status is unknown and as the ESXi host can't reply, it remains so even if I click on manual scan.

The logs are from a manual scan in fact.