Error Downloading Product Binaries from MyVMware Account

[jrhaakenson]

Running vRLCM 8.4.1.1.  When I attempt to download new product binaries using the MyVMware option I receive the following error:

Error Code: LCMMYVMWARE60015

Failed to download one or more product binaries.  Please try again after some time.

Error occurred while downloading product binaries from MyVMware accout: <account name>.  Please check the logs for more details.

Specifically I am attempting to download the IDM 3.3.5 upgrade binaries.  My VMware account has been validated with the proper credentials in LCM.  I am able to login to My VMware with this account and download the IDM products manually.  I have the permissions on my VMware account to download products.  I'm using My VMware as the location for downloading the product binaries in LCM.  It discovers the binaries just fine so my connection to My VMware is good.  I have verified enough hard drive space exists to download the new binaries.

Is there something wrong with my account preventing me from downloading binaries in vRLCM?

 

[johnbowdre]

For whatever it may be worth, I just tested downloading the 3.3.5 upgrade binaries using the MyVMware connection from my 8.4.1.1 LCM appliance and it worked fine. 

Any further clues in /var/log/vrlcm/vmware_vrlcm.log ?

[jrhaakenson]

Attached a screenshot of the log entry.  Doesn't look like anything particularly useful.  There are a lot of null values.

[johnbowdre]

Hmm, nothing meaningful below that? Looks pretty much like mine does, lots of null fields.

[jrhaakenson]

Nothing meaningful below that no.  Does it maybe have something to do with the licenses I have added to my vRLCM?  I have six different licenses added that are for:

vRealize Suite 2018 Enterprise

vRealize Operations 6 Manager for Horizon

vRealize Operations 7 Standard

vRealize Operations 7 Standard

vRealize Operations 6 Manager for Horizon

vRealize Suite 7 Enterprise

All licenses are healthy and never expire.

[jrhaakenson]

I've attached a screenshot of some additional log information provided under the initial error.  A lot of java errors.  Any clues to what this is saying?

[johnbowdre]

I don't think that the licenses stored in vRLCM should have anything to do with it; the download process should really only be concerned about what products are associated with your MyVMware account.

This line from the log excerpt stands out to me:
java.net.SocketException: Connection reset

That sounds like it's a network issue to me, possibly outbound firewall related? 

[jrhaakenson]

That's what I was looking at late yesterday also.  I'll need to check with our network firewall team again for outbound connections from our vRLCM.

[jrhaakenson]

I'm starting to think that this has to do with My VMware account not having the appropriate products registered with it.

https://docs.vmware.com/en/VMware-vRealize-Suite-Lifecycle-Manager/8.4/com.vmware.vrsuite.lcm.8.4.doc/GUID-58A150C5-9E52-4F28-9D1A-673AB4561834.html

According to this link I need both vRealize Suite 2017 or later and vRealize Network Insight or NSX Data Center Enterprise Plus entitlements in order to download binaries from vRLCM.  I have vRealize Suite 2018 Enterprise registered with my account but not Network Insight or NSX Data Center.  I'm working with my team to get these entitlements and see if that corrects the issue.

[jrhaakenson]

This wasn't the issue.  I did end up having both products entitled on My VMware account and still not able to download the product binaries in vRLCM.  Still searching for a solution.

[johnbowdre]

Any clues from your network firewall folks? I'm thinking that reviewing a capture of the traffic as observed by the firewall could be a big help in figuring out where the connection reset is happening. That's helped me investigate very similar errors in the past.

[jrhaakenson]

I had the network team pull up the firewall monitor on our vRLCM and they did not see any traffic getting blocked.  They saw the connection being made to IP 23.66.114.144 which our DNS server resolves as a23-66-114-144.deploy.static.akamaitechnologies.com.  This looks like the correct binary repository so once again, the network connection is being made but the error continues to point towards the My VMware accounts not able to download the binaries.  It's as if the My VMware accounts aren't allowed to download the binaries and so the server closes the socket connection.

[johnbowdre]

I just tested with another MyVMware account which is not entitled for any vRealize products. While vRLCM is able to validate the account when I added it, attempting to download any binaries gives a 404 response code and a message about not being able to obtain user entitlements:

That's more in line with the response I expected from an account which wasn't sufficiently entitled. It seems likely that the server would tell the client why it's refusing the request rather than just aborting the connection with no explanation.

Very strange. 

 

[jrhaakenson]

I'm putting in a support request with VMware as I have nothing else to try.  Hope they can lead me to the correct solution.

[jrhaakenson]

Good news.  After 2.5 months, I have a working solution for this.  The binary download repository for vRLCM is https://download2.vmware.com.  There are a large number of other VMware download repositories that constantly change IP addresses, but we won't get into that here.  First make sure your DNS server can resolve VMware's plethora of download repositories.  But my vRLCM DNS could so this was not the issue.

What pointed me in the right direction was running the command: curl -v https://download2.vmware.com.  This output a refused TLS 1.2 connection.  I fixed a similar issue on my vIDM appliance by modifying the /etc/ssh/sshd_config file line 117 to remove some troublesome ciphers.  So I accessed my vRLCM sshd_config file located in /etc/ssh/sshd_config and scrolled down to line 117.  The first two ciphers listed are [e-mail] and [e-mail].  By removing these two ciphers and saving the sshd_config file I was able to finally open a TLS session with https://download2.vmware.com and download the binary files I needed to upgrade my managed vRealize appliance from the vRLCM.  My cipher list on line 117 of sshd_config only contains aes256-ctr,aes192-ctr,aes128-ctr now and this seems to work.  I'm not sure what the issue with the first two ciphers mentioned above was or why they were allowed to certain VMware update repositories (such as vrealize-update.vmware.com) but not download2.vmware.com.  Furthermore I'm not sure why these two ciphers by default do not affect other users but affected my appliance.  I requested answers to these questions from my open VMware support ticket but they have not been able to provide answers at this time.  Still I hope this information is useful to any users who experience update issues with vRealize appliances.  I have used it to fix update issues on both vRLCM and vIDM in my environment.

[qc4vmware]

Wow just had this same issue I think with my 8.14 LCM.  It has been upgraded many times so maybe some cruft.  Removing those ciphers seemed to fix my download issue.  I did also just turn on fips mode so maybe related to that as well?