Port 7444 Issue at Stage 2 when attempting upgrade from vCenter 7 to 8

[dauphin77]

Hello,

We are currently encountering an issue attempting to upgrade from vCenter 7.0.3.02200 Build 24322018 to vCenter 8.0.3 Build 23794027. We are able to reach Stage 2 of the process but encounter "Error: Failed to get server certificate for validation" error. We have defaulted all certificates (Options 4 and 8 of vSphere Certificate Manager), no firewalls in place to prevent port 7444 from being accessible. 

The error is similar to what is found in the below article:

vCenter upgrade to 8 Pre-Check fails with error "Failed to get server certificate for validation"

Broadcom		remove preview
vCenter upgrade to 8 Pre-Check fails with error "Failed to get server certificate for validation" Issue/Introduction vCenter upgrade to 8 fails during Pre-Check with the below error. "Error Failed to get server certificate for validation Resolution Make sure that the sso service is reachable and started before continuing" This issue is seen when port 7444 is not accessible on the vCenter. View this on Broadcom >

We have tried numerous steps and cannot get past the port 7444 issue. When we run the curl -vvv -k https://127.0.0.1:7444 it continues to be blocked.

Looking for any additional suggestions that may be causing this issue. We have combed most of the logs and nothing seems to stand out in trying to resolve this issue.

[ChrisLeblanc]

Hi,

I'm not sure if this is related but we recently setups a few vCenter 8 sites. When I went to put a certificate from our CA, I found out that v8 now requires SHA256 level of encryption for the certificates.

Again, I don't know this could be part of it but perhaps check what the current cert has in case. Sometimes error messages can be cryptic.

[dauphin77]

Chris,

Appreciate the response. We did default all of our certs to the default certs to try and circumvent that very issue. We were still unable to proceed with the vCenter 8 upgrade (Stage 2).

This is a strange issue and shouldn't be this difficult to solve. We may just have to recommend rebuilding the vCenter to our customer, but hoping to avoid that process as this is a production environment. 

[a_p_]

Did you check whether someone hardened the old vCenter Server, and port 7444 is blocked now?
See e.g. https://knowledge.broadcom.com/external/article/320878/disable-port-7444-on-the-psc-or-embedded.html

As a side note, I assume that you mixed up the mentioned build number for the new vCenter Server version with one for ESXi hosts.

André

[dauphin77]

André,

Good catch on the wrong vCenter 8 build number. Should have been Build 24322831.

I ran through the steps in KB320878, reloaded the firewall and rebooted vcsa and did not resolve the issue. This vCenter is hardened but we had numerous other vCenter upgrades that upgraded to version 8 successfully. On a side note, I was advised that our customer had to restore this particular vCenter Server and it took them a couple tries before they were able to successfully restore after an outage but not sure if that may be part of our problem. I believe there may have been an underlying issue prior to the upgrade/restoral that may be causing our issue. 

We have utilized lsdoctor and that did not produce any significant errors or issues. Additionally, we have deployed a standalone vCenter 7 instance to compare config files but have not crossed anything that stands out yet. I have also checked roles, permissions, users and groups and all seems to match up.

My next step is to upload and run the VCF Diagnostic Tool for vSphere (VDT) against the vCenter and see if it provides any additional details or issues that may point us in the right direction in hopes of resolving this issue. I'm also still combing logs looking for any errors that may identify the problem.

One other item that I noticed and not sure if it's relevant, but the vmdird log file in /var/log/vmware does not exist. 

If you have any other suggestions please let me know. 

**Update: we did also disable FIPS Mode and that did not aide in resolving the issue. 

[Alexandru Capras]

Hi,

I'm a bit late to the party, but did you manage to solve this?

Found your issue interesting and did a bit of research:
One article says that port 7444 was originally used by STS service, but it's not used in more recent versions of vCenter.
However, the vSphere 8.0 release notes mention that 7444 is the default HTTPS port number for vCenter Single Sign-On.

Got me confused and I did check directly in production on a vCenter version 8.0.3:
Took the command from this KB320878, and instead of disabling the firewall config I checked what I have "by default":

cat /etc/vmware/appliance/firewall/sts
{
  "firewall": {
    "rules": [
      {
        "name": "sts.ext.port1",
        "port": "7444",
        "direction": "inbound",
        "portoffset": 0,
        "porttype": "dst",
        "protocol": "tcp"
      }
    ],
    "enable": true
  }


cat /etc/vmware/appliance/firewall/vmware-sso
{
  "firewall": {
    "enable": "true",
    "rules": [
      {
        "direction": "inbound",
        "name": "sts.smartcard.port",
        "protocol": "tcp",
        "porttype": "dst",
        "port": "3128",
        "portoffset": 0
      }
    ]
  }
}

Also, this blog post is worth checking out. Issue around the same port number but different cause that was solved with lsdoctor.