Failed to create a virtual machine on vCenter

[fatafly]

Dear VMware engineer,

  

Recently, I failed to create the virtual machine and migrate the virtual machine on VCSA (6.7.0.42000), but I can create the virtual machine on ESXI. The error report is as follows:

Failed to Create a virtual machine:

  PBM error occurred during PreCreateCheckCallback: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

Failed to migrate virtual machine:

I have restarted vCenter Server Application three times, and the problem still exists. but I don't know how to restart the service.

Thank you very much.

[sarikrizvi]

Check if vMotion is enabled on VMkernel portgroups.

Also share services status from vCenter.

service-control --status --all

[fatafly]

vMotion is enabled on VMkernel portgroups：

My vCenter is vCenter Server Appliance(like Linux System), I don't know how to look at the Services status.

The following is the service status of vSphere:

The following error was reported when creating VM in vCenter. ESXI can create VM normally.

[sarikrizvi]

1. Open vCenter VAMI Console

https://vCenterIP or FQDN Name:5480/

(login in it with root user and password of root)

2. Enable SSH.

3. Connect vCenter Server IP via SSH with Putty tool.

    (login in it with root user and password of root)

4. run below cmd to get services status.

shell

service-control --status --all

[fatafly]

The result of running the ”service-control --status --all“ command is as follows ：

[sarikrizvi]

vCenter Services looks good, no issue with them.

A.Can you try to install certificate..

1. From a client system Web browser, go to the base URL of the vCenter Server system or the vCenter Server Virtual Appliance without appending port numbers or 'vsphere-client' extension.
   For example:
   https://vcenter.domain.com/  Enter your vCenter FQDN details on a Web browser
2. Click the Download trusted root CA certificates link at the bottom of the grey box on the right and download the file.

3. Change the extension of the file to .zip. The file is a ZIP file of all root certificates and all CRLs in the VMware Endpoint Certificate Store (VECS).
4. Extract the contents of the  ZIP file. The result is a .certs folder that contains two types of files. Files with a number as the extension (.0, .1, and so on) are root certificates. Files with an extension that starts with an r (.r0,. r1, and so on) are CRL files associated with a certificate.
5. Install the certificate files as trusted certificates by following the process that is appropriate for your operating system.
   For most Microsoft Windows systems, you can follow the instructions at Manage trusted root certificates.

you can follow it too ---- vSphere: error when uploading files to datastores • Nolabnoparty

B. After that use vCenter FQDN name ( https://vcenter.domain.com/ )

C. Then try to create virtual Machine.

[fatafly]

Received. Thank you again for your warm support. It's a pity. My time is at 01:20. I will test it tomorrow and give you the feedback. Is that ok?

[fatafly]

Dear VMware engineer

I tried, I installed the certificate into the computer trust following the tutorial, and the VM creation failure still exists.

I found a problem, before login vCenter, use IP login, login success will automatically link into the domain name, now the United States and Europe jump.

Now：

Before.Normal state：

Now neither IP login nor domain login can create VM.

Now there are so many problems in vCenter that I can't authorize new users.

[sarikrizvi]

hmm, Can you try to change DRS to Manual and try to create VM after selecting an ESXi Host, see if that works.

[fatafly]

There is a separate host in my data center.The cluster is not open. The host in the cluster is closed.Standalone hosts do not have DRS.I think this is too much of a problem for me to recover with a snapshot, but my snapshot of VCSA was created when the initial installation of VCSA was completed.Do you know how to maximize VCSA with snapshot recovery?

[sarikrizvi]

1. you can revert snapshot to go back to old state.

Right Click on vCenter VM >>> Click on Snapshot Manager >> Select existing Snapshot >> Click on Revert.

2. If you don't want to go back to old state of your vCenter then you can delete snapshot.

Right Click on vCenter VM >>> Click on Snapshot Manager >> Select existing Snapshot >> Click on Delete/Delete All

[fatafly]

One thing I found that my vCenter is quite different from normal is the data center authority of the Administrator account:

Normal vCenter.Data center permissions for the Administrator account：

Abnormal vCenter.Data center permissions for the Administrator account:

Do you know how to restore the original authority of Administrator?

[IRIX201110141]

About the initial error.. the PBM.* stands for Policy Based Management. Iam only aware of the Storage Policy Based Management and its a service on the vCenter/VCSA and when this is not working anymore you get the error messages from above. A simple restart of that service can fix the problem. Iam pretty sure that a KB exists for that problem.

Now you see serveral problems and i think your VCSA? have a major problem and you should check if there is enough free space on the variaous partition within the PhotonOS. When the /storage/log runs out of space then the various services start dying and you will loose control in vCenter.  So check the space, free up space by deleting old and large logs and restart the entire VCSA after that.  Please dont touch the /storage/archive.

Regards,
Joerg

[nachogonzalez]

Are you using Site recovery manager?
Do you have more than one vCenter?
Being that VM? is it possible that you are trying to create a VM on a replicated datastore and that's why it is failing?

Can you try creating the VM in another datastore?
Also check the datastore policy you are selecting for the VM.

Warm regards

[nachogonzalez]

Also, please take a look at

VMware Knowledge Base

[fatafly]

Dear VMware engineer：

I was unable to start the vmware-sps service based on the KB operation you provided. The diagram below:

VMware Knowledge Base

So I found the following KB. After the operation. The diagram below:

VMware Knowledge Base

There is currently no vmware-sps service. The diagram below:

[fatafly]

In vCenter Server Appliance, this log is located at /var/log/vmware/vpxd/vpxd.log file.

2020-04-14T06:30:11.512Z info vpxd[05536] [Originator@6876 sub=vpxLro opID=14f60db1] [VpxLRO] -- BEGIN lro-759906 -- ViewManager -- vim.view.ViewManager.createContainerView -- 52cb4f80-4711-5987-a3d0-97a1f013646f(52e3a95f-a63b-ff63-83ab-b9be1acc4c47)

2020-04-14T06:30:11.514Z info vpxd[05536] [Originator@6876 sub=vpxLro opID=14f60db1] [VpxLRO] -- FINISH lro-759906

2020-04-14T06:30:11.596Z info vpxd[04105] [Originator@6876 sub=vpxLro opID=4f222563] [VpxLRO] -- BEGIN lro-759909 -- session[52cb4f80-4711-5987-a3d0-97a1f013646f]52d67378-a565-c423-9b32-18015aaf94a3 -- vim.view.View.destroy -- 52cb4f80-4711-5987-a3d0-97a1f013646f(52e3a95f-a63b-ff63-83ab-b9be1acc4c47)

2020-04-14T06:30:11.596Z info vpxd[04105] [Originator@6876 sub=vpxLro opID=4f222563] [VpxLRO] -- FINISH lro-759909

2020-04-14T06:30:19.527Z info vpxd[04008] [Originator@6876 sub=HostGateway] CmConnectionFSM::RunFSM(ST_CM_CALL_FAILED)

2020-04-14T06:30:19.535Z warning vpxd[05484] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f032c063790, h:30, <TCP '127.0.0.1 : 45094'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-04-14T06:30:19.535Z error vpxd[04008] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-04-14T06:30:19.536Z warning vpxd[04008] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-04-14T06:30:19.609Z warning vpxd[03983] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f034039caf0, h:36, <TCP '127.0.0.1 : 45100'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-04-14T06:30:19.609Z error vpxd[04008] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-04-14T06:30:19.609Z warning vpxd[04008] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-04-14T06:30:19.668Z warning vpxd[03998] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f031c2adb50, h:36, <TCP '127.0.0.1 : 45102'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-04-14T06:30:19.669Z error vpxd[04008] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-04-14T06:30:19.669Z warning vpxd[04008] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-04-14T06:30:19.720Z info vpxd[05357] [Originator@6876 sub=Vsan opID=HB-host-66@5762-362ca1a8] [CheckDatastoreCapacity] Clearing DatastoreNoCapacity configIssue

2020-04-14T06:30:19.720Z info vpxd[04055] [Originator@6876 sub=vpxLro opID=HB-host-66@5762-362ca1a8-01] [VpxLRO] -- BEGIN lro-759914 --  -- VsanClusterConfigIssuesLRO --

2020-04-14T06:30:19.720Z info vpxd[04055] [Originator@6876 sub=Vsan opID=HB-host-66@5762-362ca1a8-01] [VsanClusterConfigIssueLRO] Clear CFG_ISS_VSAN_CLUSTER_NOSTORAGE configIssue for cluster vim.ClusterComputeResource:domain-c58

2020-04-14T06:30:19.720Z info vpxd[04055] [Originator@6876 sub=vpxLro opID=HB-host-66@5762-362ca1a8-01] [VpxLRO] -- FINISH lro-759914

2020-04-14T06:30:19.729Z warning vpxd[04872] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f02dc2c2fc0, h:36, <TCP '127.0.0.1 : 45104'>, <TCP '127.0.0.1 : 18090'>>, e: 111(Connection refused)

2020-04-14T06:30:19.730Z error vpxd[04008] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-04-14T06:30:19.730Z warning vpxd[04008] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2020-04-14T06:30:19.730Z warning vpxd[04008] [Originator@6876 sub=HostGateway] Ignoring exception during refresh of HostGateway cache: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

--> [context]zKq7AVECAAAAAHXd5gAOdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbAP6dGADHOCMADN0lAFTlJQDqASYAsQsmADmiIwBxbyMAOnIjAJ1WKwHUcwBsaWJwdGhyZWFkLnNvLjAAAt2ODmxpYmMuc28uNgA=[/context]

2020-04-14T06:30:20.208Z info vpxd[03989] [Originator@6876 sub=vpxLro opID=vsan-PC-5a2bfbe73e7d7-W16-d3] [VpxLRO] -- BEGIN lro-759915 -- VpxSettings -- vim.option.OptionManager.queryView -- 52cb4f80-4711-5987-a3d0-97a1f013646f(52e3a95f-a63b-ff63-83ab-b9be1acc4c47)

2020-04-14T06:30:20.208Z info vpxd[03989] [Originator@6876 sub=vpxLro opID=vsan-PC-5a2bfbe73e7d7-W16-d3] [VpxLRO] -- FINISH lro-759915

[fatafly]

One thing I found that my vCenter is quite different from normal is the data center authority of the Administrator account:

Normal vCenter.Data center permissions for the Administrator account：

Abnormal vCenter.Data center permissions for the Administrator account:

Do you know how to restore the original authority of Administrator?