Can't access VAMI - "Unable to get deployment status."

[samsheikh]

I'm sort of a neophyte.  I've installed vCenter 7.0.3/20150588 in a non-production environment for learning purposes.  I cannot access the VAMI.  Whenever I try accessing https://<hostname>:5480/configurev2/#/ or https://<ip>:5480/configurev2/#/ , I get an "Unable to get deployment status." error.

 

I am, however, able to access vCenter client, on port 443.

 

I've tried what was outlined in following article, but it didn't work for me: https://communities.vmware.com/t5/VMware-vCenter-Discussions/VCSA-7-0-2-Upgrade-Installer-Stage-2-Export-Partition/td-p/2835625

EDIT

This is a reply to padapa and Alex_Romeo.  I'm not sure what's going on, but every time I respond, the message disappears.  This is now my 7th or 8th attempt at responding about the DNS.

DNS works.  I read a thread about how it was important, so I manually inserted the records early on.

C:\Program Files (x86)\VMware\VMware Player\bin>nslookup
Default Server:  ar---dc.net
Address:  192.168.xxx.13

> ar---vcenter.net
Server:  ar---dc.net
Address:  192.168.xxx.13

Name:    ar---vcenter.net
Address:  192.168.xxx.5

> 192.168.xxx.5
Server:  ar---dc.net
Address:  192.168.xxx.13

Name:    ar---vcenter.net
Address:  192.168.xxx.5

 

Any and all help will be greatly appreciated.

[ptarnawski]

Had a similar issue and the solution was to go to VAMI but  like this: https://<hostname>:5480/configure , drop v2 

[vjrk83]

Seeing the same issue and https://<vcsaip>:5480/configure  is giving 404 not found. 

This is a new vcenter install with build version 7.0u3e . Attempted to deploy the installation 3 times and nothing worked. 

 

Its failing in stage 2 with error "unable to get deployment status" . There is no way to change the export directory. 

[ptarnawski]

Could be a space issue: 

 

https://www.provirtualzone.com/how-to-add-extra-space-to-vcenter-for-the-upgrade/

 

 

[samsheikh]

Thank you for your response.

I tried this first, as was suggested in the link I posted (for some reason the link has a comma at the end, which is why it doesn't load), but link vjrk83, I also get a "404 - Not Found" error.

This is the shell output with regards to space:

Filesystem                                   Size  Used Avail Use% Mounted on
devtmpfs                                     5.9G     0  5.9G   0% /dev
tmpfs                                        5.9G  932K  5.9G   1% /dev/shm
tmpfs                                        5.9G  1.2M  5.9G   1% /run
tmpfs                                        5.9G     0  5.9G   0% /sys/fs/cgroup
/dev/mapper/vg_root_0-lv_root_0               47G  5.9G   39G  14% /
/dev/sda3                                    488M   30M  423M   7% /boot
/dev/sda2                                     10M  2.0M  8.1M  20% /boot/efi
tmpfs                                        5.9G  121M  5.8G   3% /tmp
/dev/mapper/vg_lvm_snapshot-lv_lvm_snapshot  147G   61M  140G   1% /storage/lvm_snapshot
/dev/mapper/core_vg-core                      25G   45M   24G   1% /storage/core
/dev/mapper/log_vg-log                       9.8G  602M  8.7G   7% /storage/log
/dev/mapper/db_vg-db                         9.8G  142M  9.2G   2% /storage/db
/dev/mapper/dblog_vg-dblog                    15G  105M   14G   1% /storage/dblog
/dev/mapper/seat_vg-seat                     9.8G   84M  9.2G   1% /storage/seat
/dev/mapper/netdump_vg-netdump               985M  2.5M  915M   1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy         9.8G   37M  9.3G   1% /storage/autodeploy
/dev/mapper/imagebuilder_vg-imagebuilder     9.8G   37M  9.3G   1% /storage/imagebuilder
/dev/mapper/updatemgr_vg-updatemgr            98G  320M   93G   1% /storage/updatemgr
/dev/mapper/archive_vg-archive                49G  142M   47G   1% /storage/archive
/dev/mapper/vtsdb_vg-vtsdb                   9.8G   68M  9.2G   1% /storage/vtsdb
/dev/mapper/vtsdblog_vg-vtsdblog             4.9G   36M  4.6G   1% /storage/vtsdblog
/dev/mapper/lifecycle_vg-lifecycle            98G  4.1G   89G   5% /storage/lifecycle

As for the space issue, can you elaborate?  I'm installing on a dedicated 1TB SSD, using the "tiny" configuration, so only about half of the space is being used.

[Navin A]

What happenes when you just try  https://<vcsaip>:5480 ?

[vjrk83]

Redirecting to /configurev2 to proceed to stage-2 . I was expecting something as mentioned in this KB  https://kb.vmware.com/s/article/83540

Since this is a new installation it might not work I guess. Anyways I was able to get the installation successful after entering the FQDN correctly. Thanks for your reply. 

[samsheikh]

I have tried both the https://<vcsaip>:5480 and https://<vcsaFQDN>:5480.  It first redirects to https://<ip or FDQN>/#/login/ and then to https://<ip or FQDN>/configurev2/#/

 

[Navin A]

Can you try

rm  /storage/applmgmt/backup_restore/restoreReconciliation-history.json

rm /etc/vmware/backupMarker.txt

reboot

[samsheikh]

I tried, but neither of those files exist.

I did reboot the machine, however, and now the vCenter portal doesn't work.  I know that sometimes it takes a bit of time for all the services to load, but I've been waiting for well over an hour, and I'm still getting nothing.

These are the services that are running:

Command> service-control --status
Running:
 lwsmd observability vmafdd vmcad vmdird vmware-pod vmware-vdtc vmware-vmon
Stopped:
 applmgmt lookupsvc observability-vapi pschealth vlcm vmcam vmonapi vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-content-library vmware-eam vmware-envoy vmware-hvc vmware-imagebuilder vmware-infraprofile vmware-netdumper vmware-perfcharts vmware-postgres-archiver vmware-rbd-watchdog vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-ui vstats vtsdb wcp

 This is the output when I started the services after stopping all of them:

Command> service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Successfully started service lwsmd
Performing start operation on service vmafdd...
Successfully started service vmafdd
Performing start operation on service vmdird...
Successfully started service vmdird
Performing start operation on service vmcad...
Successfully started service vmcad
Performing start operation on profile: NONE...
Successfully started profile: NONE.

This is the error message when I try to run a service:

Command> service-control --start vmware-vpxd
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vpxd...




Error executing start on service vpxd. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd"
            ],
            "localized": "An error occurred while starting service 'vpxd'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd"
            ],
            "localized": "An error occurred while starting service 'vpxd'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

[adamhart84]

After you run the "service-control --start --all the last line shouldn't say successfully started profile "NONE", it should say "ALL". I experienced this issue and checked the defaultStartProfile file located here /storage/vmware-vmon/defaultStartProfile.  I VI'd into that file and edited it to say "ALL" instead of "NONE" and all of my services started after that.

[padapa]

Curious to know if you DNS record is proper for this vcenter, especially if it has both the A record and the PTR records? I have seen that new versions of vcenter heavily rely on proper DNS hostnames for it to work and start the services as well.

[samsheikh]

Thank you for your replies padapa and Alex_Romeo.

One of the first things that I did was to register the DNS record of the device.  I read some threads about how this was an issue.  So, yes, the device records are sound.

 

 

 

C:\Program Files (x86)\VMware\VMware Player\bin>nslookup
Default Server:  ar---dc.net
Address:  192.168.---.13

> 192.168.---.5
Server:  ar---dc.net
Address:  192.168.---.13

Name:    ar---vcenter.net
Address:  192.168.---.5

> ar---vcenter.net
Server:  ar---dc.net
Address:  192.168.---.13

Name:    ar---vcenter.net
Address:  192.168.---.5

 

 

Is there anything anyone would like to see in the logs that may give an idea of what could be happening?  Again, I'm quite new to this stuff, so not only would I not know what log file to look inside, but I would have no idea what would be useful within that log file.

[samsheikh]

Should I try and reinstall, and if so, what should information should I provide and at what steps along the way to help figure this out?

[Alex_Romeo]

Hi,

did you register the record in DNS before installing vCenter?

regards,

Alex_Romeo

[samsheikh]

After countless re-installs, I seem to have gotten things to work - for the most part.

I am not sure if this did it, but I had to go in and manually edit the hosts and resolv.conf files, adding static entries for the new vcenter server on the old vcenter server and vice versa.  I also moved the localhost domain name server entry to the bottom of the list.

There are some oddities though.  When I enter the just the hostnames on either of the vCenter servers, there's no resolution, only the FQDN and the ip addresses resolve, even though there is an entry for the hostname in the hosts file (and on the DNS servers).  This is the first issue.

The second issue is that I can only access the VAMI using the FQDN.  When I try accessing it using the IP, I get that dreaded "0 -" on a white page.

[JosephDalpiaz]

I just fixed this issue today and figured I would comment and let everyone know what I did. A reinstall would work, BUT is obviously time consuming. If your vCenter is working as advertised, other than the VAMI, then this might fix your issue. 

First:

Log into your VCSA via ssh. Next, check the directory /var/log/vmware/upgrade for the following files:

-prechecks-succeeded

-exports-succeeded

-import-succeeded

These are 0 KB files, and if they are NOT present, we need to create them using the touch command:

touch prechecks-succeeded

touch export-succeeded

touch import-succeeded

Once the file(s) has been placed, restart the VAMI service using the following command:

/sbin/service vami-lighttp restart

You should now be able to access the VAMI and vCenter should be operating as advertised

 

Hope this helps!

 

Joe

[smarshall]

Thanks much.  This fixed it for me. 

[FrozenPea]

thank you!

[JosephDalpiaz]

Just so you know, the underlying reason the upgrade failed for me was this setting on my vDS,

 

https://kb.vmware.com/s/article/87749?lang=en_US

 

Just because you create those files, doesn't mean your upgrade was successful. To be extra sure I set 'reject mac address change'  to accept and re-ran the upgrade successfully. 

[AndreSwartz]

Thank you, I'm able to sign into VAMI. 

[JordonC]

Thanks  Adding those files fixed one of our broken RBO VCenter servers.  

[md1984]

 I can't find the directory that you are mentioning here. Mine only shows this -

root@vcsa [ /var/log/vmware ]# ls
analytics certificatemanagement deploy infraprofile observability rbd syslog vdtc vmdir vmware-updatemgr vsm
applmgmt cis-license dnsmasq.log journal perfcharts rhttpproxy topologysvc vlcm vmdird vpostgres vsphere-ui
applmgmt-audit cloudvm eam lookupsvc pod rsyslogd trustmanagement vmafdd vmon vpxd vstats
capengine content-library envoy messages procstate sca vapi vmcad vmware-imagebuilder vpxd-svcs vtsdb
certificateauthority decommission hvc netdumper procstate-20231216.gz sso vcha vmcamd vmware-sps vsan-health wcp
root@vcsa [ /var/log/vmware ]#

Would really appreciate if you could guide me on this issue.