blackout + lost vsphere + vcenter passwords

[lkomarci]

Hi everyone,

could use a little help here. I'm working in a new environment and we just had a blackout. The passwords and ip addresses are on a network drive which I can't access as the domain controllers are offline so I can't access the vcenter or vsphere to boot the virtual machines.

Whan can I do??

Thanks in advance! I'm in quite deep

Luke

[TheBobkin]

Hello Luke,

Are the DCs VMs or physical? Either way, getting them up so you can access everything else seems like the logical first step - if they DCs are VMs do you know what hosts they were registered on? You will require at least root credentials for that/a host to access either the Host Client or SSH to power on the VMs.

Bob

[lkomarci]

Hi Bob,

unfortunately the DCs are VMs, which is why none of the users can connect with their credentials. None of the PCs are getting IP addresses so naturally the DHCP is on one of those virtual DCs. I don't know the vcenter nor the ESX root passwords. Everything here was extremely well planned for disaster events - all IP addresses and credentials are stored on the network which I naturally can't access. I do know the ESX ip addresses though.

You have any ideas?

Cheers,

Luke

[TheBobkin]

Hello Luke,

"I don't know the vcenter nor the ESX root passwords."

This is the major hurdle here - if you had these it would just be a case of Host Client/SSH and powering stuff up - does anyone else in your organisation have these and/or might they be stored somewhere you can access?

If this is not possible then the only feasible way to change the root password of an ESXi host is to re-install ESXi - if you can re-install a host and configure access to the required datastores then you can register and power on the VMs required to access everything else.

"Everything here was extremely well planned for disaster events"

I wouldn't really be saying that at this point as you appear to be locked out of everything :smileyconfused:

Bob

[lkomarci]

Hello Bob,

the claim that of being very well prepared for disaster was a sarcasm, imagine my shock.

Just read VMware Knowledge Base  it's supposed to be possible to reset the ESX password but haven't ever done this - is it a safe procedure?

thanks,

L

[StephenMoll]

I imagine you are using a fairly recent version of ESXi, v6? From your linked KB:

ESXi 3.5, ESXi 4.x, ESXi 5.x and ESXi 6.x Reinstalling the ESXi host is the only supported way to reset a password on ESXi.

I would think doing this without proper access to the network resources required would be very difficult. I think exhausting all avenues to finding someone with the required passwords would be the best approach at the moment.

[lkomarci]

Oh... damn. Missed that part.

But yes, it's the ESXi 6.5.0. Bad news

[TheBobkin]

Hello Luke,

Explore every angle while trying to work out whom/where root passwords may reside - even just one ESXi root with the necessary datastores attached should be sufficient. If not then start working out what you need to be able to re-install a host (or install/use a spare one you have on hand) and connect it to the necessary storage e.g. are these iSCSI or FC which will require zoning or NFS which will require permissions. You mentioned "- all IP addresses and credentials are stored on the network which I naturally can't access"  - Do you mean that everything is stored somewhere you cannot access over the network (and thus inaccessible with no network)? Or is this stored on a site that you can physically access?

Bob

[a_p_]

all IP addresses and credentials are stored on the network

Just a thought. Are you able to access the backup? Maybe you can restore the documents with the credentials from there!?

André

[lkomarci]

Hi André,

not sure how to be exact. Network resourceaccess is managed by AD/group policy rules and considering that the DCs are all offline I'm not sure how to be exact.

Another thing that's bugging me is reinstalling the EXSi. Normally for reinstallation I'd use vsphere - insert the ESX OS image into the virtual drive and then upon reboot I'd be able to install it. But considering that I can't access vsphere or anything else for that matter

[lkomarci]

K guys,

I have the ESX root password!

Now, I can't ping the ESX so I could try to connect via vsphere. I tried giving my PC an IP address from the ESX's subnet but won't go. What can I do?

cheers!

[a_p_]

I tried giving my PC an IP address from the ESX's subnet but won't go. What can I do?

Can you confirm that your PC is connected to a physical port in the proper VLAN?

Are you able to ping out (e.g. the gateway address) from the host's DCUI (console)?

André

[lkomarci]

I'm thinking about putting a laptop on the ESX's subnet and plug it directly into the back of the ESX to figure out which one is the management port but all of the ports are taken. Two ports are on fiber and two on patch

[a_p_]

... and plug it directly into the back of the ESX ...

you may use the DCUI to find out the correct port(s). However, a direct connection may be tricky, especially if the ESXi host's Management network uses VLAN tagging, and if it has more that one vmnic. Anyway, assuming that the physical network (switches, routers, etc.) are up, and working correctly, and if the ESXi host can reach/ping its default gateway, you should be able connect to it over the network.

Another option is to enable ESXi shell access and power on the important systems (e.g. your DC/DNS/DHCP server) from the command line.

André

[lkomarci]

DCUI reached, I'll try to fire up the VMs now.

thanks André I really appreciate your help!

[StephenMoll]

You know how to get to the console from the DCUI?

If not this might help : Using ESXi Shell in ESXi 5.x and 6.x (2004746)

Once in there:

vim-cmd vmsvc/getallvms to list all VMs registered to the host.

You need the VMID from this list of any VMs you want to start.

vim-cmd vmsvc/power.on {VMID}

To power on a VM.

With any luck you'll find the host with one of your DCs and that will allow you get everything back up more easily.

[srwsol]

Wow.   This is a text book example of why one should never have anything that ESXi is dependent upon running as a virtual machine under ESXi.  

[StephenMoll]

Let me correct that for you...

"This is a text book example of why one should never have anything that ESXi is dependent upon running as a virtual machine under ESXi without a clearly defined process for it running when things go wrong."

:smileywink:

[lkomarci]

I couldn't agree more with you guys.

I'm going to think about setting the core switch as a DHCP server for certain VLANs - in case if it happens again everything should be reachable without major complications.

[lkomarci]

Hi Stephen,

yea I managed to find those commands. This is exactly what I've used.

For now as a temporary solution I've set the DC VMs to start automatically.

Cheers,

Luka

[StephenMoll]

So these are standalone hosts then, not clustered?

[lkomarci]

Yes they're standalone, no HA here.

But I'm not really an expert in this topic, the IT system is just one of my many duties here. What do you propose as an optimal solution? There are 2 ESXi hosts here, for HA 3 are recommended and I'm not sure yet how well this additional cost would be accepted but nevertheless I'd like to find a more viable solution for the current hardware.

[StephenMoll]

If your key services are resilient through other strategies, like Windows Server Failover Clustering (WSFC), then there probably isn't much point in going for HA at the moment until you can bring another host into the equation.

[lkomarci]

Now I'm by far no expert in this field but - would you prefer WSFC over vmware fault tolerance? Is it possible to engage with any of these on a running system, i.e. without having to lose any data?

cheers

[StephenMoll]

Often you have no choice. If you are installing and relying on a 3rd party solution that is built on WSFC, then often that is what you will use. Building these into a vSphere environment is pretty well understood nowadays. Fault Tolerance was considered for some portions of our system, but it is very costly in terms of resource needs, e.g. dedicated 10gbps network for FT network traffic. Also there are limits on how many FT VMs you can have. We have dozens of WSFC components, it would not be practical to run them all as FT VMs.