Failed to expand VMFS datastore... - Cannot change the host configuration

[Zippyh]

Dear reader

I am a sys-admin with 'some' VMware knowledge. Our IT partner helped us increase the server capacity with two additional drives.

First drive was added to the raid volume, the second one is branded as hot spare when one crashes.

The expanding via the ESX of the raidvolume worked perfectly. In Vcenter I can even see that VMware finds that the volume has increased.

Now, the following problem arises :

When trying to expand the existing Datastore:

- via the Vcenter directly it does not show any disk/LUN volume and you cannot continue.

- via the direct connection to the server:

     * it lets me use <Expand an existing VMFS datastore extent>

     * I can select <Local LSI Disk (naa...) Disk 3.27TB...

     * it shows me <Select partitioning options> where it shows me the 3.VMFS in blue and <Free space> 837.84GB in green.

     * when selecting the blue VMFS, you can see on the <After> that blue and green merges

     * when trying to finish it ends in the error "Cannot change the host configuration".

I have found that the GPT table was not as should be on the disk and I fixed this. Sadly enough that did not fix it.

I have checked the permissions for the logins I am using, being root & administrator, both have rights, doesn't change anything.

Going via Vcenter doesn't work, going via Server ends in error.

My IT Partner doesn't know the answer to this either and now suggest to update the hardware raidcontroller's drivers etc.

Vcenter: Version information: Version 6.5.0 Build 7515524

Selected servers (2 - Live & Backupserver (identical- Fujitsu RX2520 M1)) to upgrade: Hypervisor: VMware ESXi 6.5.0, 5969303

PS: This morning I saw the following under <Configure>

Local USB Direct-Access (mpx.vmhba32...) disk 3,73GB Attached HDD Block Adapter
Local LSI Disk (naa.6003...) disk 3,277TB HDD SAS

I personally don't know where to look anymore, any help would be welcome =)

[Wonlliv]

Hi Zippyh,

I guess you'll need to do it using the command line:

Expanding the VMFS dataastore on the disk/LUN needs to be done from the command line.

see https://kb.vmware.com/s/article/2002461

[Zippyh]

Can you explain me what the reason is? As you have the full option list via the web interface, until you click finish and it says cannot change configuration?

It's rather silly to have the expand option etc via the web, when it is in fact unusable.

As I personally have never done this, it's quite scary to go do this in a live environment.

[Zippyh]

Follow up:

So I have fixed the GPT Table and extended it to fit the new RAID volume. Sadly enough the problem persists.

As the external IT firm did not have any expierence with the CLI interface they set up a test environment for expanding it.

Something goes wrong there and the entire datastore just vanishes in a couple of minutes and cannot be restored.

Ofcourse I do not want this on my replication server, hell to the no on my live server..

Any ideas?

Thanks!

[a_p_]

Welcome to the Community,

In case that the datastore resides on the same volume as the ESXi installation, then the only way to increase the datastore is to do this from the command line. The Embedded Host Client unfortunately lets you go through the expand storage wizard in such a case, which is clearly a bug.

Increasing the datastore from the command line - as explained in the KB Article that has been posted before - will work, if you follow the steps in the exact order.

Note: In step 7, use the "partedUtil getUsableSectors ..." command to determine the last usable sector.

If you are unsure, or have questions, please ask before you try anything!

If you already did something on the production system, please explain exactly what you did so far.

André

[Zippyh]

Hey André

Thank you for your additional info, I'll take this into account and discuss it with the external IT firm which has to finish their task.

So far I have only fixed the GPT table, after going via SSH and checking everything it gave me the following message:

"

1. The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller.

"

That fix worked perfectly. But as you say, it does appear to be a bug that you can expand the datastore via the Vcenter GUI.

[a_p_]

If you are willing to share details about the current disk/partition layout, I will provide you with the commands that are supposed to do the job.

What I need is the output of the following commands:

vmkfstools -P "/vmfs/volumes/DatastoreName"

partedUtil getptbl "/vmfs/devices/disks/DeviceName"

partedUtil getUsableSectors "/vmfs/devices/disks/DeviceName"

If you want, you can provide this information for both, the production system as well as for the test system, so that you can try this ion the test system prior to do it in production.

In any case, make sure - unless you already did - that you have a backup of the VMs, just in case that something unexpected happens.

André

[a_p_]

Please provide the output for the commands that I've posted.

Note that for the first command, it's the datastore name that's required (e.g. "Datestore1").

For the other two commands it's the device name (i.e. the one without the partition ":3")

André

[Zippyh]

Hey André

Thank you for the fast reply, sorry about the confusion.

[a_p_]

Below are the steps to increase the datastore from ~2.5TB to ~3.2TB with the information that you've provided.

I'm referring to the steps in the KB article https://kb.vmware.com/s/article/2002461

With the information that are already available, we can skip to step 8.

Note: Although the steps below should work, I strongly recommend that you ensure that you have a recent backup! Just in case something unexpected happens.

Step 8:

Increase the VMFS partition: Partition 3, Starting sector remains as is, End sector equals the result from the "partedUtil getUsableSectors" command.

partedUtil resize "/vmfs/devices/disks/naa.6003005701f6b2701cd9f79d2890a47f" 3 13631488 7028342750

Step 9: (optional in this case since you already did this)

Fix the GPT partition information

partedUtil fixGpt "/vmfs/devices/disks/naa.6003005701f6b2701cd9f79d2890a47f"

Step 10: (only in case of a warning in step 8)

If you receive the warning message that's shown in step 10 in the KB article, you'll need to reboot the host at this point!

At this point, the command

partedUtil getptbl "/vmfs/devices/disks/naa.6003005701f6b2701cd9f79d2890a47f"

should show the resized vmfs partition information

Step 12: (do a refresh)

Run the command: vmkfstools -V

Step 13: (increase the VMFS file system within the resized partition)

vmkfstools --growfs "/vmfs/devices/disks/naa.6003005701f6b2701cd9f79d2890a47f:3" "/vmfs/devices/disks/naa.6003005701f6b2701cd9f79d2890a47f:3"

Step 14:

Do a Refresh, and/or Rescan from the GUI, and you should see the increase datastore size.

Please note that the forum might format the above text with line breaks. Each of the commands however have to be issued in a single line!

André

[Zippyh]

Dear André

I'm finalizing the steps with our IT partner to make progress on this.

I am trying to learn all the items on the spot, another question.

So we added two HDD to the RAID volume, 1 added, 1 as backup when another breaks.

The picture is what shows on my Backup server.

The current VMFS is 2,45TB with a free space of 522.72GB (VMFS5)

Am I correct to asume that the <Free space (837.84GB) is the recently added space via the new HDD?

And this is the part not yet available to the current VMFS/Server but will get added via the CLI.

I just want to make sure on everything.

Thanks for your help in advance

PS: We are simply trying to add green to blue.

It is really weird that you have "so many options" in VMware ESXi webmodule to expand the VMFS but none of them are working and you have to go CLI.

(I have to explain this to management.)

I mean, why does this even exist?

[a_p_]

Am I correct to asume that the <Free space (837.84GB) is the recently added space via the new HDD?

And this is the part not yet available to the current VMFS/Server but will get added via the CLI.

That's correct, what we are going to do is to expand the existing VMFS partition (blue) with the adjacent free disk space (green).

Btw, don't get confused, the "Partitions" pane in the first screenshot is buggy, and doesn't show the correct state. The colored "Partition diagram" is correct. This is a bug in an older version of the Embedded Host Client, and has been fixed in a later version (see e.g. ESXi Embedded Host Client).

I mean, why does this even exist?

Simply answer: It's a bug.

Not sure why VMware doesn't support the expansion of VMFS partitions if a volume/disk contains other partitions too. Expanding VMFS datastores from the GUI only works if the VMFS datastore is the only partition on the volume.

André

[Zippyh]

Okey, all clear.

Let's talk about when this goes wrong. Which I hope it won't.

We have a disaster recovery server in another neighboring country. But let's not talk about that.

Let's say the datastore partitions go bad, what are the options to recover from it/ repair it, without backup?

[a_p_]

It's always good to do things with care, but you shouldn't worry too much. I've done this several times, and never had any issues with it.

If something goes wrong, it's basically only the partition table which needs to be fixed again. Data on the disk won't be affected unless someone enters commands, which decrease the partition size.

André

[Zippyh]

Following what was described in the article and what André told me fluently solved the problem in less than 5min without any hiccups!

Many thanks.

[geogherkins]

After wasting several hours of frustrating inability to increase capacity (expand) datastore volume - I finally noticed other quirkiness regarding the ESXi Hypervisor (FREE) web client... and realized I was using the unsupported Firefox browser!  Egad! I opened up Chrome and all is well!  So if anyone finds themselves searching for an answer to why they can't increase the capacity of their datastore, please give Chrome a try. Or clear your cache and restart your browser.

For posterity, here are two screenshots taken from Firefox. The GUI won't let me edit the increased partition size!

[Matthew_70]

Exactly the same problem Ihad - thank you so much for the hint/solution.

[EbsGlobal_IT]

If even today, some unfortunate person like me has run into this problem, know that there is a simpler solution without resorting to the CLI. Just don't expand the partition completely but also leave 150MB free at the end, just leave a green stripe at the end and magically it works!

[sobuild]

Yes, it work!!!

[zliling]

The scaler button is wrong, it attempts to expand beyond the partition size. Reducing even with 1 (!) blocks will do the trick.