VMware Disk Error

[Sys3]

Good morning,

I am a sysadmin, and was asked this afternoon to help with a server in another department. It's been years since I dealt with VMWare or Cisco. Here is a summary:

Problem: VMware (with one Cisco Prime Virtual Machine) appears to be filling up its disk space, thus preventing collection of Cisco's wi-fi usage statistics. Each backup file is 5GB. We currently have two backup files on it. Both have already been FTP'd to a SAN for safekeeping.

We have two spare 1 TB disk drives that we can add. But the VMware software won't permit us to add them.

Attached is a screenshot of CPU, Memory, and Storage for the VMware Host. Using the DellEMC VMware apparently prevents the use of a command line interface? How can I get to a VMware command line?

If I highlight the VM, right-click and choose Edit Settings, then Add hard disk, New standard hard disk, etc. basically I get nowhere.

Configuration:

1) Dell PowerEdge R720

2) Disks:
  a) Two SAS 1 TB 7.2K
  b) Six SAS 146 GB 10K = 876 GB Total

3) Running "DellEMC Customized version of VMware ESXi Image"

4) VMware General Information shows:
  a) Hard disk 1: 300 GB
  b) Hard disk 2: 900 GB

5) One Virtual Machine:
  a) Cisco Prime Infrastructure Virtual Appliance
  b) Guest OS: Red Hat Enterprise Linux 7 (64-bit)
  c) Compatibility: ESXi 5.5 virtual machine
  d) VMware Tools: Yes
  e) CPUs: 16
  f) Memory: 24 GB

6) Inside the Cisco Prime command line interface:
  a) # show disks (I inserted commas for easier reading)
  temp. space 1% used (3,572 of 1,999,184)
  disk: 11% used (17,827,544 of 184,586,292) (This appears to be 184 GB?)
  "Internal filesystems: all internal filesystems have sufficient free space"

  b) # dir
  Usage for disk: filesystem
  18,255,405,056 bytes total used (18 GB used)
  161,156,096,000 bytes free (150 GB free)
  189,016,363,008 bytes available (176 GB available)

Confused about the disk mapping between VMware and Cisco Prime's IOS, and how to get access to control the disks.

Please advise, thanks in advance!

[continuum]

> VMware apparently prevents the use of a command line interface? 
Yes - ssh is disabled by default. Enable ssh and the connect via winscp and putty.

Other than that I dont understand your problem - maybe the screenshots you announced would help ?

 

 

[Sys3]

I appreciate your reply. Attached is the screenshot.

How will I enable SSH?

On the console, here is this info/menu with nothing about SSH.

VMware ESXi 6.7.0 (VMKernel Release Build 17700523)

System Customization
Configure Lockdown Mode
Configure Management Mode
  Hostname
  IPv4 Address
     VLAN, IPv6, DNS
Restart Management Network
Test Management Network
Network Restore Options
Configure Keyboard
View Logs

Thanks.

[continuum]

You should see an option: Configure Troubleshooting options on the yellow/grey screen.
Or just login to the webinterface - there is a clickable button to enable ssh.
https://ip-of-esxi

[Sys3]

You are so right. Somehow I missed the Troubleshooting menu option, just under the Keyboard option.

I enabled both SSH and the ESXi Shell.

My SSH session shows this:
[root@localhost:~] df -h
Filesystem Size Used Available Use% Mounted on
VMFS-5   1.3T   1.2T 72.2G 95% /vmfs/volumes/datastore1
vfat   249.7M 156.5M 93.2M 63% /vmfs/volumes/daa63a70-09bd6106-5409-07c15dcb606c
vfat     4.0G  23.2M  4.0G  1% /vmfs/volumes/60e4b641-43c2bfd0-0506-c81f66e8e651
vfat   249.7M 158.8M 91.0M 64% /vmfs/volumes/0fb72072-8b760ec0-6366-1cc1c391507b
vfat   285.8M 198.0M 87.9M 69% /vmfs/volumes/5d3b4564-555e3970-f7c0-c81f66e8e651
[root@localhost:~]

[root@localhost:~] uname -a
VMkernel localhost 6.7.0 #1 SMP Release build-17700523 Mar 3 2021 16:17:01 x86_64 x86_64 x86_64 ESXi

What do you recommend for the next steps to increase the disk space? The screenshot that I attached to my last reply, and again to this reply, shows storage used as 1.22 TB and total as 1.3 TB, which matches the above df -h result, 

Thank you so much!

[continuum]

You mentioned a disk error ???
Right now it looks to me as if you were just running out of free space ?
Inspect the content with WinSCP - maybe you find a directory that has more "stuff" than you expected ?

Ulli

[Sys3]

You are correct. We have run out of disk space.

The subject was quickly typed. I had posted it several times with a longer, more correct, subject. It kept going to spam, so I tried again with a shorter subject and of course it was accepted.

I'm finding a lot of info about my existing disks, but can't "see" my additional 1T and six 145GB disks.

Identifying disks when working with VMware ESXi (1014953)
https://kb.vmware.com/s/article/1014953

Using partedUtil command line disk partitioning utility on ESXi (1036609)
https://kb.vmware.com/s/article/1036609

Expanding Local Datastore in vSphere ESXi (2002461)
https://kb.vmware.com/s/article/2002461

Can't find the KB article that address my scenario of one 1T disk in use, with another 7 disks plugged into the Dell server but unseen.

Thanks!

[E_Espinel]

Hello.
According to what you indicate in your post, you would have 1.3TB in Storage, which makes me suppose that the scheme of disks is as follows:
two disks (1TB) in RAID 1 and 6 disks (146GB) in RAID 5 or RAID10?

If you use the custom DEll image to install the VMware vSphere (ESXi host), it must have installed the VMware PERCCLI utility from Dell, which is in the following directory opt/lsi/perccli

In a SSH  (putty) session in the ESXi host  go to the indicated directory
# cd /opt/lsi/perccli
and run the following command to verify the disk configuration of the DELL PERC controller
# ./perccli /c0 show

copy the result in the next post you send or in a file and attach the file to the post.

link: http://l4u-00.jinr.ru/pub/misc/h-w/LSI/dell-sas-hba-12gbps_reference-guide_en-us.pdf

As the VM you indicate has 2 disks assigned, please indicate in which Datastore is each disk.

 

 

[Sys3]

[root@localhost:~] cd /opt/lsi/perccli
-sh: cd: can't cd to /opt/lsi/perccli: No such file or directory

[Sys3]

[root@localhost:~] cd /opt/
[root@localhost:/opt] ls -las
total 12
4 drwxr-xr-x 1 root root 512 Jul 11 13:50 .
4 drwxr-xr-x 1 root root 512 Jul 12 17:06 ..
4 drwxr-xr-x 1 root root 512 Jul 11 13:50 vmware

[Sys3]

[root@localhost:~] du -h -d 1
 14.1M ./etc
120.0K ./opt
 17.2M ./var
648.3M ./tardisks
  4.0K ./tardisks.noauto
127.3M ./bin
 20.0K ./tmp
173.6M ./lib64
 40.5M ./lib
281.1M ./usr
128.0K ./proc
 12.0K ./vmimages
  5.1T ./dev
  1.2T ./vmfs
116.0K ./mbr
  6.3T .

[root@localhost:~] df -h
Filesystem Size Used Available Use% Mounted on
VMFS-5 1.3T   1.2T 70.6G 95% /vmfs/volumes/datastore1
vfat 249.7M 156.5M 93.2M 63% /vmfs/volumes/daa63a70-09bd6106-5409-07c15dcb606c
vfat   4.0G  22.8M  4.0G  1% /vmfs/volumes/60e4b641-43c2bfd0-0506-c81f66e8e651
vfat 249.7M 158.8M 91.0M 64% /vmfs/volumes/0fb72072-8b760ec0-6366-1cc1c391507b
vfat 285.8M 198.0M 87.9M 69% /vmfs/volumes/5d3b4564-555e3970-f7c0-c81f66e8e651

[Sys3]

[root@localhost:~] find / -size [phone]c -exec du -h {} \;
103.4M /tardisks/vmx.v00
112.6M /tardisks/vim.v00
188.7M /tardisks/sb.v00
 59.1M /tardisks/s.v00
 43.4M /tardisks/vsan.v00
900.0G /dev/deltadisks/1fbcba-Cisco Prime Infrastructure Virtual Appliance_1-000002-delta.vmdk
300.0G /dev/deltadisks/23bcb9-Cisco Prime Infrastructure Virtual Appliance-000002-delta.vmdk
  1.3T /dev/lvm/5d3b456a-874167c8-e90a-c81f66e8e651
408.4G /dev/disks/naa.6c81f660eafda20024cdc87406346170
408.4G /dev/disks/naa.6c81f660eafda20024cdc87406346170:1
931.0G /dev/disks/naa.6c81f660eafda2001c3d0cf522e651d1
  2.5G /dev/disks/naa.6c81f660eafda2001c3d0cf522e651d1:9
286.0M /dev/disks/naa.6c81f660eafda2001c3d0cf522e651d1:8
110.0M /dev/disks/naa.6c81f660eafda2001c3d0cf522e651d1:7
250.0M /dev/disks/naa.6c81f660eafda2001c3d0cf522e651d1:6
250.0M /dev/disks/naa.6c81f660eafda2001c3d0cf522e651d1:5
923.6G /dev/disks/naa.6c81f660eafda2001c3d0cf522e651d1:3
  4.0G /dev/disks/naa.6c81f660eafda2001c3d0cf522e651d1:2
 44.3M /vmfs/volumes/daa63a70-09bd6106-5409-07c15dcb606c/sb.v00
 42.1M /vmfs/volumes/0fb72072-8b760ec0-6366-1cc1c391507b/sb.v00
 67.1M /vmfs/volumes/0fb72072-8b760ec0-6366-1cc1c391507b/s.v00
 53.5M /vmfs/volumes/5d3b4564-555e3970-f7c0-c81f66e8e651/packages/vmtoolsRepo/vmtools/linux.iso
140.7M /vmfs/volumes/5d3b4564-555e3970-f7c0-c81f66e8e651/packages/vmtoolsRepo/vmtools/windows.iso
255.0M /vmfs/volumes/5d3b456b-a5049938-8733-c81f66e8e651/.fdc.sf
256.0M /vmfs/volumes/5d3b456b-a5049938-8733-c81f66e8e651/.pbc.sf
251.0M /vmfs/volumes/5d3b456b-a5049938-8733-c81f66e8e651/.sbc.sf
300.0G /vmfs/volumes/5d3b456b-a5049938-8733-c81f66e8e651/Cisco Prime Infrastructure Virtual Appliance/Cisco Prime Infrastructure Virtual Appliance-flat.vmdk
900.0G /vmfs/volumes/5d3b456b-a5049938-8733-c81f66e8e651/Cisco Prime Infrastructure Virtual Appliance/Cisco Prime Infrastructure Virtual Appliance_1-flat.vmdk
 14.9G /vmfs/volumes/5d3b456b-a5049938-8733-c81f66e8e651/Cisco Prime Infrastructure Virtual Appliance/Cisco Prime Infrastructure Virtual Appliance-000002-delta.vmdk
 12.0G /vmfs/volumes/5d3b456b-a5049938-8733-c81f66e8e651/Cisco Prime Infrastructure Virtual Appliance/Cisco Prime Infrastructure Virtual Appliance-b53dca09.vswp
 33.2G /vmfs/volumes/5d3b456b-a5049938-8733-c81f66e8e651/Cisco Prime Infrastructure Virtual Appliance/Cisco Prime Infrastructure Virtual Appliance_1-000002-delta.vmdk
110.0M /vmfs/volumes/5d3b456b-a5049938-8733-c81f66e8e651/Cisco Prime Infrastructure Virtual Appliance/vmx-Cisco Prime Infrastructure Virtual Appliance-3040725513-1.vswp

[E_Espinel]

Hello.
Attached is a link where you can get it and the instructions to install the VMware PERCCLI Utility
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=mrxx5

You have to unzip it until you get the .vib file.

 

 

[Sys3]

Will do tomorrow.

Many thanks for everyone's help today.

[Sys3]

Per yesterday's recommendations, I have installed the VMware PERCCLI Utility.

I've been running the various PERCCLI show commands, below is a summary.

To recap, our situation is our VMware host is 94% full as shown in the attached screenshot. It has one Cisco Prime Virtual Machine. We are looking for a way to expand the 1.3 TB of storage onto the six 134 GB disk drives. If that is not possible, we are willing to pull out the six small drives (if possible) and put in two spare 1 TB disk drives for expansion.

How should we proceed?

 

[root@localhost:/opt/lsi/perccli] ./perccli show all
Ctl Model        Ports PDs DGs DNOpt VDs VNOpt BBU sPR DS EHS ASOs Hlth
  0 PERCH710Mini     8   8   2     0   2     0 Opt On  3  N      0 Opt

[root@localhost:/opt/lsi/perccli] ./perccli /c0/v0 show expansion
VD       Size OCE NoArrExp WithArrExp Status
 0 931.000 GB N   -        -          -

[root@localhost:/opt/lsi/perccli] ./perccli /c0/v1 show expansion
VD       Size OCE NoArrExp WithArrExp Status
 1 408.375 GB N   -        -          -

[root@localhost:/opt/lsi/perccli] ./perccli /c0 show all

TOPOLOGY :

------------------------------------------------------------------------------

DG Arr Row EID:Slot DID Type   State BT       Size PDC  PI SED DS3  FSpace TR

------------------------------------------------------------------------------

 0 -   -   -        -   RAID1  Optl  N  931.000 GB dflt N  N   dflt N      N

 0 0   -   -        -   RAID1  Optl  N  931.000 GB dflt N  N   dflt N      N

 0 0   0   32:0     0   DRIVE  Onln  N  931.000 GB dflt N  N   dflt -      N

 0 0   1   32:1     1   DRIVE  Onln  N  931.000 GB dflt N  N   dflt -      N

 1 -   -   -        -   RAID10 Optl  N  408.375 GB dflt N  N   dflt N      N

 1 0   -   -        -   RAID1  Optl  N  136.125 GB dflt N  N   dflt N      N

 1 0   0   32:2     2   DRIVE  Onln  N  136.125 GB dflt N  N   dflt -      N

 1 0   1   32:3     3   DRIVE  Onln  N  136.125 GB dflt N  N   dflt -      N

 1 1   -   -        -   RAID1  Optl  N  136.125 GB dflt N  N   dflt N      N

 1 1   0   32:4     4   DRIVE  Onln  N  136.125 GB dflt N  N   dflt -      N

 1 1   1   32:5     5   DRIVE  Onln  N  136.125 GB dflt N  N   dflt -      N

 1 2   -   -        -   RAID1  Optl  N  136.125 GB dflt N  N   dflt N      N

 1 2   0   32:6     6   DRIVE  Onln  N  136.125 GB dflt N  N   dflt -      N

 1 2   1   32:7     7   DRIVE  Onln  N  136.125 GB dflt N  N   dflt -      N

 

Virtual Drives = 2

VD LIST :

--------------------------------------------------------------------

DG/VD TYPE   State Access Consist Cache Cac sCC       Size Name

--------------------------------------------------------------------

0/0   RAID1  Optl  RW     Yes     RWBD  -   OFF 931.000 GB VMWARE

1/1   RAID10 Optl  RW     Yes     RWBD  -   OFF 408.375 GB Storage2

 

Physical Drives = 8

PD LIST :

------------------------------------------------------------------------------

EID:Slt DID State DG       Size Intf Med SED PI SeSz Model            Sp Type

------------------------------------------------------------------------------

32:0      0 Onln   0 931.000 GB SAS  HDD N   N  512B ST91000640SS     U  -

32:1      1 Onln   0 931.000 GB SAS  HDD N   N  512B ST91000640SS     U  -

32:2      2 Onln   1 136.125 GB SAS  HDD N   N  512B ST9146803SS      U  -

32:3      3 Onln   1 136.125 GB SAS  HDD N   N  512B ST9146803SS      U  -

32:4      4 Onln   1 136.125 GB SAS  HDD N   N  512B ST9146803SS      U  -

32:5      5 Onln   1 136.125 GB SAS  HDD N   N  512B ST9146803SS      U  -

32:6      6 Onln   1 136.125 GB SAS  HDD N   N  512B ST9146803SS      U  -

32:7      7 Onln   1 136.125 GB SAS  HDD N   N  512B ST9146803SS      U  -

------------------------------------------------------------------------------

 

Enclosures = 1

Enclosure LIST :

--------------------------------------------------------------------

EID State Slots PD PS Fans TSs Alms SIM Port# ProdID VendorSpecific

--------------------------------------------------------------------

 32 OK        8  8  0    0   0    0   0 00 x1 BP12G+ 419002U

[E_Espinel]

 

Hello.
According to the description of the Dell PowerEdge R720, it can have up to 16 drive slots on the front.
Your computer could have one 8X2.5" bay or two 8X2.5" bays.
You currently have 8 disks in the first disk bay (0 -7), if you have more slots in the next disk bay, you could install the two 1TB disks and configure a RAID 1 with them.

If you only have one 8X2.5" bay of disks, you would have to upgrade to install the second 8x2.5" bay to install more disks.
With only one 8x2.5" bay your system would look like this

Another option is to get a full backup of the VM, delete the VM from storage, then delete the RAID 10 (146GB disks), remove the 146GB disks and install the two 1TB disks and set up a RAID 1 between them.
Finally recover the VM from the backup.

 

 

 

[Sys3]

I agree with your strategy to backup the VM, rebuild the RAID with new disks, and restore the VM.

How should I back up this VM? After all, it is nearly 1T in size.

Then, how shall I restore the VM?

Thanks!

[E_Espinel]

Hello

I recommend you to backup the VM using the Veeam Backup & Replication Community Edition (Free).

You need a server, VM or PC with at least 8GB of memory and disk space (you can use an external USB disk) to install the Veeam Backup server.

Veeam Backup & Replication Community Edition

https://www.veeam.com/virtual-machine-backup-solution-free.html

Backup VM

https://helpcenter.veeam.com/docs/backup/vsphere/backup.html?ver=110

Restore VM

https://helpcenter.veeam.com/docs/backup/qsg_vsphere/vm_restore.html?ver=110

 

 

 

 

 

[Sys3]

We are going to do this either today or early next week.

I will keep you posted on our progress.

I will select the "Correct Answer" button when this is wrapped up.

Thank you very much, Enrique.