Datastore no longer available

[ylafont22]

 

Good afternoon. Apologies if this I am posting in an incorrect forum (was not really sure of the best place, please move is it incorrect).   I am in middle of a pickle which I need to troubleshoot as I am not an ESXI expert.   Today someone noticed  an issue where a datastore is report 0 size, this has been up for months.   The host now  reports machine are invalid.    

I figured I reboot, but do not think it would solve the issue,   can anyone provide any steps to take and resolve this issue? Find what the problem is?   

The volumen is list with the command esxcli storage filesystem list

and this is how there appear on the host

 

 

Just  update - i did rescan of the datastore, and new "grey out" storage has appeared with a warning on kernel.log

21-05-27T00:23:35.908Z cpu21:2097708)WARNING: NMP: vmk_NmpSatpIssueTUR:1074: Device naa.6141877033256b002644af641f601b56 path vmhba2:C2:T0:L0 has been unmapped from the array

is my datastore gone?????   can i create a new on with UID and find my files?

thank you in advance - assistance is vey much appreciated. 

 

 

[E_Espinel]

Hello.
Have you already checked the Hardware logs or do you have access to the server manager service (type ILO, IMM, Clarity)?

Please run the following commands to see the list of devices and partitions on the ESXi host, paste the result in the post

# esxcli storage core adapter list

# df -h

 

[ylafont22]

e_espinel, thank you for  the assistance.

Server is still up and accesible via the gui and  cli, cant get to anything else.  Info below. 

 

[E_Espinel]

Hello.
Sorry I did not see your update of your first post, it is preferable to make a reply to add information.
Apparently the ESXi host lost a disk or access to it, do you know how many disk and/or volumes it has (internal, external) and what is its configuration (RAID).

Please check if you have the following directory on the ESXi host
/opt/lsi/storcli

If you have it you could enter the directory and run
./storcli show all

Please paste in the post your result

Please indicate what version of vmware you have installed so we can look for the respective storcli to install it.

 

[ylafont22]

No worries, i appreciate the assistance. thank you again.

 

There should  3 Disk ( raid 5)  one volume. 

External Raid.

/opt only has the vmware directory (no lsi)

esxi 6.7.0 (Build 8169922)

 

Please let me know what else can be provided.

[E_Espinel]

Hello.
Download the MegaRAID Storcli Version 007.1211.0000.0000 from the attached link (Management Software and Tools)

https://www.broadcom.com/support/download-search?pa=&pf=RAID+Controller+Cards&pg=Storage+Adapters%2C+Controllers%2C+and+ICs&pn=&po=

You will have to unpack the zip file and in the path 007.1211.0000.0000_Unified_StorCLI>Unified_storcli_all_os>MMwareOP> you will find the vmware-storcli.vib file.

then copy vmware-storcli.vib to ESxi host in some directory (/tmp)

Install the StorCli (vmware-storcli.vib) with the following command:
# esxcli software vib install -v /tmp/vmware-storcli.vib --no-sig-check

You should get something like this (the file is different)

the StorCLI does not require a reboot.

After installing Storcli apply as indicated in the above post

 

 

 

[ylafont22]

Installed successfully, 

Hope this does not matter, notice the number of controllers is "0",  I believe the install control if a Perc H730

esxcli storage core adapter list
HBA Name Driver Link State UID Capabilities Description
-------- -------- ---------- -------------------- ------------ ----------------------------------------------------------
vmhba0 vmw_ahci link-n/a sata.vmhba0 (0000:00:11.4) Intel Corporation Wellsburg AHCI Controller
vmhba1 vmw_ahci link-n/a sata.vmhba1 (0000:00:1f.2) Intel Corporation Wellsburg RAID Controller
vmhba2 lsi_mr3 link-n/a sas.5141877033256b00 (0000:82:00.0) Avago (LSI) PERC H730 Adapter
vmhba32 vmkusb link-n/a usb.vmhba32 () USB

[E_Espinel]

Hello.
For Avago (LSI) PERC H730 adapter, you need the vmware-perccli-007.0127.0000.0000.vib you can get it from the following link

https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=1prwx

You must run it on a PC and in the Dell folder look for the indicated vib.

Then copy it to the ESXi host and install it.
# esxcli software vib install -v /tmp/vmware-perccli-xxx.xxxx.xxxx.xxxx.xxxx.xxxx.vib --no-sig-check

Browse to the /opt/lsi/perccli directory and enter it to execute the following commands

./perccli show

If the controller has ID 0 (as expected) run the command
./perccli /c0 show

 

[ylafont22]

Can we confirm the file download  "perccli_7.1-007.0127_vmw_ZPE.exe"   from https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=1prwx

Get a "Bad VIB archive header" error,  

found these in the esxiupdate.log

2021-05-28T01:59:24Z esxupdate: 6696441: downloader: INFO: Downloading ./vmware-perccli-007.0127.000 to /tmp/vib_2zpwny7z
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: Traceback (most recent call last):
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Downloader.py", line 244, in _getfromurl
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Downloader.py", line 188, in _retry
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Downloader.py", line 208, in _download_to_file
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Downloader.py", line 180, in _urlopen
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/urllib/request.py", line 451, in open
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/urllib/request.py", line 269, in __init__
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/urllib/request.py", line 295, in full_url
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/urllib/request.py", line 324, in _parse
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: ValueError: unknown url type: './vmware-perccli-007.0127.000'
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR:
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: During handling of the above exception, another exception occurred:
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR:
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: Traceback (most recent call last):
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Transaction.py", line 275, in _getVibsFromUrls
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Downloader.py", line 335, in Get
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Downloader.py", line 246, in _getfromurl
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: vmware.esximage.Downloader.DownloaderError: ('./vmware-perccli-007.0127.000', '/tmp/vib_2zpwny7z', "unknown url type: './vmware-perccli-007.0127.000'")
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR:
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: During handling of the above exception, another exception occurred:
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR:
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: Traceback (most recent call last):
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/usr/lib/vmware/esxcli-software", line 475, in <module>
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: main()
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/usr/lib/vmware/esxcli-software", line 466, in main
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: ret = CMDTABLE[command](options)
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/usr/lib/vmware/esxcli-software", line 361, in VibInstallCmd
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: nosigcheck=opts.nosigcheck)
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Transaction.py", line 235, in InstallVibsFromSources
2021-05-28T01:59:24Z esxupdate: 6696441: root: ERROR: File "/build/mts/release/bora-8169922/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Transaction.py", line 165, in GetVibsFromSources

did i miss a step?

 

[E_Espinel]

Hello.
Please try one of the following variants in the command

#esxcli software vib install -v=/tmp/vmware--perccli-007.0127.0000.0000.vib --no-sig-check
or
esxcli software vib install -n=/tmp/vmware--perccli-007.0127.0000.0000.0000.vib --no-sig-check

 

[ylafont22]

No go. 

 

 

[E_Espinel]

Hello

# esxcli software vib install -v "/tmp/vmware-perccli-007.0127.0000.0000.vib"

or

# esxcli software vib install -v file:/tmp/vmware-perccli-007.0127.0000.0000.vib

or

# esxcli software vib install -v file:/tmp/vmware-perccli-007.0127.0000.0000.vib --no-sig-check

 

Run the following command to check for Dell software installed on the ESXi Host

# esxcli software vib list

 

[ylafont22]

no go, on these as well

 

according to documentation the command is correct, all that is required is -v along with --no-sig-check

https://www.dell.com/support/kbdoc/en-us/000177280/how-to-use-the-poweredge-raid-controller-perc-command-line-interface-cli-utility-to-manage-your-raid-controller

 

 

 

[E_Espinel]

Hello.
an important detail, which I overlooked
the install command must be executed from the root (/) not inside /tmp.

 

[ylafont22]

i believe i tried that previously,  but just got the same result "Bad VIB archive header"

 

[E_Espinel]

Hello.
Please delete the file vmware-perccli-007.0127.0000.0000.vib from the tmp directory.
load the file vmware-perccli-007.0127.0027.0000.0000.vib using the Browser Storage of the web client
retry the installation.

 

[E_Espinel]

Hello.
we are going to use another version of Perccli and package
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=5v7xx
find the file vmware-perccli-007.1420.vib and load it on ESXi
run the installation
esxcli software vib install -v /path/xxxxxx.vib (full path must be used for installation)

 

[ylafont22]

Ureka! that one did the trick.

 

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2021.05.28 14:44:46 =~=~=~=~=~=~=~=~=~=~=~=
clasclarear
[root@localhost:/opt/lsi/perccli] [root@localhost:/opt/lsi/perccli] clear [root@localhost:/opt/lsi/perccli] ./perccli /c0 show
Generating detailed summary of the adapter, it may take a while to complete.

CLI Version = 007.1420.0000.0000 Dec 10, 2020
Operating system = VMkernel 6.7.0
Controller = 0
Status = Success
Description = None

Product Name = PERC H730 Adapter
Serial Number = 59500AH
SAS Address = 5141877033256b00
PCI Address = 00:82:00:00
System Time = 05/28/2021 18:45:04
Mfg. Date = 09/08/15
Controller Time = 05/28/2021 18:40:15
FW Package Build = 25.3.0.0016
BIOS Version = 6.23.03.0_4.16.07.00_0x060C0200
FW Version = 4.250.00-4402
Driver Name = lsi_mr3
Driver Version = 7.702.13.00
Current Personality = RAID-Mode
Vendor Id = 0x1000
Device Id = 0x5D
SubVendor Id = 0x1028
SubDevice Id = 0x1F43
Host Interface = PCI-E
Device Interface = SAS-12G
Bus Number = 130
Device Number = 0
Function Number = 0
Domain ID = 0
Security Protocol = None
Drive Groups = 1

TOPOLOGY :
========

-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type State BT Size PDC PI SED DS3 FSpace TR
-----------------------------------------------------------------------------
0 - - - - RAID5 OfLn N 18.190 TB dflt N N dflt N N
0 0 - - - RAID5 Dgrd N 18.190 TB dflt N N dflt N N
0 0 0 :0 0 DRIVE Failed N 9.094 TB dflt N N dflt - N
0 0 1 :1 1 DRIVE Onln N 9.094 TB dflt N N dflt - N
0 0 2 :2 2 DRIVE Failed N 9.094 TB dflt N N dflt - N
-----------------------------------------------------------------------------

DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID
DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Optl=Optimal|Dgrd=Degraded
Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active
PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign
DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present
TR=Transport Ready

Virtual Drives = 1

VD LIST :
=======

--------------------------------------------------------------
DG/VD TYPE State Access Consist Cache Cac sCC Size Name
--------------------------------------------------------------
0/0 RAID5 OfLn RW No NRWTD - OFF 18.190 TB
--------------------------------------------------------------

VD=Virtual Drive| DG=Drive Group|Rec=Recovery
Cac=CacheCade|OfLn=OffLine|Pdgd=Partially Degraded|Dgrd=Degraded
Optl=Optimal|dflt=Default|RO=Read Only|RW=Read Write|HD=Hidden|TRANS=TransportReady|B=Blocked|
Consist=Consistent|R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack|
AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled
Check Consistency

Physical Drives = 3

PD LIST :
=======

---------------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
---------------------------------------------------------------------------------
:0 0 Failed 0 9.094 TB SATA HDD N N 512B ST10000DM0004-2GR11L U -
:1 1 Onln 0 9.094 TB SATA HDD N N 512B ST10000DM0004-2GR11L U -
:2 2 Failed 0 9.094 TB SATA HDD N N 512B ST10000DM0004-1ZC101 U -
---------------------------------------------------------------------------------

EID=Enclosure Device ID|Slt=Slot No.|DID=Device ID|DG=DriveGroup
DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare
UBad=Unconfigured Bad|Sntze=Sanitize|Onln=Online|Offln=Offline|Intf=Interface
Med=Media Type|SED=Self Encryptive Drive|PI=Protection Info
SeSz=Sector Size|Sp=Spun|U=Up|D=Down|T=Transition|F=Foreign
UGUnsp=UGood Unsupported|UGShld=UGood shielded|HSPShld=Hotspare shielded
CFShld=Configured shielded|Cpybck=CopyBack|CBShld=Copyback Shielded
UBUnsp=UBad Unsupported|Rbld=Rebuild

BBU_Info :
========

----------------------------------------------
Model State RetentionTime Temp Mode MfgDate
----------------------------------------------
BBU Optimal 0 hour(s) 58C - 0/00/00
----------------------------------------------

[E_Espinel]

Hello.
In the PERC H730 Adapter we have a RAID 5 with 3 Disks of which 2 are failed.
In general in a RAID 5 with 2 failed disks the data is lost.

You can try to recover the RAID, for which you need 2 good disks to change them in their order of failure.

The disks would be ST10000DM0004 10TB 7200 RPM 256MB Cache SATA 6.0Gb/s 3.5". but you need to physically confirm if you have a p/n for your order.

You need to get the event log to determine the order of failure, usually the first one to fail is the damaged one and the second one to fail was traced by the first one, but it can be good.

The ESXi host is booting from another controller or device.

 

[E_Espinel]

Hello

To get the event log of the controller

perccli /cx show events file=<absolute path>

This command prints the system log to a text file and saves the file in the specified location

[ylafont22]

How can two drive fail at the same time without warning!  I have never seen that in all my time.   Crossing  me fingers  on recovery.   I am attaching log for review and will order drives as soon as you say the word.

 

 

 

 

 

[E_Espinel]

Hello.
The failure of two disks in a RAID 5 is more common than you think.
Attached is the event log analysis and recovery procedure.

I have done my good deed for the day, best of luck.

 

[ylafont22]

Frist, thank you for all the assistance in demystifying what has occurred, I would have never found the files to read the logs for SCSI card. As well as having t dedication and professionalism in providing guidance!  much a apreciaste. 

Drives ordered, however I am doubtful of any success since the reports says the drive failed on two different days.   Had the dates been the same i would be more optimistic.  

We never saw a light indicating the drives had failed or seen a warning form esxi, is there something that needs to be installed or configured for this to in place?  how are the devices monitored form the ESXI server?

lets us pray!

 

[E_Espinel]

Hello
Personally I prefer the physical server to send alerts about problems.
This function of sending alerts via SMTP is very common in the service devices (IMM, ILO, IDRAC, etc.) of most computers.

I would understand that your equipment is DELL, so I attach the following link.
https://techstat.net/how-to-setup-simple-smtp-alerts-for-dells-idrac/

Another option is from vCenter Server, I attach link on this
https://digitalthoughtdisruption.com/2020/03/19/setup-email-alert-for-alarms-in-vcenter-6-7-u3/

 

Finally it is always advisable to have a management network with the service devices (IMM, ILO, IDARC ... etc.) of the equipment. This network allows the verification and monitoring of the equipment, including remote access in many cases and of course the sending of alerts.

 

.

[ylafont22]

Actually, it is supermicro motherboard with the dell SCSI 

let me get through this ordeal,  Drives wont be in for another 4 days or so with the holidays and I may have some time to investigarte your recomendatorios. 

I will post back with  with results!

again i want to you for you knowledge and oversight on this. Really, really, appreciated.

[ylafont22]

Just want to double check there is nothing left to try, even though the drive comes back up.   no data though

 Thank you again for the assistance and guidance. 

[ylafont22]

!SUCCESS! - I don't believe it!  

The report provided showed the first drive failed over an year ago with a recent secondary drive failure,  I fully expected to rebuild the raid after both drives were reported as failed and no data returned when the second drive as re-enabled.    I had also read somewhere that a drive should only be swapped while the machine is on. 

The drive what swapped out while the machine was on.

as soon I accessed the  RAID utility noticed it was rebuilding, since i had already lost all hope, i figured there was no hard in letting it complete the process.   14 hours later, I rebooted  and found the data had been restored.   I can't believe it. 

e_espinel, I could not have done this without you!   if you would not have alerted me to enable the second drive failure,  this would have not turned out this way. Thank you again!

 

 

[E_Espinel]

Hello.
For a technician it is always nice to receive good news.
You always have to have faith and keep hope.

Do not forget to activate the alerts, either from the server or in the vcenter server.