VSAN disk group creation failed

[Silvester]

Dear all,

I've created a new cluster and enabled VSAN on it, with 3 hosts. Everything looks fine so far,

However when I select 1 SSD and 4 HDD from one of the hosts and create a disk group, the process failed with the following message:

Operation failed, diagnostics report: Disk with uuid 52b244d6-7cda-613f-46b8-eb786e6207e5 didn't appear in the vsan cluster directory within 10 seconds.

What's interesting is that, there not such a UUID existing on that host.

I've tried disabled/enabled VSAN on the cluster, and create disk group from other hosts within the cluster, with the same result.

Any idea on how to continue the troubleshooting?

Thanks!

Silvester

[Badarinadh Vissamsetti]

Please post the output of vdq -i, vdq -q and esxcfg-scsidevs -l

And what is the network speed of vmkernel nics that you used for vSAN..? Virtual SAN requires a private 1Gb network. As a best practice, use 10Gb network.

[Silvester]

I'm using the 10Gb network for the vSAN kernrnel.

The output of vdq -i:

[

   {

      "SSD" : "naa.600605b005a783701acbdef240ab6fb8",

       "MD" : [

              "naa.600605b008d44de01b7ea39bda5e9bc3",

              "naa.600605b008d44de01b7ea6d30b7c9b9d",

              "naa.600605b008d44de01b7e9e25870f7118",

              "naa.600605b008d44de01b7eaacb4807dcc5",

              ]

   },

]

Output of vdq -q:

[
   {
      "Name"     : "naa.600605b005a783701acbdef240ab6fb8",
      "VSANUUID" : "52ec0da9-7e39-8749-b729-62a01411c931",
      "State"    : "In-use for VSAN",
      "Reason"   : "None",
      "IsSSD"    : "1",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7ea39bda5e9bc3",
      "VSANUUID" : "52756247-baf6-c8a1-10f3-2ea0d564e972",
      "State"    : "In-use for VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7ead416d91ca0f",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "1",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7ea6d30b7c9b9d",
      "VSANUUID" : "52b244d6-7cda-613f-46b8-eb786e6207e5",
      "State"    : "In-use for VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7e9e25870f7118",
      "VSANUUID" : "52416344-3051-95a0-cbd2-884442f5d0d0",
      "State"    : "In-use for VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "mpx.vmhba32:C0:T0:L0",
      "VSANUUID" : "",
      "State"    : "Ineligible for use by VSAN",
      "Reason"   : "Has partitions",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7eaacb4807dcc5",
      "VSANUUID" : "52098586-205a-0fe0-f6ce-8fb922daf058",
      "State"    : "In-use for VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7e8a995cc62833",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7ea64903437c1c",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7ea89f26e5579e",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7e893b47e3b64f",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7ea80d1e364892",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7eab464f5a234b",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

]

Output of esxcfg-scsidevs

mpx.vmhba0:C0:T19:L0

   Device Type: Enclosure Svc Dev

   Size: 0 MB

   Display Name: Local CISCO Enclosure Svc Dev (mpx.vmhba0:C0:T19:L0)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/genscsi/mpx.vmhba0:C0:T19:L0

   Devfs Path: /vmfs/devices/genscsi/mpx.vmhba0:C0:T19:L0

   Vendor: CISCO     Model: UCS 240           Revis: 0809

   SCSI Level: 5  Is Pseudo: false Status: on      

   Is RDM Capable: false Is Removable: false

   Is Local: true  Is SSD: false

   Other Names:

      vml.020d000000f40f1b1e5d2600bd554353203234

   VAAI Status: unsupported

mpx.vmhba32:C0:T0:L0

   Device Type: Direct-Access

   Size: 14336 MB

   Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0

   Devfs Path: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0

   Vendor: HV        Model: Hypervisor_0      Revis: 1.01

   SCSI Level: 2  Is Pseudo: false Status: on      

   Is RDM Capable: false Is Removable: true

   Is Local: true  Is SSD: false

   Other Names:

      vml.0000000000766d68626133323a303a30

   VAAI Status: unsupported

naa.600605b005a783701acbdef240ab6fb8

   Device Type: Direct-Access

   Size: 189781 MB

   Display Name: Local LSI Disk (naa.600605b005a783701acbdef240ab6fb8)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/naa.600605b005a783701acbdef240ab6fb8

   Devfs Path: /vmfs/devices/disks/naa.600605b005a783701acbdef240ab6fb8

   Vendor: LSI       Model: MR9271-8i         Revis: 3.40

   SCSI Level: 5  Is Pseudo: false Status: on      

   Is RDM Capable: false Is Removable: false

   Is Local: true  Is SSD: true

   Other Names:

      vml.0200000000600605b005a783701acbdef240ab6fb84d5239323731

   VAAI Status: unsupported

naa.600605b008d44de01b7e893b47e3b64f

   Device Type: Direct-Access

   Size: 857353 MB

   Display Name: Local LSI Disk (naa.600605b008d44de01b7e893b47e3b64f)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7e893b47e3b64f

   Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7e893b47e3b64f

   Vendor: LSI       Model: MR9271-8i         Revis: 3.40

   SCSI Level: 5  Is Pseudo: false Status: on      

   Is RDM Capable: false Is Removable: false

   Is Local: true  Is SSD: false

   Other Names:

      vml.0200000000600605b008d44de01b7e893b47e3b64f4d5239323731

   VAAI Status: unsupported

naa.600605b008d44de01b7e8a995cc62833

   Device Type: Direct-Access

   Size: 857353 MB

   Display Name: Local LSI Disk (naa.600605b008d44de01b7e8a995cc62833)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7e8a995cc62833

   Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7e8a995cc62833

   Vendor: LSI       Model: MR9271-8i         Revis: 3.40

   SCSI Level: 5  Is Pseudo: false Status: on      

   Is RDM Capable: false Is Removable: false

   Is Local: true  Is SSD: false

   Other Names:

      vml.0200000000600605b008d44de01b7e8a995cc628334d5239323731

   VAAI Status: unsupported

naa.600605b008d44de01b7e9e25870f7118

   Device Type: Direct-Access

   Size: 857353 MB

   Display Name: Local LSI Disk (naa.600605b008d44de01b7e9e25870f7118)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7e9e25870f7118

   Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7e9e25870f7118

   Vendor: LSI       Model: MR9271-8i         Revis: 3.40

   SCSI Level: 5  Is Pseudo: false Status: on      

   Is RDM Capable: false Is Removable: false

   Is Local: true  Is SSD: false

   Other Names:

      vml.0200000000600605b008d44de01b7e9e25870f71184d5239323731

   VAAI Status: unsupported

naa.600605b008d44de01b7ea39bda5e9bc3

   Device Type: Direct-Access

   Size: 857353 MB

   Display Name: Local LSI Disk (naa.600605b008d44de01b7ea39bda5e9bc3)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ea39bda5e9bc3

   Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ea39bda5e9bc3

   Vendor: LSI       Model: MR9271-8i         Revis: 3.40

   SCSI Level: 5  Is Pseudo: false Status: on      

   Is RDM Capable: false Is Removable: false

   Is Local: true  Is SSD: false

   Other Names:

      vml.0200000000600605b008d44de01b7ea39bda5e9bc34d5239323731

   VAAI Status: unsupported

naa.600605b008d44de01b7ea64903437c1c

   Device Type: Direct-Access

   Size: 857353 MB

   Display Name: Local LSI Disk (naa.600605b008d44de01b7ea64903437c1c)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ea64903437c1c

   Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ea64903437c1c

   Vendor: LSI       Model: MR9271-8i         Revis: 3.40

   SCSI Level: 5  Is Pseudo: false Status: on      

   Is RDM Capable: false Is Removable: false

   Is Local: true  Is SSD: false

   Other Names:

      vml.0200000000600605b008d44de01b7ea64903437c1c4d5239323731

   VAAI Status: unsupported

naa.600605b008d44de01b7ea6d30b7c9b9d

   Device Type: Direct-Access

   Size: 857353 MB

   Display Name: Local LSI Disk (naa.600605b008d44de01b7ea6d30b7c9b9d)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ea6d30b7c9b9d

   Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ea6d30b7c9b9d

   Vendor: LSI       Model: MR9271-8i         Revis: 3.40

   SCSI Level: 5  Is Pseudo: false Status: on      

   Is RDM Capable: false Is Removable: false

   Is Local: true  Is SSD: false

   Other Names:

      vml.0200000000600605b008d44de01b7ea6d30b7c9b9d4d5239323731

   VAAI Status: unsupported

naa.600605b008d44de01b7ea80d1e364892

   Device Type: Direct-Access

   Size: 857353 MB

   Display Name: Local LSI Disk (naa.600605b008d44de01b7ea80d1e364892)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ea80d1e364892

   Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ea80d1e364892

   Vendor: LSI       Model: MR9271-8i         Revis: 3.40

   SCSI Level: 5  Is Pseudo: false Status: on      

   Is RDM Capable: false Is Removable: false

   Is Local: true  Is SSD: false

   Other Names:

      vml.0200000000600605b008d44de01b7ea80d1e3648924d5239323731

   VAAI Status: unsupported

naa.600605b008d44de01b7ea89f26e5579e

   Device Type: Direct-Access

   Size: 857353 MB

   Display Name: Local LSI Disk (naa.600605b008d44de01b7ea89f26e5579e)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ea89f26e5579e

   Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ea89f26e5579e

   Vendor: LSI       Model: MR9271-8i         Revis: 3.40

   SCSI Level: 5  Is Pseudo: false Status: on      

   Is RDM Capable: false Is Removable: false

   Is Local: true  Is SSD: false

   Other Names:

      vml.0200000000600605b008d44de01b7ea89f26e5579e4d5239323731

   VAAI Status: unsupported

naa.600605b008d44de01b7eaacb4807dcc5

   Device Type: Direct-Access

   Size: 857353 MB

   Display Name: Local LSI Disk (naa.600605b008d44de01b7eaacb4807dcc5)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7eaacb4807dcc5

   Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7eaacb4807dcc5

   Vendor: LSI       Model: MR9271-8i         Revis: 3.40

   SCSI Level: 5  Is Pseudo: false Status: on      

   Is RDM Capable: false Is Removable: false

   Is Local: true  Is SSD: false

   Other Names:

      vml.0200000000600605b008d44de01b7eaacb4807dcc54d5239323731

   VAAI Status: unsupported

naa.600605b008d44de01b7eab464f5a234b

   Device Type: Direct-Access

   Size: 857353 MB

   Display Name: Local LSI Disk (naa.600605b008d44de01b7eab464f5a234b)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7eab464f5a234b

   Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7eab464f5a234b

   Vendor: LSI       Model: MR9271-8i         Revis: 3.40

   SCSI Level: 5  Is Pseudo: false Status: on      

   Is RDM Capable: false Is Removable: false

   Is Local: true  Is SSD: false

   Other Names:

      vml.0200000000600605b008d44de01b7eab464f5a234b4d5239323731

   VAAI Status: unsupported

naa.600605b008d44de01b7ead416d91ca0f

   Device Type: Direct-Access

   Size: 189781 MB

   Display Name: Local LSI Disk (naa.600605b008d44de01b7ead416d91ca0f)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ead416d91ca0f

   Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ead416d91ca0f

   Vendor: LSI       Model: MR9271-8i         Revis: 3.40

   SCSI Level: 5  Is Pseudo: false Status: on      

   Is RDM Capable: false Is Removable: false

   Is Local: true  Is SSD: true

   Other Names:

      vml.0200000000600605b008d44de01b7ead416d91ca0f4d5239323731

   VAAI Status: unsupported

Thanks a lot!

Silvester

[Badarinadh Vissamsetti]

If you see the output of vdq -q you can actually find that VSAN UUID 52b244d6-7cda-613f-46b8-eb786e6207e5, which you mentioned you couldn't find. That is attached to the disk naa.600605b008d44de01b7ea6d30b7c9b9d. So there could be some problem with that disk. To confirm that, do not select this data disk while disk group creation. If that works, then you can be sure there is some problem with that disk.

surprising factor is that, vdq output shows that VSAN cluster creation is successful. You can observe that some of the disk state shows that In-Use for VSAN. Can you also post the output for "esxcli vsan cluster get"

[Silvester]

OK, I managed to delete the disk group, and tries to create another disk group, using some other disks than "600605b008d44de01b7ea6d30b7c9b9d", now this time it fails with the following message

"Operation failed, diagnostics report: Sysinfo error on operation returned status : Not found. Please see the VMkernel log for detailed error information"

And I notice this time no disk group is created (vdq -i returns nothing).

Silvester

[Paudie O'Riordan]

Be nice to see the vmkernel logs..

Can you post?

a vm-support would be great

Thanks

[Silvester]

Hi Paudieo

Here is the VMKernel/vmkwarning/vmksummary log files etracted from the ESX host. I reproduced the error twice before downloading the logs.

With my knowledge I was not able to identify anything abnormal. Any hint/suggestions will be much apprecated!

If you need other log files, please feel free to let me know.

Silvester

[Paudie O'Riordan]

Hi Silvester,

the vmkernel logs have some data, but maybe out of context, it would it really be great if you could send me a full vm-support

I need to check out some more details regarding the disk naa =  naa.600605b005a783701acbdef240ab6fb8

there are a few events that I need to check out

vm-support not only collects log files , but also some configuration data that would be helpful

Appreciate it !!

[Paudie O'Riordan]

Also there is some vsan specific log files that would be included on a vm-support

this would narrow the problem considerably

[Silvester]

Here it is the vm-support output.

BTW, the disk ID you mention is the SSD I create the disk group with.

Thanks!

Silvester

[Silvester]

Hi Paudeio,

If you have reviewed my vm-support log files, would you kindly suggest how I shall proceed?

Thanks!

Silvester