VMware vSphere

 View Only
Back to discussions
Expand all | Collapse all

Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

  • 1.  Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Apr 22, 2021 03:37 PM

    Hi all,

    after moving from ESXI 6.7 to 7.0.2 I find very slw Server boot, hanging on "vmw_satp_alua loaded successfully" start message.

    Following attached the vmkernel.log file.

    I don't have any Microsoft Cluster in my VMs, I could read it's one of the slowing factors.

     

    Do you have any idea what could be to reduce the boot performance?

    2021-04-22T14:04:11.751Z cpu26:2098301)LVM: 17286: File system '[QNAP VM1 9TB SSD(sata) 1, 5f4e5ddc-6551a6ac-d144-38eaa7318da0]' (LV 5f4e5ddc-3585623c-ba1e-38eaa7318da0) mounted in 'rw' mode.
    2021-04-22T14:04:11.777Z cpu26:2098301)NFS: 1342: Invalid volume UUID 5f733e7c-79ef862a-fb8f-1402ec77987c
    2021-04-22T14:04:11.824Z cpu26:2098301)LVM: 17286: File system '[QNAP VM2 SSD 9TB 1, 5f733e7c-a02e549c-d824-1402ec77987c]' (LV 5f733e7c-79ef862a-fb8f-1402ec77987c) mounted in 'rw' mode.
    2021-04-22T14:04:11.849Z cpu26:2098301)NFS: 1342: Invalid volume UUID 5f733f08-e31b0702-91d0-1402ec77987c
    2021-04-22T14:04:11.897Z cpu26:2098301)LVM: 17286: File system '[QNAP VM2 16TB HDD 1, 5f733f08-0cf7805c-3f08-1402ec77987c]' (LV 5f733f08-e31b0702-91d0-1402ec77987c) mounted in 'rw' mode.
    2021-04-22T14:04:11.927Z cpu26:2098301)NFS: 1342: Invalid volume UUID 602d44c8-c111e63e-f14c-1402ec779b60
    2021-04-22T14:04:11.982Z cpu26:2098301)LVM: 17286: File system '[QNAP VM3 18TB HDD1, 602d44c8-ce9f4e92-b84c-1402ec779b60]' (LV 602d44c8-c111e63e-f14c-1402ec779b60) mounted in 'rw' mode.
    2021-04-22T14:04:12.011Z cpu26:2098301)NFS: 1342: Invalid volume UUID 602d4507-f5746cd6-c0c0-1402ec779b60
    2021-04-22T14:04:12.064Z cpu26:2098301)CpuSched: 817: user latency of 2098311 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T14:04:12.065Z cpu26:2098301)LVM: 17286: File system '[QNAP VM3 10TB SSD 1, 602d4507-013b61fe-3c74-1402ec779b60]' (LV 602d4507-f5746cd6-c0c0-1402ec779b60) mounted in 'rw' mode.
    2021-04-22T14:04:12.079Z cpu26:2098301)NFS: 1342: Invalid volume UUID 606db675-f3b1d5b8-50f9-38eaa79007d8
    2021-04-22T14:04:12.111Z cpu26:2098301)LVM: 17286: File system '[OSDATA-606db676-0aa7fb72-c417-38eaa79007d8, 606db676-0aa7fb72-c417-38eaa79007d8]' (LV 606db675-f3b1d5b8-50f9-38eaa79007d8) mounted in 'rw' mode.
    2021-04-22T14:04:53.399Z cpu6:2097802)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device naa.6e843b640b4b3b4de9fed4402da808da repeated 3 times
    2021-04-22T14:04:53.399Z cpu6:2097802)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device naa.6e843b6e311bd35dbb1fd428edaefddc repeated 3 times
    2021-04-22T14:04:53.399Z cpu6:2097802)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device naa.6e843b61523a9a0dc05dd4b8bd8b92d7 repeated 3 times
    2021-04-22T14:04:53.399Z cpu6:2097802)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device naa.6e843b62aa61ca4dcad9d4c40da5e2d7 repeated 2 times
    2021-04-22T14:07:42.090Z cpu19:2098301)CpuSched: 817: user latency of 2098412 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T14:11:14.117Z cpu17:2098301)CpuSched: 817: user latency of 2098512 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T14:14:40.794Z cpu8:2098301)CpuSched: 817: user latency of 2098612 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T14:18:12.234Z cpu23:2098301)CpuSched: 817: user latency of 2098712 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T14:21:44.143Z cpu8:2098301)CpuSched: 817: user latency of 2098812 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T14:25:16.100Z cpu21:2098301)CpuSched: 817: user latency of 2098912 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T14:28:48.537Z cpu13:2098301)CpuSched: 817: user latency of 2099012 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T14:32:14.531Z cpu13:2098301)CpuSched: 817: user latency of 2099113 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T14:35:41.503Z cpu0:2098301)CpuSched: 817: user latency of 2099213 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T14:39:09.812Z cpu18:2098301)CpuSched: 817: user latency of 2099313 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T14:42:33.417Z cpu16:2098301)CpuSched: 817: user latency of 2099413 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T14:46:08.145Z cpu16:2098301)CpuSched: 817: user latency of 2099513 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T14:49:43.126Z cpu10:2098301)CpuSched: 817: user latency of 2099613 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T14:53:08.373Z cpu4:2098301)CpuSched: 817: user latency of 2099713 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T14:56:37.968Z cpu0:2098301)CpuSched: 817: user latency of 2099813 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T15:00:07.643Z cpu20:2098301)CpuSched: 817: user latency of 2099913 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T15:03:36.670Z cpu20:2098301)CpuSched: 817: user latency of 2100013 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T15:07:09.722Z cpu7:2098301)CpuSched: 817: user latency of 2100113 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T15:10:36.066Z cpu8:2098301)CpuSched: 817: user latency of 2100213 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T15:14:08.379Z cpu7:2098301)CpuSched: 817: user latency of 2100313 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T15:17:40.849Z cpu5:2098301)CpuSched: 817: user latency of 2100413 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
    2021-04-22T15:21:04.834Z cpu42:2100511)VC: 4710: Device rescan time 35 msec (total number of devices 13)
    2021-04-22T15:21:04.834Z cpu42:2100511)VC: 4713: Filesystem probe time 69 msec (devices probed 13 of 13)
    2021-04-22T15:21:04.834Z cpu42:2100511)VC: 4715: Refresh open volume time 0 msec
    2021-04-22T15:21:04.879Z cpu42:2100511)WARNING: VisorFS: 1091: Attempt to setattr non sticky dir/file from tar mount
    2021-04-22T15:21:05.566Z cpu47:2097684)Jumpstart plugin system-storage activated.
    2021-04-22T15:21:05.574Z cpu20:2097673)Activating Jumpstart plugin vsan.
    2021-04-22T15:21:05.574Z cpu18:2097658)Activating Jumpstart plugin backup-configurations.
    2021-04-22T15:21:05.574Z cpu33:2097707)Activating Jumpstart plugin tag-boot-bank.
    2021-04-22T15:21:05.575Z cpu32:2097701)Activating Jumpstart plugin crond.
    2021-04-22T15:21:05.575Z cpu25:2097698)Activating Jumpstart plugin restore-datafiles.
    2021-04-22T15:21:05.579Z cpu1:2097599)Loading module swapobj ...
    2021-04-22T15:21:05.612Z cpu32:2097701)Jumpstart plugin crond activated.
    2021-04-22T15:21:05.617Z cpu18:2097658)Jumpstart plugin backup-configurations activated.
    2021-04-22T15:21:05.623Z cpu25:2097698)Jumpstart plugin restore-datafiles activated.
    2021-04-22T15:21:05.684Z cpu1:2097599)Elf: 2060: module swapobj has license VMware
    2021-04-22T15:21:05.686Z cpu1:2097599)SwapObj: 63: SwapObj backend registration successful.
    2021-04-22T15:21:05.686Z cpu1:2097599)Mod: 4789: Initialization of swapobj succeeded with module ID 86.
    2021-04-22T15:21:05.686Z cpu1:2097599)swapobj loaded successfully.
    2021-04-22T15:21:05.701Z cpu33:2097707)Jumpstart plugin tag-boot-bank activated.
    2021-04-22T15:21:05.817Z cpu20:2097673)Jumpstart plugin vsan activated.
    2021-04-22T15:21:05.824Z cpu39:2097693)Activating Jumpstart plugin vmkapps.
    2021-04-22T15:21:05.824Z cpu6:2097641)Activating Jumpstart plugin mount-disk-fs.
    2021-04-22T15:21:05.927Z cpu1:2097599)VC: 4710: Device rescan time 30 msec (total number of devices 13)
    2021-04-22T15:21:05.927Z cpu1:2097599)VC: 4713: Filesystem probe time 68 msec (devices probed 12 of 13)
    2021-04-22T15:21:05.927Z cpu1:2097599)VC: 4715: Refresh open volume time 2 msec
    2021-04-22T15:21:05.930Z cpu6:2097641)Jumpstart plugin mount-disk-fs activated.

     

    -------------

    Running sxcli storage core device list I get the following:

    [root@localhost:~] esxcli storage core device list
    naa.6e843b635343d58d27d2d478dd9a1bd3
    Display Name: QNAP iSCSI Disk (naa.6e843b635343d58d27d2d478dd9a1bd3)
    Has Settable Display Name: true
    Size: 10104448
    Device Type: Direct-Access
    Multipath Plugin: NMP
    Devfs Path: /vmfs/devices/disks/naa.6e843b635343d58d27d2d478dd9a1bd3
    Vendor: QNAP
    Model: iSCSI Storage
    Revision: 4.0
    SCSI Level: 5
    Is Pseudo: false
    Status: on
    Is RDM Capable: true
    Is Local: false
    Is Removable: false
    Is SSD: false
    Is VVOL PE: false
    Is Offline: false
    Is Perennially Reserved: false
    Queue Full Sample Size: 0
    Queue Full Threshold: 0
    Thin Provisioning Status: unknown
    Attached Filters:
    VAAI Status: supported
    Other UIDs: vml.02000000006e843b635343d58d27d2d478dd9a1bd3695343534920
    Is Shared Clusterwide: true
    Is SAS: false
    Is USB: false
    Is Boot Device: false
    Device Max Queue Depth: 128
    No of outstanding IOs with competing worlds: 32
    Drive Type: unknown
    RAID Level: unknown
    Number of Physical Drives: unknown
    Protection Enabled: false
    PI Activated: false
    PI Type: 0
    PI Protection Mask: NO PROTECTION
    Supported Guard Types: NO GUARD SUPPORT
    DIX Enabled: false
    DIX Guard Type: NO GUARD SUPPORT
    Emulated DIX/DIF Enabled: false

    naa.6e843b61523a9a0dc05dd4b8bd8b92d7
    Display Name: QNAP iSCSI Disk (naa.6e843b61523a9a0dc05dd4b8bd8b92d7)
    Has Settable Display Name: true
    Size: 10485760
    Device Type: Direct-Access
    Multipath Plugin: NMP
    Devfs Path: /vmfs/devices/disks/naa.6e843b61523a9a0dc05dd4b8bd8b92d7
    Vendor: QNAP
    Model: iSCSI Storage
    Revision: 4.0
    SCSI Level: 5
    Is Pseudo: false
    Status: on
    Is RDM Capable: true
    Is Local: false
    Is Removable: false
    Is SSD: false
    Is VVOL PE: false
    Is Offline: false
    Is Perennially Reserved: false
    Queue Full Sample Size: 0
    Queue Full Threshold: 0
    Thin Provisioning Status: unknown
    Attached Filters:
    VAAI Status: supported
    Other UIDs: vml.02000100006e843b61523a9a0dc05dd4b8bd8b92d7695343534920
    Is Shared Clusterwide: true
    Is SAS: false
    Is USB: false
    Is Boot Device: false
    Device Max Queue Depth: 128
    No of outstanding IOs with competing worlds: 32
    Drive Type: unknown
    RAID Level: unknown
    Number of Physical Drives: unknown
    Protection Enabled: false
    PI Activated: false
    PI Type: 0
    PI Protection Mask: NO PROTECTION
    Supported Guard Types: NO GUARD SUPPORT
    DIX Enabled: false
    DIX Guard Type: NO GUARD SUPPORT
    Emulated DIX/DIF Enabled: false

    t10.ATA_____ORTIAL_SSD______________________________0000060629__________
    Display Name: Local ATA Disk (t10.ATA_____ORTIAL_SSD______________________________0000060629__________)
    Has Settable Display Name: false
    Size: 488386
    Device Type: Direct-Access
    Multipath Plugin: HPP
    Devfs Path: /vmfs/devices/disks/t10.ATA_____ORTIAL_SSD______________________________0000060629__________
    Vendor: ATA
    Model: ORTIAL SSD
    Revision: 117K
    SCSI Level: 5
    Is Pseudo: false
    Status: on
    Is RDM Capable: false
    Is Local: true
    Is Removable: false
    Is SSD: true
    Is VVOL PE: false
    Is Offline: false
    Is Perennially Reserved: false
    Queue Full Sample Size: 0
    Queue Full Threshold: 0
    Thin Provisioning Status: yes
    Attached Filters:
    VAAI Status: unsupported
    Other UIDs: vml.010000000030303030303630363239202020202020202020204f525449414c
    Is Shared Clusterwide: false
    Is SAS: false
    Is USB: false
    Is Boot Device: false
    Device Max Queue Depth: 31
    No of outstanding IOs with competing worlds: 31
    Drive Type: unknown
    RAID Level: unknown
    Number of Physical Drives: unknown
    Protection Enabled: false
    PI Activated: false
    PI Type: 0
    PI Protection Mask: NO PROTECTION
    Supported Guard Types: NO GUARD SUPPORT
    DIX Enabled: false
    DIX Guard Type: NO GUARD SUPPORT
    Emulated DIX/DIF Enabled: false

    naa.6e843b62b0eba9dd7b05d4843d92a3de
    Display Name: QNAP iSCSI Disk (naa.6e843b62b0eba9dd7b05d4843d92a3de)
    Has Settable Display Name: true
    Size: 10485760
    Device Type: Direct-Access
    Multipath Plugin: NMP
    Devfs Path: /vmfs/devices/disks/naa.6e843b62b0eba9dd7b05d4843d92a3de
    Vendor: QNAP
    Model: iSCSI Storage
    Revision: 4.0
    SCSI Level: 5
    Is Pseudo: false
    Status: on
    Is RDM Capable: true
    Is Local: false
    Is Removable: false
    Is SSD: false
    Is VVOL PE: false
    Is Offline: false
    Is Perennially Reserved: false
    Queue Full Sample Size: 0
    Queue Full Threshold: 0
    Thin Provisioning Status: unknown
    Attached Filters:
    VAAI Status: supported
    Other UIDs: vml.02000000006e843b62b0eba9dd7b05d4843d92a3de695343534920
    Is Shared Clusterwide: true
    Is SAS: false
    Is USB: false
    Is Boot Device: false
    Device Max Queue Depth: 128
    No of outstanding IOs with competing worlds: 32
    Drive Type: unknown
    RAID Level: unknown
    Number of Physical Drives: unknown
    Protection Enabled: false
    PI Activated: false
    PI Type: 0
    PI Protection Mask: NO PROTECTION
    Supported Guard Types: NO GUARD SUPPORT
    DIX Enabled: false
    DIX Guard Type: NO GUARD SUPPORT
    Emulated DIX/DIF Enabled: false

    naa.6e843b6a8bf8ab4dc621d4324db3f4d9
    Display Name: QNAP iSCSI Disk (naa.6e843b6a8bf8ab4dc621d4324db3f4d9)
    Has Settable Display Name: true
    Size: 9961472
    Device Type: Direct-Access
    Multipath Plugin: NMP
    Devfs Path: /vmfs/devices/disks/naa.6e843b6a8bf8ab4dc621d4324db3f4d9
    Vendor: QNAP
    Model: iSCSI Storage
    Revision: 4.0
    SCSI Level: 5
    Is Pseudo: false
    Status: on
    Is RDM Capable: true
    Is Local: false
    Is Removable: false
    Is SSD: false
    Is VVOL PE: false
    Is Offline: false
    Is Perennially Reserved: false
    Queue Full Sample Size: 0
    Queue Full Threshold: 0
    Thin Provisioning Status: unknown
    Attached Filters:
    VAAI Status: supported
    Other UIDs: vml.02000000006e843b6a8bf8ab4dc621d4324db3f4d9695343534920
    Is Shared Clusterwide: true
    Is SAS: false
    Is USB: false
    Is Boot Device: false
    Device Max Queue Depth: 128
    No of outstanding IOs with competing worlds: 32
    Drive Type: unknown
    RAID Level: unknown
    Number of Physical Drives: unknown
    Protection Enabled: false
    PI Activated: false
    PI Type: 0
    PI Protection Mask: NO PROTECTION
    Supported Guard Types: NO GUARD SUPPORT
    DIX Enabled: false
    DIX Guard Type: NO GUARD SUPPORT
    Emulated DIX/DIF Enabled: false

    naa.6e843b6e311bd35dbb1fd428edaefddc
    Display Name: QNAP iSCSI Disk (naa.6e843b6e311bd35dbb1fd428edaefddc)
    Has Settable Display Name: true
    Size: 9437184
    Device Type: Direct-Access
    Multipath Plugin: NMP
    Devfs Path: /vmfs/devices/disks/naa.6e843b6e311bd35dbb1fd428edaefddc
    Vendor: QNAP
    Model: iSCSI Storage
    Revision: 4.0
    SCSI Level: 5
    Is Pseudo: false
    Status: on
    Is RDM Capable: true
    Is Local: false
    Is Removable: false
    Is SSD: false
    Is VVOL PE: false
    Is Offline: false
    Is Perennially Reserved: false
    Queue Full Sample Size: 0
    Queue Full Threshold: 0
    Thin Provisioning Status: unknown
    Attached Filters:
    VAAI Status: supported
    Other UIDs: vml.02000200006e843b6e311bd35dbb1fd428edaefddc695343534920
    Is Shared Clusterwide: true
    Is SAS: false
    Is USB: false
    Is Boot Device: false
    Device Max Queue Depth: 128
    No of outstanding IOs with competing worlds: 32
    Drive Type: unknown
    RAID Level: unknown
    Number of Physical Drives: unknown
    Protection Enabled: false
    PI Activated: false
    PI Type: 0
    PI Protection Mask: NO PROTECTION
    Supported Guard Types: NO GUARD SUPPORT
    DIX Enabled: false
    DIX Guard Type: NO GUARD SUPPORT
    Emulated DIX/DIF Enabled: false

    naa.6e843b62aa61ca4dcad9d4c40da5e2d7
    Display Name: QNAP iSCSI Disk (naa.6e843b62aa61ca4dcad9d4c40da5e2d7)
    Has Settable Display Name: true
    Size: 16637440
    Device Type: Direct-Access
    Multipath Plugin: NMP
    Devfs Path: /vmfs/devices/disks/naa.6e843b62aa61ca4dcad9d4c40da5e2d7
    Vendor: QNAP
    Model: iSCSI Storage
    Revision: 4.0
    SCSI Level: 5
    Is Pseudo: false
    Status: on
    Is RDM Capable: true
    Is Local: false
    Is Removable: false
    Is SSD: false
    Is VVOL PE: false
    Is Offline: false
    Is Perennially Reserved: false
    Queue Full Sample Size: 0
    Queue Full Threshold: 0
    Thin Provisioning Status: unknown
    Attached Filters:
    VAAI Status: supported
    Other UIDs: vml.02000100006e843b62aa61ca4dcad9d4c40da5e2d7695343534920
    Is Shared Clusterwide: true
    Is SAS: false
    Is USB: false
    Is Boot Device: false
    Device Max Queue Depth: 128
    No of outstanding IOs with competing worlds: 32
    Drive Type: unknown
    RAID Level: unknown
    Number of Physical Drives: unknown
    Protection Enabled: false
    PI Activated: false
    PI Type: 0
    PI Protection Mask: NO PROTECTION
    Supported Guard Types: NO GUARD SUPPORT
    DIX Enabled: false
    DIX Guard Type: NO GUARD SUPPORT
    Emulated DIX/DIF Enabled: false

    mpx.vmhba0:C0:T5:L0
    Display Name: Local hp CD-ROM (mpx.vmhba0:C0:T5:L0)
    Has Settable Display Name: false
    Size: 0
    Device Type: CD-ROM
    Multipath Plugin: HPP
    Devfs Path:
    Vendor: hp
    Model: DVDRAM GUB0N
    Revision: UK00
    SCSI Level: 5
    Is Pseudo: false
    Status: off
    Is RDM Capable: false
    Is Local: true
    Is Removable: true
    Is SSD: false
    Is VVOL PE: false
    Is Offline: false
    Is Perennially Reserved: true
    Queue Full Sample Size: 0
    Queue Full Threshold: 0
    Thin Provisioning Status: unknown
    Attached Filters:
    VAAI Status: unknown
    Other UIDs: vml.0005000000766d686261303a353a30
    Is Shared Clusterwide: false
    Is SAS: false
    Is USB: false
    Is Boot Device: false
    Device Max Queue Depth: 16
    No of outstanding IOs with competing worlds: 16
    Drive Type: unknown
    RAID Level: unknown
    Number of Physical Drives: unknown
    Protection Enabled: false
    PI Activated: false
    PI Type: 0
    PI Protection Mask: NO PROTECTION
    Supported Guard Types: NO GUARD SUPPORT
    DIX Enabled: false
    DIX Guard Type: NO GUARD SUPPORT
    Emulated DIX/DIF Enabled: false

    naa.6e843b66f1b4efbd2f4cd4c8bdadf5d3
    Display Name: QNAP iSCSI Disk (naa.6e843b66f1b4efbd2f4cd4c8bdadf5d3)
    Has Settable Display Name: true
    Size: 18874368
    Device Type: Direct-Access
    Multipath Plugin: NMP
    Devfs Path: /vmfs/devices/disks/naa.6e843b66f1b4efbd2f4cd4c8bdadf5d3
    Vendor: QNAP
    Model: iSCSI Storage
    Revision: 4.0
    SCSI Level: 5
    Is Pseudo: false
    Status: on
    Is RDM Capable: true
    Is Local: false
    Is Removable: false
    Is SSD: false
    Is VVOL PE: false
    Is Offline: false
    Is Perennially Reserved: false
    Queue Full Sample Size: 0
    Queue Full Threshold: 0
    Thin Provisioning Status: unknown
    Attached Filters:
    VAAI Status: supported
    Other UIDs: vml.02000000006e843b66f1b4efbd2f4cd4c8bdadf5d3695343534920
    Is Shared Clusterwide: true
    Is SAS: false
    Is USB: false
    Is Boot Device: false
    Device Max Queue Depth: 128
    No of outstanding IOs with competing worlds: 32
    Drive Type: unknown
    RAID Level: unknown
    Number of Physical Drives: unknown
    Protection Enabled: false
    PI Activated: false
    PI Type: 0
    PI Protection Mask: NO PROTECTION
    Supported Guard Types: NO GUARD SUPPORT
    DIX Enabled: false
    DIX Guard Type: NO GUARD SUPPORT
    Emulated DIX/DIF Enabled: false

    t10.ATA_____ORTIAL_SSD______________________________0000060618__________
    Display Name: Local ATA Disk (t10.ATA_____ORTIAL_SSD______________________________0000060618__________)
    Has Settable Display Name: false
    Size: 488386
    Device Type: Direct-Access
    Multipath Plugin: HPP
    Devfs Path: /vmfs/devices/disks/t10.ATA_____ORTIAL_SSD______________________________0000060618__________
    Vendor: ATA
    Model: ORTIAL SSD
    Revision: 117K
    SCSI Level: 5
    Is Pseudo: false
    Status: on
    Is RDM Capable: false
    Is Local: true
    Is Removable: false
    Is SSD: true
    Is VVOL PE: false
    Is Offline: false
    Is Perennially Reserved: false
    Queue Full Sample Size: 0
    Queue Full Threshold: 0
    Thin Provisioning Status: yes
    Attached Filters:
    VAAI Status: unsupported
    Other UIDs: vml.010000000030303030303630363138202020202020202020204f525449414c
    Is Shared Clusterwide: false
    Is SAS: false
    Is USB: false
    Is Boot Device: true
    Device Max Queue Depth: 31
    No of outstanding IOs with competing worlds: 31
    Drive Type: unknown
    RAID Level: unknown
    Number of Physical Drives: unknown
    Protection Enabled: false
    PI Activated: false
    PI Type: 0
    PI Protection Mask: NO PROTECTION
    Supported Guard Types: NO GUARD SUPPORT
    DIX Enabled: false
    DIX Guard Type: NO GUARD SUPPORT
    Emulated DIX/DIF Enabled: false

    naa.6e843b640b4b3b4de9fed4402da808da
    Display Name: QNAP iSCSI Disk (naa.6e843b640b4b3b4de9fed4402da808da)
    Has Settable Display Name: true
    Size: 10485760
    Device Type: Direct-Access
    Multipath Plugin: NMP
    Devfs Path: /vmfs/devices/disks/naa.6e843b640b4b3b4de9fed4402da808da
    Vendor: QNAP
    Model: iSCSI Storage
    Revision: 4.0
    SCSI Level: 5
    Is Pseudo: false
    Status: on
    Is RDM Capable: true
    Is Local: false
    Is Removable: false
    Is SSD: false
    Is VVOL PE: false
    Is Offline: false
    Is Perennially Reserved: false
    Queue Full Sample Size: 0
    Queue Full Threshold: 0
    Thin Provisioning Status: unknown
    Attached Filters:
    VAAI Status: supported
    Other UIDs: vml.02000100006e843b640b4b3b4de9fed4402da808da695343534920
    Is Shared Clusterwide: true
    Is SAS: false
    Is USB: false
    Is Boot Device: false
    Device Max Queue Depth: 128
    No of outstanding IOs with competing worlds: 32
    Drive Type: unknown
    RAID Level: unknown
    Number of Physical Drives: unknown
    Protection Enabled: false
    PI Activated: false
    PI Type: 0
    PI Protection Mask: NO PROTECTION
    Supported Guard Types: NO GUARD SUPPORT
    DIX Enabled: false
    DIX Guard Type: NO GUARD SUPPORT
    Emulated DIX/DIF Enabled: false

     

     

     



  • 2.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Apr 26, 2021 10:24 AM

    If I try to apply this:

     

    https://blogs.vmware.com/virtualblocks/2020/07/31/using-perennially-reserved-flag-wsfc-rdms/

    boot times do not slow down, and I also get this warning i big number of time in the vmkernel.log file.

    cpu6:2098298)WARNING: Partition: 1437: Device "naa.6e843b66f1b4efbd2f4cd4c8bdadf5d3" with a VMFS partition is marked perennially reserved. This is not supported and may lead to data loss.

     

     

    Any idea how can I investigate?



  • 3.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Jun 14, 2021 08:14 AM

    I have the exact same problem. It takes about an hour to boot on 7.0.2, while 6.7 boots immediately.

    EDIT: Just tried 7.0.1 and that booted just fine. I am using the custom Lenovo images.



  • 4.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Jun 14, 2021 09:00 AM

    I solved fresh installing 7.0.2, and not upgrading from 6.7



  • 5.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Jun 15, 2021 01:55 PM

    We have the exact same problem using a HPE customized 7.0 U2 image. Strangely enough, the first Host updated without a problem.



  • 6.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Jun 15, 2021 02:10 PM

    This issue happened on all my 5 HPE hosts. 



  • 7.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Jun 16, 2021 06:58 AM

    Which kind of storage are you using? Maybe it is related to 3PAR Volumes?



  • 8.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Jun 16, 2021 07:07 AM

    I am using Unity so I don't think it's related to storage.



  • 9.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Jun 16, 2021 07:25 AM

    In my 10 year vsphere career I never had such issues. Also vcenter has many strange problems. Maybe it is time to invest in quality than feature quantity.

    Fresh installing the host did resolve the problem for me also, but it is PITA if you have many hosts.



  • 10.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Jun 16, 2021 08:42 AM

    I am sure it's related to storage, and possible to iSCSI ( I have 6 iSCSI targets for each host).

    If I disconnect iSCSI targets, boot is fast.



  • 11.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Jun 16, 2021 09:33 AM

    I am not using iSCSI or the above mentioned RDM Disks. Everything is attached over FC.

    In addition I found out, that VMs beeing backed up residing on an upgraded Host need consolidation after a snapshot backup. The manual consolidation finishes without an error, BUT the warning remains and the disks are not consolidated properly.

    Only way to consolidate the disk is to shut down the VM or vmotion it to a not upgraded 6.7 Host. A newly installed 7.0 U2 Hosts does not have this behavior.

    It seems that there are some major underlying issues in the upgrade process. At the moment I cannot recommend an Upgrade from 6.7 to 7 U2a (at least for our environment).



  • 12.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Jun 16, 2021 09:37 AM

    I did not face issues you mention. Consider that at the moment I still have a couple oh hosts not fresh installed, with the underlying issue.

    I think I'll schedule a fresh install with major priority after your mentions.

     

    Thanks.



  • 13.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Jul 13, 2021 05:42 PM

    HI everyone.  

    This just happened to me this morning. I used Lifecycle Manager to upgrade one host from 6.7U3 to 7.0.2 Dell EMC and it now takes 45 minutes to boot.  It gets stuck at the exact same spot and also has similar events in the log files.

    Was there ever a fix determined for this ? I would prefer to not have to rebuild each host from scratch,  I've opened a SR and will post back with any updates.



  • 14.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Jul 16, 2021 12:06 PM

    Here is the KB Article describing the problem;

    https://kb.vmware.com/s/article/84249

    You need to copy boot.cfg from bootbank to /altbootbank and edit it. 

    SSh to the ESXi host in question.
      - cp /bootbank/boot.cfg /altbootbank/boot.cfg
      - vi /altbootbank/boot.cfg

          Change the first line 'bootstate=0' > 'bootstate=3'
          Change last line 'updated=N' > 'update=N-1' (eg update=3 > updated=2)
        
       - Save file
    Reboot and test it. 



  • 15.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Feb 06, 2022 10:32 AM

    >This issue is resolved in VMware vSphere ESXi 7.0 U2c. 

    Nope had the same issue with 7.0 U3c, and same workaround fixed it, thx



  • 16.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Feb 09, 2022 04:34 PM

    Hello! Had the same issue with a Lenovo SR650 server when trying to upgrade from 6.5 17167537 to 7.0 U2A when using Life Cylcle Manager. Tried to upgrade by mounting the ISO with XCC card and no issue at all. I've just done one host out of 6 so far. Will see if it is ok for the other ones.



  • 17.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Jun 09, 2022 12:05 PM

    fresh install of VMware-VMvisor-Installer-7.0.0.update03-19482537.x86_64-DellEMC_Customized-A04 to BOSS adapter in DELL R740. Storage is DELL SC3020 with SAS 12G direct connectivity. After first reboot the booting screen hangs on "vmw_satp_alua loaded successfully" and reboot - shutdown process was very slow - "reboot_run_handlersScsiEarly" , BUT i had connected 2 luns to the server visible was only 1. 

    i think, that is problem with storage and alua. 

    In my case i simply add these commands to ESXi host and after reboot boots in some seconds and i see both LUNs. 

    • esxcli system module parameters set -p issue_scsi_cmd_to_bringup_drive=0 -m lsi_msgpt3
    • esxcli storage nmp satp rule add -s VMW_SATP_ALUA -V COMPELNT -P VMW_PSP_RR -o disable_action_OnRetryErrors -e "Dell EMC SC Series Claim Rule" -O "policy=iops;iops=3"

     

     

     

     



  • 18.  RE: Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

    Posted Dec 12, 2022 11:15 PM

    Everyone on this thread should read this KB.

    I was seeing this problem but only on the server with a different brand USB drive.

    Are you seeing "All Paths Down" in your syslog.log? This means you lost access to the boot drive.  The problem in this KB could be the culprit. The step for me that was slowing down boot was setting up the swap file. This now makes perfect sense.

    What I find to be absolutely shocking: This has been a bug for 2 years. Not only is it not fixed, but they are carrying on as though it's no big deal because ESX still works. They are saying both "this is not a problem with the vmlinux driver", and "have a conversation with your USB/SD manufacturer about dropouts". 

    And now we're hearing booting from USB profiles is getting deprecated. Gee, is it because you don't know how to write a driver?

    https://kb.vmware.com/s/article/83450