Падает хост ESXI 6.7

[AlexFail]

Добрый день! Столкнулся с такой проблемой, примерно раз в день начал падать хост esxi 6.7. В логе в vmkvarning вот такие записи.

0:00:00:05.209 cpu0:2097152)WARNING: VMKAcpi: 318: \_SB_.PC00.LPC0.TMR_: skipping GSIV 0 conflict

0:00:00:05.260 cpu0:2097152)WARNING: Chipset: 396: Bus 5 (04) is already defined

2019-11-15T13:03:54.730Z cpu14:2097846)WARNING: Failed to init interrupt.

2019-11-15T13:03:54.875Z cpu14:2097846)WARNING: Failed to init interrupt.

2019-11-15T13:03:57.350Z cpu37:2097861)WARNING: ScsiPath: 8915: Adapter Invalid does not exist

2019-11-15T13:03:57.350Z cpu17:2097863)WARNING: PCI: 1209: 0000:00:14.0 is nameless

2019-11-15T13:04:02.143Z cpu26:2097926)WARNING: etherswitch: PortCfg_ModInit:910: Skipped initializing etherswitch portcfg for VSS to use cswitch and portcfg module

2019-11-15T13:04:06.041Z cpu26:2098075)WARNING: FBFT not enabled

2019-11-15T13:04:11.074Z cpu8:2097791)WARNING: ScsiDeviceIO: 10107: Mode Sense cmd reported block size 0, does not match the current logical block size 512(with physical block size 512) for device.

2019-11-15T13:04:11.074Z cpu8:2097791)WARNING: ScsiDeviceIO: 10109: The device mpx.vmhba32:C0:T0:L0 is marked format corrupt.

2019-11-15T13:04:15.558Z cpu8:2097791)WARNING: NFS: 1227: Invalid volume UUID 5d9b4bb4-89590e3c-9b54-ac1f6bd20130

2019-11-15T13:04:17.148Z cpu23:2098725)WARNING: APEI: 319: Could not initialize EINJ

2019-11-15T13:04:27.882Z cpu5:2098993)WARNING: NTPClock: 1561: system clock synchronized to upstream time servers

2019-11-15T13:04:44.460Z cpu32:2100049)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 0

2019-11-15T13:04:44.460Z cpu32:2100049)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 1

2019-11-15T13:05:14.521Z cpu25:2100282)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 0

2019-11-15T13:05:14.521Z cpu25:2100282)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 1

2019-11-15T13:05:44.492Z cpu4:2100440)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 0

2019-11-15T13:06:14.572Z cpu46:2100595)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 0

2019-11-15T13:06:14.572Z cpu46:2100595)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 1

2019-11-15T13:06:14.572Z cpu46:2100595)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 2

2019-11-15T13:06:14.572Z cpu46:2100595)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 3

2019-11-15T13:06:14.572Z cpu46:2100595)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 4

2019-11-15T13:06:14.572Z cpu46:2100595)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 5

2019-11-15T13:06:14.572Z cpu46:2100595)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 6

2019-11-15T13:06:14.572Z cpu46:2100595)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 7

2019-11-15T13:06:14.572Z cpu46:2100595)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 8

2019-11-15T13:06:14.572Z cpu46:2100595)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 9

2019-11-15T13:06:14.572Z cpu46:2100595)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 10

2019-11-15T13:06:14.572Z cpu46:2100595)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 11

2019-11-15T13:12:16.069Z cpu14:2098993)WARNING: NTPClock: 1561: system clock synchronized to upstream time servers

Подскажите пожалуйста, где поискать проблему?

Спасибо.

[dimsua]

Здравствуйте.

ESXI установлен на флешке?

[AlexFail]

Добрый день.

Да на usb.

Локально хост доступен для управления, пропадает связь с ним по сети, так же падают виртуальные машины.

[dimsua]

В логах встречается
2019-11-15T13:04:11.074Z cpu8:2097791)WARNING: ScsiDeviceIO: 10109: The device mpx.vmhba32:C0:T0:L0 is marked format corrupt.

Это какая-то проблема с работой флешки.

Лучше ее заменить.
А может достаточно будет контакты почистить.

[AlexFail]

Добрый день. Заменил флешку на новую, проблема не исчезла. Что можно еще посмотреть или какой лог выгрузить?

Спасибо.

[E_Espinel]

Здравствуйте.
мы должны исключить все возможные варианты.
1.- Обновите прошивку всего серверного оборудования (Bios, контроллер HBA, сетевые карты и т. Д.).
2.- Если сервером является Lenovo, HP, Supermicro и т. Д. VMware должен быть установлен с пользовательским ISO-образом производителя.

указать все технические характеристики сервера,

такие как: тип процессора, контроллер дисков, дисков, карт и другие устройства. также ваши уровни прошивки.

[Finikiez]

Добрый день!

А что в ваших терминах "падает".

Вы наблюдете PSOD? Или сервер зависает?

Или что происходит?

[AlexFail]

Пропадает пинг до сервера, но локально сервер доступен, psod нет.

ВМ также недоступны.

1. Прошивки актуальны.

2. Установлен стандартный образ, есть аналогичный сервер, с такими же прошивками, проблему не наблюдаю.

3.

м.плата supermicro X11DPH-I

проц. Intel Xeon Gold 5118 x2

рк LSI MegaRAID 9361-8i

log vmwarning

0:00:00:05.208 cpu0:2097152)WARNING: VMKAcpi: 318: \_SB_.PC00.LPC0.TMR_: skipping GSIV 0 conflict

0:00:00:05.259 cpu0:2097152)WARNING: Chipset: 396: Bus 5 (04) is already defined

2019-11-22T11:10:07.688Z cpu31:2097846)WARNING: Failed to init interrupt.

2019-11-22T11:10:07.838Z cpu31:2097846)WARNING: Failed to init interrupt.

2019-11-22T11:10:10.303Z cpu36:2097861)WARNING: ScsiPath: 8915: Adapter Invalid does not exist

2019-11-22T11:10:10.303Z cpu24:2097863)WARNING: PCI: 1209: 0000:00:14.0 is nameless

2019-11-22T11:10:15.121Z cpu26:2097926)WARNING: etherswitch: PortCfg_ModInit:910: Skipped initializing etherswitch portcfg for VSS to use cswitch and portcfg module

2019-11-22T11:10:19.104Z cpu21:2098075)WARNING: FBFT not enabled

2019-11-22T11:10:24.143Z cpu16:2097791)WARNING: ScsiDeviceIO: 10107: Mode Sense cmd reported block size 0, does not match the current logical block size 512(with physical block size 512) for device.

2019-11-22T11:10:24.143Z cpu16:2097791)WARNING: ScsiDeviceIO: 10109: The device mpx.vmhba32:C0:T0:L0 is marked format corrupt.

2019-11-22T11:10:28.639Z cpu4:2097791)WARNING: NFS: 1227: Invalid volume UUID 5d9b4bb4-89590e3c-9b54-ac1f6bd20130

2019-11-22T11:10:30.090Z cpu42:2098725)WARNING: APEI: 319: Could not initialize EINJ

2019-11-22T11:10:40.709Z cpu0:2098994)WARNING: NTPClock: 1561: system clock synchronized to upstream time servers

2019-11-22T11:10:56.953Z cpu46:2100064)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 0

2019-11-22T11:10:56.953Z cpu46:2100064)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 1

2019-11-22T11:11:27.047Z cpu36:2100319)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 0

2019-11-22T11:11:27.047Z cpu36:2100319)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 1

2019-11-22T11:11:57.024Z cpu41:2100477)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 0

2019-11-22T11:12:27.307Z cpu43:2100632)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 0

2019-11-22T11:12:27.307Z cpu43:2100632)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 1

2019-11-22T11:12:27.307Z cpu43:2100632)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 2

2019-11-22T11:12:27.307Z cpu43:2100632)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 3

2019-11-22T11:12:27.307Z cpu43:2100632)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 4

2019-11-22T11:12:27.307Z cpu43:2100632)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 5

2019-11-22T11:12:27.307Z cpu43:2100632)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 6

2019-11-22T11:12:27.307Z cpu43:2100632)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 7

2019-11-22T11:12:27.308Z cpu43:2100632)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 8

2019-11-22T11:12:27.308Z cpu43:2100632)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 9

2019-11-22T11:12:27.308Z cpu43:2100632)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 10

2019-11-22T11:12:27.308Z cpu43:2100632)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 11

log wmkernel

2019-11-22T11:11:04.784Z cpu18:2097640)lsi_mr3: mfi_TaskMgmt:662: Processing taskMgmt virt reset for device: vmhba2:C2:T0:L0

2019-11-22T11:11:04.784Z cpu18:2097640)lsi_mr3: mfi_TaskMgmt:667: Virtual Reset request from Wld 2100065

2019-11-22T11:11:04.784Z cpu18:2097640)lsi_mr3: mfi_VirtReset:619: VIRT_REST completed. Initiator ID 0x430813173500 World ID 2100065

2019-11-22T11:11:04.784Z cpu18:2097640)VSCSI: 2691: handle 8203(vscsi0:12):Completing reset (0 outstanding commands)

2019-11-22T11:11:04.784Z cpu18:2097640)VSCSI: 2903: handle 8204(vscsi0:13):Reset [Retries: 0/0] from (vmm0:vCenter_Server)

2019-11-22T11:11:04.784Z cpu18:2097640)lsi_mr3: mfi_TaskMgmt:662: Processing taskMgmt virt reset for device: vmhba2:C2:T0:L0

2019-11-22T11:11:04.784Z cpu18:2097640)lsi_mr3: mfi_TaskMgmt:667: Virtual Reset request from Wld 2100065

2019-11-22T11:11:04.784Z cpu18:2097640)lsi_mr3: mfi_VirtReset:619: VIRT_REST completed. Initiator ID 0x4308131738c0 World ID 2100065

2019-11-22T11:11:04.784Z cpu18:2097640)VSCSI: 2691: handle 8204(vscsi0:13):Completing reset (0 outstanding commands)

2019-11-22T11:11:10.629Z cpu0:2100089)Vmxnet3: 18569: indLROPktToGuest: 0, vcd->umkShared->vrrsSelected: 1 port 0x2000005

2019-11-22T11:11:10.629Z cpu0:2100089)Vmxnet3: 18810: Using default queue delivery for vmxnet3 for port 0x2000005

2019-11-22T11:11:10.629Z cpu0:2100089)NetPort: 1359: enabled port 0x2000005 with mac 00:0c:29:c6:44:6c

2019-11-22T11:11:15.663Z cpu14:2100086)NetPort: 1580: disabled port 0x2000005

2019-11-22T11:11:15.668Z cpu14:2100086)Vmxnet3: 18569: indLROPktToGuest: 0, vcd->umkShared->vrrsSelected: 1 port 0x2000005

2019-11-22T11:11:15.668Z cpu14:2100086)Vmxnet3: 18810: Using default queue delivery for vmxnet3 for port 0x2000005

2019-11-22T11:11:15.668Z cpu14:2100086)NetPort: 1359: enabled port 0x2000005 with mac 00:0c:29:c6:44:6c

2019-11-22T11:11:26.879Z cpu33:2099940 opID=c3ced407)World: 11943: VC opID vim-cmd-a9-5c89-5cc5 maps to vmkernel opID c3ced407

2019-11-22T11:11:26.879Z cpu33:2099940 opID=c3ced407)Config: 703: "SIOControlFlag2" = 1, Old Value: 0, (Status: 0x0)

2019-11-22T11:11:26.971Z cpu36:2100319)MemSched: vm 2100319: 5745: extended swap to 20480 pgs

2019-11-22T11:11:27.032Z cpu36:2100319)MemSched: vm 2100319: 5745: extended swap to 20992 pgs

2019-11-22T11:11:27.033Z cpu36:2100319)World: vm 2100320: 7076: Starting world vmm0:vku-control of type 8

2019-11-22T11:11:27.033Z cpu36:2100319)Sched: vm 2100320: 6196: Adding world 'vmm0:vku-control', group 'host/user', cpu: shares=-3 min=0 minLimit=-1 max=-1, mem: shares=-3 min=0 minLimit=-1 max=-1

2019-11-22T11:11:27.033Z cpu36:2100319)Sched: vm 2100320: 6211: renamed group 12640 to vm.2100319

2019-11-22T11:11:27.033Z cpu36:2100319)Sched: vm 2100320: 6228: group 12640 is located under group 4

2019-11-22T11:11:27.034Z cpu36:2100319)World: vm 2100324: 7076: Starting world vmm1:vku-control of type 8

2019-11-22T11:11:27.047Z cpu36:2100319)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 0

2019-11-22T11:11:27.047Z cpu36:2100319)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 1

2019-11-22T11:11:27.076Z cpu36:2100319)VSCSI: 3810: handle 8205(vscsi0:0):Creating Virtual Device for world 2100320 (FSS handle 461319) numBlocks=838860800 (bs=512)

2019-11-22T11:11:27.076Z cpu36:2100319)VSCSI: 273: handle 8205(vscsi0:0):Input values: res=0 limit=-2 bw=-1 Shares=1000

2019-11-22T11:11:27.091Z cpu12:2100320)VMMVMKCall: 244: Received INIT from world 2100320

2019-11-22T11:11:27.108Z cpu12:2100320)LSI: 1781: LSI: Initialized rings for scsi0 async=1

2019-11-22T11:11:27.109Z cpu38:2100324)VMMVMKCall: 244: Received INIT from world 2100324

2019-11-22T11:11:27.111Z cpu42:2100329)Net: 2456: connected vku-control eth1 to VM Network, portID 0x2000006

2019-11-22T11:11:27.111Z cpu42:2100329)Net: 2456: connected vku-control eth0 to External port group, portID 0x3000004

2019-11-22T11:11:27.120Z cpu20:2099359)Config: 703: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0)

2019-11-22T11:11:27.123Z cpu42:2100320)----------------  ----------------  ----------------  ----------------  ----------------

2019-11-22T11:11:27.123Z cpu42:2100320)            name       reserved,KB   peakReserved,KB          alloc,KB      peakAlloc,KB

2019-11-22T11:11:27.123Z cpu42:2100320)----------------  ----------------  ----------------  ----------------  ----------------

2019-11-22T11:11:27.123Z cpu42:2100320)             VMM             22612             22612              9004              9004

2019-11-22T11:11:27.123Z cpu42:2100320)          PFRAME              8212              8212                20                20

2019-11-22T11:11:27.123Z cpu42:2100320)      SWAP_CACHE               512               512               512               512

2019-11-22T11:11:27.123Z cpu42:2100320)       CANDIDATE                 4                 4                 4                 4

2019-11-22T11:11:27.123Z cpu42:2100320)  CHECKPOINT_BUF               256               256                 0                 0

2019-11-22T11:11:27.123Z cpu42:2100320)      PSHARE_P2M                20                20                20                20

2019-11-22T11:11:27.123Z cpu42:2100320)            CBRC                 0                 0                 0                 0

2019-11-22T11:11:27.123Z cpu42:2100320)           FTCPT                 0                 0                 0                 0

2019-11-22T11:11:27.123Z cpu42:2100320)      ASYNCREMAP                 4                 4                 4                 4

2019-11-22T11:11:27.123Z cpu42:2100320)   VMMEMSERVICES                64                64                64                64

2019-11-22T11:11:27.123Z cpu42:2100320)FSR_UNSHARED_GBL                 0                 0                 0                 0

2019-11-22T11:11:27.123Z cpu42:2100320)----------------  ----------------  ----------------  ----------------  ----------------

2019-11-22T11:11:27.123Z cpu42:2100320)------------------------------  ------------  ------------  ------------  ------------  ------------

2019-11-22T11:11:27.123Z cpu42:2100320)                          name        min,KB        max,KB   minLimit,KB       eMin,KB   rMinPeak,KB

2019-11-22T11:11:27.123Z cpu42:2100320)------------------------------  ------------  ------------  ------------  ------------  ------------

2019-11-22T11:11:27.123Z cpu42:2100320)                    vm.2100319             0            -1            -1         58360         60144

2019-11-22T11:11:27.123Z cpu42:2100320)            worldGroup.2100319             0            -1            -1           608           608

2019-11-22T11:11:27.123Z cpu42:2100320)                    uw.2100319         20028            -1            -1         20028         21820

2019-11-22T11:11:27.123Z cpu42:2100320)               vsiHeap.2100319             0            -1            -1           136           136

2019-11-22T11:11:27.123Z cpu42:2100320)                    pt.2100319           432            -1            -1           704           704

2019-11-22T11:11:27.123Z cpu42:2100320)            cartelheap.2100319             0            -1            -1           296           296

2019-11-22T11:11:27.123Z cpu42:2100320)             uwshmempt.2100319             0            -1            -1             0             0

2019-11-22T11:11:27.123Z cpu42:2100320)      uwAsyncRemapHeap.2100319             0            -1            -1           136           136

2019-11-22T11:11:27.123Z cpu42:2100320)               uwCrypt.2100319          4096          4096          4096          4096          4096

2019-11-22T11:11:27.123Z cpu42:2100320)              uwregbmp.2100319             8            -1            -1             8             8

2019-11-22T11:11:27.123Z cpu42:2100320)                   vmm.2100319             0            -1            -1             0             0

2019-11-22T11:11:27.123Z cpu42:2100320)               vmmanon.2100319         31684            -1            -1         31684         31684

2019-11-22T11:11:27.123Z cpu42:2100320)                 vmcpt.2100320           272            -1            -1           272           272

2019-11-22T11:11:27.123Z cpu42:2100320)             vmmregbmp.2100319           256            -1            -1           256           256

2019-11-22T11:11:27.123Z cpu42:2100320)     vmmAsyncRemapHeap.2100319             0            -1            -1           136           136

2019-11-22T11:11:27.123Z cpu42:2100320)------------------------------  ------------  ------------  ------------  ------------  ------------

2019-11-22T11:11:27.698Z cpu28:2100320)VSCSI: 2623: handle 8205(vscsi0:0):Reset request on FSS handle 461319 (0 outstanding commands) from (vmm0:vku-control)

2019-11-22T11:11:27.698Z cpu0:2097640)VSCSI: 2903: handle 8205(vscsi0:0):Reset [Retries: 0/0] from (vmm0:vku-control)

2019-11-22T11:11:27.698Z cpu0:2097640)lsi_mr3: mfi_TaskMgmt:662: Processing taskMgmt virt reset for device: vmhba2:C2:T0:L0

2019-11-22T11:11:27.698Z cpu0:2097640)lsi_mr3: mfi_TaskMgmt:667: Virtual Reset request from Wld 2100320

2019-11-22T11:11:27.698Z cpu0:2097640)lsi_mr3: mfi_VirtReset:619: VIRT_REST completed. Initiator ID 0x43081317bf00 World ID 2100320

2019-11-22T11:11:27.698Z cpu0:2097640)VSCSI: 2691: handle 8205(vscsi0:0):Completing reset (0 outstanding commands)

2019-11-22T11:11:33.649Z cpu40:2100324)VSCSI: 2623: handle 8205(vscsi0:0):Reset request on FSS handle 461319 (0 outstanding commands) from (vmm0:vku-control)

2019-11-22T11:11:33.649Z cpu0:2097640)VSCSI: 2903: handle 8205(vscsi0:0):Reset [Retries: 0/0] from (vmm0:vku-control)

2019-11-22T11:11:33.649Z cpu0:2097640)lsi_mr3: mfi_TaskMgmt:662: Processing taskMgmt virt reset for device: vmhba2:C2:T0:L0

2019-11-22T11:11:33.649Z cpu0:2097640)lsi_mr3: mfi_TaskMgmt:667: Virtual Reset request from Wld 2100320

2019-11-22T11:11:33.649Z cpu0:2097640)lsi_mr3: mfi_VirtReset:619: VIRT_REST completed. Initiator ID 0x43081317bf00 World ID 2100320

2019-11-22T11:11:33.649Z cpu0:2097640)VSCSI: 2691: handle 8205(vscsi0:0):Completing reset (0 outstanding commands)

2019-11-22T11:11:33.714Z cpu45:2100320)VSCSIFs: 3934: handle 8205(vscsi0:0):Invalid Opcode (0xa3) from (vmm0:vku-control)

2019-11-22T11:11:46.198Z cpu45:2100331)Vmxnet3: 18569: indLROPktToGuest: 0, vcd->umkShared->vrrsSelected: 1 port 0x3000004

2019-11-22T11:11:46.198Z cpu45:2100331)Vmxnet3: 18810: Using default queue delivery for vmxnet3 for port 0x3000004

2019-11-22T11:11:46.198Z cpu45:2100331)NetPort: 1359: enabled port 0x3000004 with mac 00:0c:29:ab:8b:22

2019-11-22T11:11:46.280Z cpu46:2100329)Vmxnet3: 18569: indLROPktToGuest: 0, vcd->umkShared->vrrsSelected: 1 port 0x2000006

2019-11-22T11:11:46.280Z cpu46:2100329)Vmxnet3: 18810: Using default queue delivery for vmxnet3 for port 0x2000006

2019-11-22T11:11:46.280Z cpu46:2100329)NetPort: 1359: enabled port 0x2000006 with mac 00:50:56:ae:a2:5c

2019-11-22T11:11:56.881Z cpu6:2099947 opID=2ba9ba6)World: 11943: VC opID vim-cmd-a9-5c89-5ce6 maps to vmkernel opID 2ba9ba6

2019-11-22T11:11:56.881Z cpu6:2099947 opID=2ba9ba6)Config: 703: "SIOControlFlag2" = 1, Old Value: 0, (Status: 0x0)

2019-11-22T11:11:56.952Z cpu40:2100477)MemSched: vm 2100477: 5745: extended swap to 20480 pgs

2019-11-22T11:11:57.013Z cpu41:2100477)World: vm 2100478: 7076: Starting world vmm0:SDC-1 of type 8

2019-11-22T11:11:57.013Z cpu41:2100477)Sched: vm 2100478: 6196: Adding world 'vmm0:SDC-1', group 'host/user', cpu: shares=-3 min=0 minLimit=-1 max=-1, mem: shares=-3 min=0 minLimit=-1 max=-1

2019-11-22T11:11:57.013Z cpu41:2100477)Sched: vm 2100478: 6211: renamed group 13807 to vm.2100477

2019-11-22T11:11:57.013Z cpu41:2100477)Sched: vm 2100478: 6228: group 13807 is located under group 4

2019-11-22T11:11:57.024Z cpu41:2100477)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 0

2019-11-22T11:11:57.058Z cpu41:2100477)VSCSI: 3810: handle 8206(vscsi0:0):Creating Virtual Device for world 2100478 (FSS handle 657971) numBlocks=209715200 (bs=512)

2019-11-22T11:11:57.058Z cpu41:2100477)VSCSI: 273: handle 8206(vscsi0:0):Input values: res=0 limit=-2 bw=-1 Shares=1000

2019-11-22T11:11:57.080Z cpu28:2100478)VMMVMKCall: 244: Received INIT from world 2100478

2019-11-22T11:11:57.081Z cpu28:2100478)LSI: 1781: LSI: Initialized rings for scsi0 async=1

2019-11-22T11:11:57.083Z cpu28:2100486)Net: 2456: connected SDC-1 eth0 to VM Network, portID 0x2000007

2019-11-22T11:11:57.083Z cpu28:2100486)NetPort: 1359: enabled port 0x2000007 with mac 00:00:00:00:00:00

2019-11-22T11:11:57.085Z cpu21:2099354)Config: 703: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0)

2019-11-22T11:11:57.088Z cpu28:2100478)----------------  ----------------  ----------------  ----------------  ----------------

2019-11-22T11:11:57.088Z cpu28:2100478)            name       reserved,KB   peakReserved,KB          alloc,KB      peakAlloc,KB

2019-11-22T11:11:57.088Z cpu28:2100478)----------------  ----------------  ----------------  ----------------  ----------------

2019-11-22T11:11:57.088Z cpu28:2100478)             VMM             12620             12620              4792              4792

2019-11-22T11:11:57.088Z cpu28:2100478)          PFRAME              4108              4108                12                12

2019-11-22T11:11:57.088Z cpu28:2100478)      SWAP_CACHE               512               512               512               512

2019-11-22T11:11:57.088Z cpu28:2100478)       CANDIDATE                 4                 4                 4                 4

2019-11-22T11:11:57.088Z cpu28:2100478)  CHECKPOINT_BUF               256               256                 0                 0

2019-11-22T11:11:57.088Z cpu28:2100478)      PSHARE_P2M                20                20                20                20

2019-11-22T11:11:57.088Z cpu28:2100478)            CBRC                 0                 0                 0                 0

2019-11-22T11:11:57.088Z cpu28:2100478)           FTCPT                 0                 0                 0                 0

2019-11-22T11:11:57.088Z cpu28:2100478)      ASYNCREMAP                 4                 4                 4                 4

2019-11-22T11:11:57.088Z cpu28:2100478)   VMMEMSERVICES                64                64                64                64

2019-11-22T11:11:57.088Z cpu28:2100478)FSR_UNSHARED_GBL                 0                 0                 0                 0

2019-11-22T11:11:57.088Z cpu28:2100478)----------------  ----------------  ----------------  ----------------  ----------------

2019-11-22T11:11:57.088Z cpu28:2100478)------------------------------  ------------  ------------  ------------  ------------  ------------

2019-11-22T11:11:57.088Z cpu28:2100478)                          name        min,KB        max,KB   minLimit,KB       eMin,KB   rMinPeak,KB

2019-11-22T11:11:57.088Z cpu28:2100478)------------------------------  ------------  ------------  ------------  ------------  ------------

2019-11-22T11:11:57.088Z cpu28:2100478)                    vm.2100477             0            -1            -1         32608         34332

2019-11-22T11:11:57.088Z cpu28:2100478)            worldGroup.2100477             0            -1            -1           760           760

2019-11-22T11:11:57.088Z cpu28:2100478)                    uw.2100477          8624            -1            -1          8624         10416

2019-11-22T11:11:57.088Z cpu28:2100478)               vsiHeap.2100477             0            -1            -1           136           136

2019-11-22T11:11:57.088Z cpu28:2100478)                    pt.2100477           224            -1            -1           556           556

2019-11-22T11:11:57.088Z cpu28:2100478)            cartelheap.2100477             0            -1            -1           296           296

2019-11-22T11:11:57.088Z cpu28:2100478)             uwshmempt.2100477             0            -1            -1             0             0

2019-11-22T11:11:57.088Z cpu28:2100478)      uwAsyncRemapHeap.2100477             0            -1            -1           136           136

2019-11-22T11:11:57.088Z cpu28:2100478)               uwCrypt.2100477          4096          4096          4096          4096          4096

2019-11-22T11:11:57.088Z cpu28:2100478)              uwregbmp.2100477             8            -1            -1             8             8

2019-11-22T11:11:57.088Z cpu28:2100478)                   vmm.2100477             0            -1            -1             0             0

2019-11-22T11:11:57.088Z cpu28:2100478)               vmmanon.2100477         17588            -1            -1         17588         17588

2019-11-22T11:11:57.088Z cpu28:2100478)                 vmcpt.2100478           144            -1            -1           144           144

2019-11-22T11:11:57.088Z cpu28:2100478)             vmmregbmp.2100477           128            -1            -1           128           128

2019-11-22T11:11:57.088Z cpu28:2100478)     vmmAsyncRemapHeap.2100477             0            -1            -1           136           136

2019-11-22T11:11:57.088Z cpu28:2100478)------------------------------  ------------  ------------  ------------  ------------  ------------

2019-11-22T11:11:58.272Z cpu28:2100478)VSCSI: 2623: handle 8206(vscsi0:0):Reset request on FSS handle 657971 (0 outstanding commands) from (vmm0:SDC-1)

2019-11-22T11:11:58.272Z cpu2:2097640)VSCSI: 2903: handle 8206(vscsi0:0):Reset [Retries: 0/0] from (vmm0:SDC-1)

2019-11-22T11:11:58.272Z cpu2:2097640)lsi_mr3: mfi_TaskMgmt:662: Processing taskMgmt virt reset for device: vmhba2:C2:T0:L0

2019-11-22T11:11:58.272Z cpu2:2097640)lsi_mr3: mfi_TaskMgmt:667: Virtual Reset request from Wld 2100478

2019-11-22T11:11:58.272Z cpu2:2097640)lsi_mr3: mfi_VirtReset:619: VIRT_REST completed. Initiator ID 0x4308131803c0 World ID 2100478

2019-11-22T11:11:58.272Z cpu2:2097640)VSCSI: 2691: handle 8206(vscsi0:0):Completing reset (0 outstanding commands)

2019-11-22T11:11:58.272Z cpu28:2100478)VSCSI: 2623: handle 8206(vscsi0:0):Reset request on FSS handle 657971 (0 outstanding commands) from (vmm0:SDC-1)

2019-11-22T11:11:58.272Z cpu2:2097640)VSCSI: 2903: handle 8206(vscsi0:0):Reset [Retries: 0/0] from (vmm0:SDC-1)

2019-11-22T11:11:58.272Z cpu2:2097640)lsi_mr3: mfi_TaskMgmt:662: Processing taskMgmt virt reset for device: vmhba2:C2:T0:L0

2019-11-22T11:11:58.272Z cpu2:2097640)lsi_mr3: mfi_TaskMgmt:667: Virtual Reset request from Wld 2100478

2019-11-22T11:11:58.272Z cpu2:2097640)lsi_mr3: mfi_VirtReset:619: VIRT_REST completed. Initiator ID 0x4308131803c0 World ID

[Finikiez]

Если я правильно понимаю, то чтобы решить проблему вы перезагружаете сервер.

Логи, которые вы вставляете также похожи на логи, при старте ВМ после перезапуска гипервизора.

Прикрепите более ранние логи за момент наблюдения проблемы _До_ перезагрузки сервера.

[AlexFail]

Я правильно понимаю, нужен лог vmkwarning?

Спасибо.

[AlexFail]

vmkwarning до перезагузки

2019-11-25T10:37:36.036Z cpu32:2098977)WARNING: NTPClock: 1259: system clock stepped to 1574678255.894764000, no longer synchronized to upstream time servers

2019-11-25T10:37:35.894Z cpu32:2098977)WARNING: NTPClock: 1561: system clock synchronized to upstream time servers

0:00:00:05.218 cpu0:2097152)WARNING: VMKAcpi: 318: \_SB_.PC00.LPC0.TMR_: skipping GSIV 0 conflict

0:00:00:05.269 cpu0:2097152)WARNING: Chipset: 396: Bus 5 (04) is already defined

2019-11-25T11:29:22.687Z cpu31:2097846)WARNING: Failed to init interrupt.

2019-11-25T11:29:22.836Z cpu31:2097846)WARNING: Failed to init interrupt.

2019-11-25T11:29:25.973Z cpu18:2097861)WARNING: ScsiPath: 8915: Adapter Invalid does not exist

2019-11-25T11:29:25.973Z cpu39:2097863)WARNING: PCI: 1209: 0000:00:14.0 is nameless

2019-11-25T11:29:30.107Z cpu47:2097926)WARNING: etherswitch: PortCfg_ModInit:910: Skipped initializing etherswitch portcfg for VSS to use cswitch and portcfg module

2019-11-25T11:29:34.259Z cpu44:2098077)WARNING: FBFT not enabled

2019-11-25T11:29:39.322Z cpu35:2097791)WARNING: ScsiDeviceIO: 10107: Mode Sense cmd reported block size 0, does not match the current logical block size 512(with physical block size 512) for device.

2019-11-25T11:29:39.322Z cpu35:2097791)WARNING: ScsiDeviceIO: 10109: The device mpx.vmhba32:C0:T0:L0 is marked format corrupt.

2019-11-25T11:29:43.839Z cpu35:2097791)WARNING: NFS: 1227: Invalid volume UUID 5d9b4bb4-89590e3c-9b54-ac1f6bd20130

2019-11-25T11:29:45.329Z cpu8:2098727)WARNING: APEI: 319: Could not initialize EINJ

2019-11-25T11:29:55.947Z cpu43:2098995)WARNING: NTPClock: 1561: system clock synchronized to upstream time servers

2019-11-25T11:30:12.479Z cpu31:2100058)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 0

2019-11-25T11:30:12.479Z cpu31:2100058)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 1

2019-11-25T11:30:42.561Z cpu34:2100312)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 0

2019-11-25T11:30:42.561Z cpu34:2100312)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 1

2019-11-25T11:31:12.561Z cpu46:2100490)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 0

2019-11-25T11:31:42.582Z cpu29:2100645)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 0

2019-11-25T11:31:42.582Z cpu29:2100645)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 1

2019-11-25T11:31:42.582Z cpu29:2100645)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 2

2019-11-25T11:31:42.582Z cpu29:2100645)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 3

2019-11-25T11:31:42.582Z cpu29:2100645)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 4

2019-11-25T11:31:42.582Z cpu29:2100645)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 5

2019-11-25T11:31:42.582Z cpu29:2100645)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 6

2019-11-25T11:31:42.582Z cpu29:2100645)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 7

2019-11-25T11:31:42.582Z cpu29:2100645)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 8

2019-11-25T11:31:42.583Z cpu29:2100645)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 9

2019-11-25T11:31:42.583Z cpu29:2100645)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 10

2019-11-25T11:31:42.583Z cpu29:2100645)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 11

2019-11-25T11:37:49.029Z cpu4:2098995)WARNING: NTPClock: 1561: system clock synchronized to upstream time servers

2019-11-25T11:43:11.084Z cpu27:2101074)WARNING: MonLoader: 734: MonLoaderCallout_GetSharedHostPage: Invalid page offset 0 for region 8 vcpu 0

2019-11-25T12:16:36.814Z cpu39:2101600)WARNING: FSS: 7801: failed to query file handle id 1707693

2019-11-25T12:16:36.814Z cpu39:2101600)WARNING: FSS: 7801: failed to query file handle id 1642163

2019-11-25T12:16:40.946Z cpu12:2101572)WARNING: PCI: 179: 0000:00:00.0: Bypassing non-ACS capable device in hierarchy

2019-11-25T12:16:41.032Z cpu12:2101572)WARNING: PCI: 179: 0000:00:1c.0: Bypassing non-ACS capable device in hierarchy

2019-11-25T12:16:41.036Z cpu12:2101572)WARNING: PCI: 179: 0000:00:1c.4: Bypassing non-ACS capable device in hierarchy

2019-11-25T12:16:41.040Z cpu12:2101572)WARNING: PCI: 179: 0000:00:1c.5: Bypassing non-ACS capable device in hierarchy

2019-11-25T12:16:41.064Z cpu12:2101572)WARNING: PCI: 179: 0000:03:00.0: Bypassing non-ACS capable device in hierarchy

2019-11-25T12:16:41.070Z cpu12:2101572)WARNING: PCI: 179: 0000:04:00.0: Bypassing non-ACS capable device in hierarchy

2019-11-25T12:16:44.810Z cpu26:2101600)WARNING: PCI: 179: 0000:00:00.0: Bypassing non-ACS capable device in hierarchy

2019-11-25T12:16:44.916Z cpu26:2101600)WARNING: PCI: 179: 0000:00:1c.0: Bypassing non-ACS capable device in hierarchy

2019-11-25T12:16:44.920Z cpu26:2101600)WARNING: PCI: 179: 0000:00:1c.4: Bypassing non-ACS capable device in hierarchy

2019-11-25T12:16:44.924Z cpu26:2101600)WARNING: PCI: 179: 0000:00:1c.5: Bypassing non-ACS capable device in hierarchy

2019-11-25T12:16:44.953Z cpu26:2101600)WARNING: PCI: 179: 0000:03:00.0: Bypassing non-ACS capable device in hierarchy

2019-11-25T12:16:44.959Z cpu26:2101600)WARNING: PCI: 179: 0000:04:00.0: Bypassing non-ACS capable device in hierarchy

2019-11-25T12:16:52.336Z cpu12:2101572)WARNING: cswitch: VLAN_PortGetVLANData:397: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x2000001

2019-11-25T12:16:52.336Z cpu12:2101572)WARNING: cswitch: VLAN_PortGetVLANData:397: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x2000002

2019-11-25T12:16:52.336Z cpu12:2101572)WARNING: cswitch: VLAN_PortGetVLANData:397: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x2000003

2019-11-25T12:16:52.337Z cpu12:2101572)WARNING: cswitch: VLAN_PortGetVLANData:397: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x3000001

2019-11-25T12:16:52.337Z cpu12:2101572)WARNING: cswitch: VLAN_PortGetVLANData:397: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x3000002

2019-11-25T12:16:52.338Z cpu12:2101572)WARNING: cswitch: VLAN_PortGetVLANData:397: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x3000003

2019-11-25T12:16:52.339Z cpu12:2101572)WARNING: Tcpip_Vmk: 846: vmk_get_gateway failed with error = 0x2d, status = 0xbad0105

2019-11-25T12:16:58.360Z cpu24:2101600)WARNING: cswitch: VLAN_PortGetVLANData:397: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x2000001

2019-11-25T12:16:58.360Z cpu24:2101600)WARNING: cswitch: VLAN_PortGetVLANData:397: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x2000002

2019-11-25T12:16:58.360Z cpu24:2101600)WARNING: cswitch: VLAN_PortGetVLANData:397: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x2000003

2019-11-25T12:16:58.361Z cpu24:2101600)WARNING: cswitch: VLAN_PortGetVLANData:397: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x3000001

2019-11-25T12:16:58.361Z cpu24:2101600)WARNING: cswitch: VLAN_PortGetVLANData:397: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x3000002

2019-11-25T12:16:58.361Z cpu24:2101600)WARNING: cswitch: VLAN_PortGetVLANData:397: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x3000003

2019-11-25T12:16:58.363Z cpu24:2101600)WARNING: Tcpip_Vmk: 846: vmk_get_gateway failed with error = 0x2d, status = 0xbad0105

[AlexFail]

Либо могу приложить support bundle, если это поможет.

[Finikiez]

Давайте посмотрим на весь support bundle.

vmkwarning - это выжимка из vmkernel с сообщениями уровня warning. Я обычно смотрю vmkernel.

[AlexFail]

Выгрузил на gdrive.

https://drive.google.com/open?id=1hyzEAK--wpZJvDOwbIAopvb7Mx0iDozT

[AlexFail]

Есть у кого идеи? Попробовал поменять у всех вм lan адаптер с e1000e  на vmxnet3, результата не принесло.

[Finikiez]

Добрый день!

У вас все проблемы начинаются со строчки

2019-11-25T11:25:15.817Z cpu24:2097883)i40en: i40en_HandleMddEvent:6567: Malicious Driver Detection event 0x12 on TX queue 0 PF number 0x00 VF number 0x00

2019-11-25T11:25:15.817Z cpu24:2097883)i40en: i40en_HandleMddEvent:6593: TX driver issue detected, PF reset issued

Я думаю, что это чем-то напоминает статью https://kb.vmware.com/s/article/59568

Попробуйте обновить драйвер i40en и прошивку.

Сейчас у вас

   Driver Info:

      NICDriverInfo:

         Bus Info: 0000:1a:00:0

         Driver: i40en

         Firmware Version: 3.31 0x80000c92 1.1747.0

         Version: 1.3.1

[AlexFail]

Добрый день!

Спасибо большое за помощь, обновил драйвер, второй день - полет нормальный.

Не смог обновить прошивку сетевого адаптера x722 intel, вы не могли бы подсказать как это сделать?

[Finikiez]

Способ обновления прошивки сетевой карты зависит от производителя сервера. Обычно энтерпрайз вендоры (Lenovo, Dell, HPE и т.д.) выпускают загрузочные ISO образы или утилиты для Windows\Linux, с помощью которых можно провести обновление.

Что делать в случае с Supermicro, я, к сожалению, сходу не подскажу.
Попробуйте на их сайте поискать прошивки. Обычно в архивах также лежат инструкции как и что делать.