ESXi

 View Only
  • 1.  Unified vMotion vmk usage?

    Posted Dec 29, 2012 12:26 AM

    Has anyone else noticed when migrating a vm between 2 ESXi 5.1 hosts which do not have shared storage that the storage copy portion of the vmotion doesn't use the vmotion vmknic portgroups defined?

    I have the following vmk port groups on each host:

    vmk0 - management traffic only

    vmk1 - vMotion traffic only

    vmk2 - vMotion traffic only

    When I initiate a vmotion between hosts, I notice all the traffic is initially between vmk0 on both hosts until the storage copy is done.  vmk1/vmk2 are only active during the last few seconds of the vmotion when the memory is transferred.  (see below esxtop results)

    Wondering how to tell the the storage copy portion of the vmotion to use the vmotion vmknics I've defined so I can control the VLAN used as well as take advantage of multi-nic for the storage copy portion as well as the memory copy portion?

    The below shows typical esxtop information during the majority of the vmotion while the storage is being copied.  Notice it is all vmk0 traffic which is not even specified as a vmotion vmknic.

    11:52:42pm up 8 days  8:07, 349 worlds, 0 VMs, 0 vCPUs; CPU load average: 0.01, 0.01, 0.01

       PORT-ID              USED-BY  TEAM-PNIC DNAME              PKTTX/s  MbTX/s    PKTRX/s  MbRX/s %DRPTX %DRPRX

      33554433           Management        n/a vSwitch0              0.00    0.00       0.00    0.00   0.00   0.00

      33554434               vmnic0          - vSwitch0          29661.75   15.30   82627.22  934.17   0.00   0.00

      33554435     Shadow of vmnic0        n/a vSwitch0              0.00    0.00       0.00    0.00   0.00   0.00

      33554436                 vmk0     vmnic0 vSwitch0          29661.94   15.30   16530.30  900.88   0.00   0.00

      33554437               vmnic1          - vSwitch0              0.00    0.00      19.61    0.01   0.00   0.00

      33554438     Shadow of vmnic1        n/a vSwitch0              0.00    0.00       0.00    0.00   0.00   0.00

      33554439                 vmk1     vmnic0 vSwitch0              0.00    0.00       0.93    0.00   0.00   0.00

      33554440                 vmk2     vmnic1 vSwitch0              0.00    0.00       0.93    0.00   0.00   0.00

    The below shows typical esxtop information during the last few seconds of the vmotion while the memory is transfered.  Notice that vmk0 utilization drops and vmk1/vmk2 are active.

    11:55:12pm up 8 days  8:10, 371 worlds, 1 VMs, 0 vCPUs; CPU load average: 0.05, 0.03, 0.02

       PORT-ID              USED-BY  TEAM-PNIC DNAME              PKTTX/s  MbTX/s    PKTRX/s  MbRX/s %DRPTX %DRPRX

      33554433           Management        n/a vSwitch0              0.00    0.00       0.00    0.00   0.00   0.00

      33554434               vmnic0          - vSwitch0           7586.46    4.30   32505.95  373.34   0.00   0.00

      33554435     Shadow of vmnic0        n/a vSwitch0              0.00    0.00       0.00    0.00   0.00   0.00

      33554436                 vmk0     vmnic0 vSwitch0             28.61    0.49      34.74    0.09   0.00   0.00

      33554437               vmnic1          - vSwitch0           7546.34    3.80   32509.67  373.82   0.00   0.00

      33554438     Shadow of vmnic1        n/a vSwitch0              0.00    0.00       0.00    0.00   0.00   0.00

      33554439                 vmk1     vmnic0 vSwitch0           7558.23    3.81    6586.19  360.17   0.00   0.00

      33554440                 vmk2     vmnic1 vSwitch0           7546.71    3.80    6599.38  360.77   0.00   0.00

    vmk1/vmk2 on both hosts can reach each other via vmkping etc.  Logs from the receiving host show connections made on both vmk1/vmk2, yet the storage migration still only used vmk0 as shown above.  Only the memory copy uses vmk1/vmk2 which are the designated vmotion vmknics.

    2012-12-28T23:55:08.189Z cpu14:475203)Migrate: vm 475204: 3273: Setting VMOTION info: Dest ts = 1356738745905490, src ip = <172.21.165.196> dest ip = <172.21.165.195> Dest wid = 0 using UNSHARED swap
    2012-12-28T23:55:08.199Z cpu14:475203)Tcpip_Vmk: 1213: Affinitizing 172.21.165.195 to world 475212, Success
    2012-12-28T23:55:08.199Z cpu14:475203)VMotion: 2688: 1356738745905490 D: Set ip address '172.21.165.195' worldlet affinity to recv World ID 475212
    2012-12-28T23:55:08.199Z cpu14:475203)Hbr: 3308: Migration start received (worldID=475204) (migrateType=1) (event=0) (isSource=0) (sharedConfig=0)
    2012-12-28T23:55:08.347Z cpu0:8771)MigrateNet: vm 8771: 2061: Accepted connection from <::ffff:172.21.165.196>
    2012-12-28T23:55:08.347Z cpu0:8771)MigrateNet: vm 8771: 2131: dataSocket 0x410024909fc0 receive buffer size is 563272
    2012-12-28T23:55:08.347Z cpu2:475209)MigrateNet: 1165: 1356738745905490 D: Successfully bound connection to vmknic '172.21.165.195'
    2012-12-28T23:55:08.348Z cpu0:8771)MigrateNet: vm 8771: 2061: Accepted connection from <::ffff:172.21.165.196>
    2012-12-28T23:55:08.348Z cpu0:8771)MigrateNet: vm 8771: 2131: dataSocket 0x41002490ad20 receive buffer size is 563272
    2012-12-28T23:55:08.348Z cpu0:8771)VMotionUtil: 3184: 1356738745905490 D: Stream connection 1 added.
    2012-12-28T23:55:08.349Z cpu0:8771)MigrateNet: vm 8771: 2061: Accepted connection from <::ffff:172.22.165.196>
    2012-12-28T23:55:08.349Z cpu0:8771)MigrateNet: vm 8771: 2131: dataSocket 0x410024b23b10 receive buffer size is 563272
    2012-12-28T23:55:08.349Z cpu0:8771)VMotionUtil: 3184: 1356738745905490 D: Stream connection 2 added.
    2012-12-28T23:55:08.382Z cpu20:475212)XVMotion: 1768: Allocating pool 0.
    2012-12-28T23:55:08.383Z cpu20:475212)XVMotion: 584: Bitmap page: len = 16384, pgLen = 1, bitSet = 5199, bitClear = 11185.
    2012-12-28T23:55:08.383Z cpu20:475212)XVMotion: 584: Bitmap block: len = 131072, pgLen = 4, bitSet = 95824, bitClear = 35248.
    2012-12-28T23:55:10.567Z cpu20:475212)VMotionRecv: 570: 1356738745905490 D: Estimated network bandwidth 58.517 MB/s during disk copy.
    2012-12-28T23:55:11.730Z cpu2:8240)<6>tg3 : vmnic0: RX NetQ allocated on 1
    2012-12-28T23:55:11.730Z cpu2:8240)<6>tg3 : vmnic0: NetQ set RX Filter: 1 [d4:ae:52:b5:b3:63 0]
    2012-12-28T23:55:14.996Z cpu20:475212)VMotionRecv: 638: 1356738745905490 D: Estimated network bandwidth 215.124 MB/s during pre-copy
    2012-12-28T23:55:15.347Z cpu6:475203)Swap: vm 475204: 1185: Swapfile already initialized. No work to do.
    2012-12-28T23:55:15.375Z cpu21:475212)VMotionRecv: 2386: 1356738745905490 D: DONE paging in
    2012-12-28T23:55:15.375Z cpu21:475212)VMotionRecv: 2394: 1356738745905490 D: Estimated network bandwidth 218.587 MB/s during page-in
    2012-12-28T23:55:15.385Z cpu6:475203)VSCSI: 3734: handle 8198(vscsi0:0):Using sync mode due to sparse disks
    2012-12-28T23:55:15.385Z cpu6:475203)VSCSI: 3775: handle 8198(vscsi0:0):Creating Virtual Device for world 475204 (FSS handle 9566254)
    2012-12-28T23:55:15.577Z cpu9:475204)VMMVMKCall: 208: Received INIT from world 475204
    2012-12-28T23:55:15.579Z cpu9:475204)PVSCSI: 3070: scsi0: wdt=1 intrCoalescingMode=2 flags=0xf
    2012-12-28T23:55:15.580Z cpu9:475402)Net: 1858: connected .Windows_2008_R2_Standard_SP1 eth0 to OSS_1113, portID 0x200000c
    2012-12-28T23:55:15.586Z cpu9:475402)NetPort: 1380: enabled port 0x200000c with mac 00:50:56:a9:e3:c0
    2012-12-28T23:55:15.586Z cpu9:475204)VMotion: 5679: 1356738745905490 D: Received all changed pages.
    2012-12-28T23:55:15.588Z cpu9:475204)VmMemMigrate: vm 475204: 5005: Regular swap file bitmap checks out.
    2012-12-28T23:55:15.589Z cpu9:475204)VMotion: 5458: 1356738745905490 D: Resume handshake successful
    2012-12-28T23:55:15.593Z cpu9:475402)Hbr: 3405: Migration end received (worldID=475204) (migrateType=1) (event=1) (isSource=0) (sharedConfig=0)
    2012-12-28T23:55:15.593Z cpu18:475208)XVMotion: 584: Bitmap page: len = 16384, pgLen = 1, bitSet = 5199, bitClear = 11185.
    2012-12-28T23:55:15.593Z cpu18:475208)XVMotion: 584: Bitmap block: len = 131072, pgLen = 4, bitSet = 95824, bitClear = 35248.
    2012-12-28T23:55:15.603Z cpu7:475403)Swap: vm 475204: 3254: Starting prefault for the migration swap file
    2012-12-28T23:55:15.619Z cpu7:475403)Swap: vm 475204: 3429: Finish swapping in migration swap file. (faulted 0 pages, pshared 0 pages). Success.
    2012-12-28T23:55:15.839Z cpu12:9984)Config: 347: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0)

    Thank you |



  • 2.  RE: Unified vMotion vmk usage?

    Posted Dec 29, 2012 10:42 AM

    That's interesting. According to http://frankdenneman.nl/vmotion/vsphere-5-1-vmotion-deepdive/ this should not be the case!?

    Please provide the details of your network configuration (IP settings, VLANs, ... of Management and vMotion), maybe some screen shots will help.

    Which build of vCenter Server and ESXi do you use?

    André