vSphere Storage Appliance

 View Only

  • 1.  ESXTOP & Queues

    Posted Oct 18, 2012 08:49 PM

    Hi

    It drives me slightly crazy, maybe you can help me out...

    Based on KB: 1267

    I set the queue to 64 and reviewed it:

    l2xmaxqdepth               int   64     Maximum queue depth to report for target devices.

    cat /proc/scsi/qla2xxx/8

    Device queue depth = 0x40 = 64

    So thats the device driver queue also known als LUN queue, correct?

    But then there are some other queues:

    esxtop - d:disk adapter:

                                            AQLEN

    vmhba0 -                       0    31      0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

    vmhba2 -                       1   975     5.99     0.00     5.99     0.00     0.02     0.00     0.00     0.06     0.01     0.07     0.00

    vmhba3 -                       0  2176     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

    The value listed under AQLEN is the queue depth of the storage adapter. This is the maximum number of ESX VMKernel active commands that the adapter driver is configured to support.

    esxtop - u: disk device

                                                         DQLEN

    mpx.vmhba32:C0:T0:L0       -               1     -    0    0    0  0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

    mpx.vmhba34:C0:T0:L0       -               1     -    0    0    0  0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

    naa.xxxxxxxxxxxxxxxx       -            128     -    0    0    0  0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

    The value listed under DQLEN is the queue depth of the storage device. This is the maximum number of ESX VMKernel active commands that the device is configured to support.

    And last but not least:

    http://communities.vmware.com/docs/DOC-6490

    The kernel queue can be thought of as kind of an overflow queue for the  device driver queues.

    AQLEN means that the HBA can handle up to 2176 active commands.

    DQLEN limits the commands per storage device.

    l2xmaxqdepth limits the active commands per LUN.

    Where is the difference? Shouldn't be the values the same? This is where I'm confused the most!!

    So basiacally I can multiply the LUN queue with the number of connected datastores to know how much active commands my host is allowed to issue down to the storage. This value will be limit by AQLEN to 2176 per vmhba.

    But wait, there is more!?

    I can also limit the number of commands within the HBA BIOS.

    Here I can set the "Execution throttle" which will further reduce the AQLEN down to let's say 256 or even lower.

    Regards

    Patrick



  • 2.  RE: ESXTOP & Queues

    Posted Oct 29, 2012 09:04 AM

    No ideas?



  • 3.  RE: ESXTOP & Queues

    Posted Nov 01, 2012 10:02 AM

    Do you have Storage IO Control enabled?



  • 4.  RE: ESXTOP & Queues

    Posted Nov 01, 2012 11:02 AM

    Just booted my lab.

    At the first glance esxtop showed me a DQLEN of 32 for all disks. Then I verified that SIOC is diabled on all datastores. Afterwards I checked esxtop again and one disk showed me a DQLEN of 128, the other disks remained on 32.

    //EDIT: The one disk who is flapping between 128 and 32 is a local RAID volume. So the l2xmaxqdepth/qla2xxx setting should'nt take effect here at all!? I completly overlooked this.

    I guess the last time I set the value to 32 and rebooted the system the changed didn't take effect, why ever. Now the DQLEN and the setting in the qla2xxx drivers are identical and I’m less confused.

    But as long as SOIC is disabled, the DQLEN should be the same as l2xmaxqdepth.

    If SIOC is enabled, SOIC can dynamically ajust the DQLEN value?

    Regards

    Patrick



  • 5.  RE: ESXTOP & Queues

    Broadcom Employee
    Posted Nov 02, 2012 01:21 PM

    For LUNs presented from a SAN via a HBA, the DQLEN will 32 by default.

    For the LUN with a DQLEN of 128, is this a local disk?

    You can also see different DQLEN in the case of 'gateway' disks, for instance EMC's LUNZ devices.

    HTH

    Cormac



  • 6.  RE: ESXTOP & Queues

    Posted Nov 02, 2012 06:16 PM

    Cormac schrieb:

    For the LUN with a DQLEN of 128, is this a local disk?

    Yes.

    But why is it flapping between 32 and 128?

    Patrick



  • 7.  RE: ESXTOP & Queues

    Broadcom Employee
    Posted Nov 05, 2012 04:04 PM

    I've not seen this before.

    Are you sure it is the same device and not just an esxtop refresh issue where the NAA ids are changing order/place?

    To answer your other questions, the change to the HBA device queue depth will not have any effect on your local disk which uses a compeltely different driver.

    My only other thought was adaptive queue settings - but what you describe as 'flapping' does not match the behaviour of adaptive queue depths - see KB http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008113

    HTH

    Cormac



  • 8.  RE: ESXTOP & Queues

    Posted Nov 01, 2012 10:46 AM

    Patrick wrote:

    AQLEN means that the HBA can handle up to 2176 active commands.

    DQLEN limits the commands per storage device.

    l2xmaxqdepth limits the active commands per LUN.

    Where is the difference? Shouldn't be the values the same? This is where I'm confused the most!!

    So basiacally I can multiply the LUN queue with the number of connected datastores to know how much active commands my host is allowed to issue down to the storage. This value will be limit by AQLEN to 2176 per vmhba.


    I agree it is confusing but keep in mind:

    • 1 HBA --> many LUNs
    • 1 LUN --> many hosts / HBAs

    If I can find the time I will try to describe this a bit better in a blog post.