PowerCLI

 View Only
Expand all | Collapse all

Monitor packet drops across all ESXi hosts

  • 1.  Monitor packet drops across all ESXi hosts

    Posted Feb 12, 2013 06:53 PM

    Hi All

    I need to remotely monitor our ESXi hosts to check for packets being dropped on each NIC on each host, maybe each hour, as part of an investigation we're doing into some problems we're having.

    I know there is the vsish command, but does anyone have an example they can share to capture the details remotely from all hosts in the environment ?  I need to know which NICs are dropping packets, and when...

    Many thanks

    Simon



  • 2.  RE: Monitor packet drops across all ESXi hosts

    Posted Feb 12, 2013 07:07 PM

    Since you posted in the PowerCLI community, would a PowerCLI solution be acceptable ?

    If yes, you could use the Get-Stat cmdlet and retrieve the net.dropperdRx.summation and net.droppedTx.summation counters.

    Would that be a viable solution for you ?

    Let me know if you need a sampe script ?



  • 3.  RE: Monitor packet drops across all ESXi hosts

    Posted Feb 12, 2013 07:19 PM

    Hi Luc

    So I could get that direct from Virtual Centre ?  Excellent news.  I guess the logging level would need to be raised to 3 or 4 ?

    A sample script would be much appreciated :smileyhappy:

    Regards

    Simon



  • 4.  RE: Monitor packet drops across all ESXi hosts

    Posted Feb 12, 2013 07:49 PM

    I would propose to get the counters directly from the ESXi servers, then you don't need to bother about statistics levels.

    Something like this would show for each vmnic on an ESXi how many packets were dropped (send & receive) during the interval (which is 20 seconds for Realtime).

    $esxName = "esx1.local.test" 
    $start
    = (Get-Date).AddMinutes(-2) $metrics = "net.droppedRx.summation","net.droppedTx.summation"
    $esx
    = Get-VMHost -Name $esxName
    Get-Stat
    -Entity $esx -Stat $metrics -Realtime -Start $start |
    where
    {$_.Instance -ne ""} |
    Group-Object
    -Property Instance | %{     $ordered = $_.Group | Sort-Object -Property Timestamp,MetricId
        for($i = 2; $i -lt $ordered.Count;$i = $i +2){         New-Object PSObject -Property @{
               
    VMHost = $ordered[$i].Entity.Name
               
    VmNic = $ordered[$i].Instance
               
    "Receive Dropped Packets" = $ordered[$i].Value - $ordered[$i - 2].Value
                "Transmit Dropped Packets" = $ordered[$i + 1].Value - $ordered[$i - 1].Value
                Timestamp = $ordered[$i].Timestamp
               
    "Interval (seconds)" = $ordered[$i].IntervalSecs
            }     } } | ft -AutoSize

    The script looks at the last 2 minutes.



  • 5.  RE: Monitor packet drops across all ESXi hosts

    Posted Feb 12, 2013 09:14 PM

    You are simply brilliant Mr LucD!



  • 6.  RE: Monitor packet drops across all ESXi hosts

    Posted Feb 12, 2013 10:34 PM

    That works well - may thanks.

    I've changed the addminutes to 60, and have a loop cycling through the ESX hosts.

    Could the script be changed to ignore a line if the dropped packets was 0 ?



  • 7.  RE: Monitor packet drops across all ESXi hosts

    Posted Feb 13, 2013 03:29 AM

    Sure, something like this should do the trick

    $esxName = "esx1.local.test" 
    $start = (Get-Date).AddMinutes(-2)
    
    $metrics = "net.droppedRx.summation","net.droppedTx.summation" 
    $esx = Get-VMHost -Name $esxName
    
    Get-Stat
    -Entity $esx -Stat $metrics -Realtime -Start $start | where {$_.Instance -ne ""} | Group-Object -Property Instance | %{     $ordered = $_.Group | Sort-Object -Property Timestamp,MetricId
        for($i = 2; $i -lt $ordered.Count;$i = $i +2){         $droppedRx = $ordered[$i].Value - $ordered[$i - 2].Value
            $droppedTx = $ordered[$i + 1].Value - $ordered[$i - 1].Value
            if($droppedRx -ne 0 -and $droppedTx -ne 0){             New-Object PSObject -Property @{                 VMHost = $ordered[$i].Entity.Name
                   
    VmNic = $ordered[$i].Instance
                   
    "Receive Dropped Packets" = $droppedRx
                   
    "Transmit Dropped Packets" = $droppedTx
                   
    Timestamp = $ordered[$i].Timestamp
                   
    "Interval (seconds)" = $ordered[$i].IntervalSecs
                }         }     } }
    | ft -AutoSize


  • 8.  RE: Monitor packet drops across all ESXi hosts

    Posted Feb 13, 2013 03:45 AM

    Hi Luc,

    Thanks for sharing the script here, how can this be changed to search through the multiple ESXi hosts from Get-VMHost instead of hard code it manually one by one ?



  • 9.  RE: Monitor packet drops across all ESXi hosts

    Posted Feb 13, 2013 04:03 AM

    That is quite easy actually, just add the hostname as an extra property on the Grou-Object cmdlet.

    Something like this

    $start = (Get-Date).AddMinutes(-2)
    
    $metrics = "net.droppedRx.summation","net.droppedTx.summation" 
    $esx = Get-VMHost
    
    Get-Stat
    -Entity $esx -Stat $metrics -Realtime -Start $start | where {$_.Instance -ne ""} | Group-Object -Property {$_.Entity.Name,$_.Instance} | %{     $ordered = $_.Group | Sort-Object -Property Timestamp,MetricId
        for($i = 2; $i -lt $ordered.Count;$i = $i +2){         $droppedRx = $ordered[$i].Value - $ordered[$i - 2].Value
            $droppedTx = $ordered[$i + 1].Value - $ordered[$i - 1].Value
            if($droppedRx -ne 0 -and $droppedTx -ne 0){             New-Object PSObject -Property @{                 VMHost = $ordered[$i].Entity.Name
                   
    VmNic = $ordered[$i].Instance
                    "Receive Dropped Packets" = $droppedRx
                   
    "Transmit Dropped Packets" = $droppedTx
                   
    Timestamp = $ordered[$i].Timestamp
                   
    "Interval (seconds)" = $ordered[$i].IntervalSecs
                }         }     } }
    | ft -AutoSize


  • 10.  RE: Monitor packet drops across all ESXi hosts

    Posted Feb 13, 2013 10:00 AM

    Perfect - thanks again Luc.

    Albert, if you're interested, this is the script I've made from what Luc gave me.  Its scheduled to run hourly, and I can just sort the reports folder for any files with size>0 to identify where packets are being lost across all hosts.  It needs a file listing all ESXi hosts called serverlist.txt in the same folder...

    add-pssnapin VMware.VimAutomation.Core


    $inputfile = "F:\Droppedpackets\Serverlist.txt"

    $Date = Get-Date

    $connectioninfo = Import-Csv $inputfile

    ForEach ($Server in $connectioninfo) {

    $esxName = $Server.Management_Server

    Write-Host $esxName

    $outputfile = "F:\Droppedpackets\Reports\" + $esxName + "_"  + $Date.Year  + "-" + $Date.Month + "-" + $Date.Day + "_at_" + $Date.Hour + ".csv"

    Connect-VIServer $esxName

    $start = (Get-Date).AddMinutes(-60)

    $metrics = "net.droppedRx.summation","net.droppedTx.summation"
    $esx = Get-VMHost -Name $esxName

    Get-Stat -Entity $esx -Stat $metrics -Realtime -Start $start |
    where {$_.Instance -ne ""} |
    Group-Object -Property Instance | %{
        $ordered = $_.Group | Sort-Object -Property Timestamp,MetricId
        for($i = 2; $i -lt $ordered.Count;$i = $i +2){
            $droppedRx = $ordered[$i].Value - $ordered[$i - 2].Value
            $droppedTx = $ordered[$i + 1].Value - $ordered[$i - 1].Value
            if($droppedRx -ne 0 -and $droppedTx -ne 0){
                New-Object PSObject -Property @{
                    VMHost = $ordered[$i].Entity.Name
                    VmNic = $ordered[$i].Instance
                    "Receive Dropped Packets" = $droppedRx
                    "Transmit Dropped Packets" = $droppedTx
                    Timestamp = $ordered[$i].Timestamp
                    "Interval (seconds)" = $ordered[$i].IntervalSecs
                }
            }
        }
    } | Export-Csv -NoTypeInformation -Append $outputfile

    Disconnect-VIServer -Confirm:$False

    }



  • 11.  RE: Monitor packet drops across all ESXi hosts

    Posted Feb 14, 2013 11:07 AM

    Hi Luc

    There seems to be a problem with this script - when it detects packet drops, it put -1 in the dropped packets fields ?

    Transmit Dropped   PacketsTimestampReceive Dropped Packets
    114/02/2013 01:351
    -114/02/2013 01:35-1
    114/02/2013 01:411
    -114/02/2013 01:41-1

    Any idea why ?

    Regards

    Simon



  • 12.  RE: Monitor packet drops across all ESXi hosts

    Posted Feb 14, 2013 11:52 AM

    My mistake, I interpreted the summation as a cummulative summation while it is in fact a summation over the interval.

    Try with this version

    $esxName = "esx1.local.test" 
    $start = (Get-Date).AddMinutes(-60)
    
    $metrics = "net.droppedRx.summation","net.droppedTx.summation" 
    $esx = Get-VMHost -Name $esxName
    
    Get-Stat
    -Entity $esx -Stat $metrics -Realtime -Start $start | where {$_.Instance -ne ""} | Group-Object -Property Instance | %{     $ordered = $_.Group | Sort-Object -Property Timestamp,MetricId
       
    for($i = 0; $i -lt $ordered.Count;$i = $i +2){         $droppedRx = $ordered[$i].Value
            $droppedTx = $ordered[$i + 1].Value
            if($droppedRx -ne 0 -or $droppedTx -ne 0){             New-Object PSObject -Property @{                 VMHost = $ordered[$i].Entity.Name
                   
    VmNic = $ordered[$i].Instance
                   
    "Receive Dropped Packets" = $droppedRx
                   
    "Transmit Dropped Packets" = $droppedTx
                   
    Timestamp = $ordered[$i].Timestamp
                    "Interval (seconds)" = $ordered[$i].IntervalSecs
                }         }     } } | ft -AutoSize

    Update: just noticed that the test for 0 values needs to have an OR instead of an AND.

    If 1 of the 2 values is not 0, the script needs to display the record.



  • 13.  RE: Monitor packet drops across all ESXi hosts

    Posted Sep 25, 2014 08:02 PM

    Hi LucD - script is great.   In the environment I'm currently working there are over 100 VMs per host, and we have over 100 hosts globally. 

    Is it possible to show how this can be modified to query all the VMs in a cluster and report the VMs actually dropping recieve packets and the # of packets dropped?

    I've toyed a bit too long with getting this modified to query VMs and it fails miserably time and time again, and I'm beginning to prove I'm not very good at scripting.  :smileysad:



  • 14.  RE: Monitor packet drops across all ESXi hosts

    Posted Sep 25, 2014 08:35 PM

    I use something like the following to get the dropped packets per VM for all the ESXi hosts.

    It allows the definition of a start and finish time.

    Note that the vNICs will be displayed as the interval deviceID of these vNICs on the VM.

    You could fetch the VM and translate the deviceID into the label like "Network Adapter 1", but that would only work if the VM still exists and if the vNIC configuration hasn't changed in between.

    $start = Get-Date "22/09/14 11:30"

    $finish = Get-Date "22/09/14 17:30"

    $metrics = "net.droppedRx.summation","net.droppedTx.summation"

    foreach($esx in (Get-VMHost)){

        $vms = Get-VM -Location $esx

        if($vms){

            Get-Stat -Entity $vms -Stat $metrics -Start $start -Finish $finish -ErrorAction SilentlyContinue |

            where {$_.Instance -ne ""} |

            Group-Object -Property {$_.Entity.Name,$_.Instance} | %{

                $_.Group | Group-Object -Property Timestamp | %{

                    New-Object PSObject -Property @{

                        VMHost = $esx.Name

                        VM = $_.Group[0].Entity.Name

                        VmNic = $_.Group[0].Instance

                        "Receive Dropped Packets" = $_.Group | where {$_.MetricId -eq "net.droppedRx.summation"} |

                            Select -ExpandProperty Value

                        "Transmit Dropped Packets" = $_.Group | where {$_.MetricId -eq "net.droppedTx.summation"} |

                            Select -ExpandProperty Value

                        Timestamp = $_.Group[0].Timestamp

                        "Interval (seconds)" = $_.Group[0].IntervalSecs

                    }

                }

            }

        }

    }