High interrupts by NDIS.sys

[vExperts]

The interrupts are pretty high on Print server. Examined via Xperf and found that the ndis.sys is the highest consumer.

OS: win2003 SP2 R2 32bit.

NDIS version : 5.2.3790.3959

Type : Virtual machine with updated vmtools installed.

Used procmon and traced network activity – found idle process established a connection to several client machines.

Examined from manage>Mycomputer sessions and openfiles – users opened \PIPE\spoolss.

Used another tool pipelist.exe – the highest pipes are used by spools process.

We had been getting alerts with High CPU utilization, then we decided to add one more Vcpu (now we have total 2) and added.

That point onwards alerts are generated with CPU interrupts rather high cpu utilization.

Can you helps us ?

[admin_harting]

Hi

I have exactly the same problem. We need to find a solution for that as soon aspossible.

Thanks

Alex

[Siiliev]

Hi to all,

We are facing the same issue , we have also tried to assign different cores to the different network adapters /using affinity tool/ it worked for some period and then the server started crashing. This is an ongoing issue which is causing a lot of trouble . I was thinking if some function of the driver is causing this (virtualized pro 1000 adapters). Tonight i will try to disable the tcp ip offloading and some other functions of the driver.

Any ideas will be appreciated.

BR

Simeon

[vExperts]

Hi,

Are you using the storage type "Netapp" ?

Regards,

Rajesh L

[Siiliev]

Hi,

Well we are using another type of storage appliance, however i cant see the connection between the storage system and the loosy networking.

Also there are a lot of more servers connected to this storage. In this case i am talking not for file server /actually the disk IO from this server is really low/

It seems that the issue is outstanding the network card drivers / features. I have even changed them to another type and the behaivour is still the same.

We are thinking for preinstallation of the os and the isa server FW. My other option is to make kernel debugging of the ndis.sys and see which function exactly is taking so much time and why .

br

Simeon

[vExperts]

You don’t get any clue if you do kernel debugging even by looking at the functions inside. You can try tracing kernel events by using ‘tracelog.exe’ and generate out put via Microsoft performance tool or Windbg.exe even.

Yes, logically there would no connectivity between storage & network but it is there in this case explicitly if you use netapp storage. It is the case happens only with Vmware.

Another reason, if you cluster is loaded with heavy resources not able to accommodate you might encounter this situation.

This is not so easy to trace out.

[mtrekker]

I have the same situation. Any resolution to the problem ?

VMware: ESX 4.1.0

Guest OS: Windows 2003 Enterprise Server SP2

Network card: "VMware Accelerated AMD PCNet Adapter"

Storage type: SAN

NDIS.sys: 5.2.3790.3959

CPU/MMU Virtualization: Use hardware support for both CPU and MMU virtualization (VT/AMD-V and EPT/RVI)

Here is the RATTV3 output:

Cumulative Histogram for:      NDIS.sys  (NDIS.sys)
  #,                  Range, Label,       ISR Count,  DPC Count,  DPCTmr Count
  0,    0.00us to    0.99us, <1.00us,          2418,       1476,            40
  1,    1.00us to    1.99us, <2.00us,         39210,         12,          1055
  2,    2.00us to    2.99us, <3.00us,        156583,      11175,           778
  3,    3.00us to    3.99us, <4.00us,        102648,      56906,           530
  4,    4.00us to    4.99us, <5.00us,         52271,      40023,           391
  5,    5.00us to    5.99us, <6.00us,         30869,      27005,           433
  6,    6.00us to    6.99us, <7.00us,         22550,      22164,           401
  7,    7.00us to    7.99us, <8.00us,         16144,      26719,           241
  8,    8.00us to    8.99us, <9.00us,         12154,      14274,           143
  9,    9.00us to    9.99us, <10.00us,         8289,       9202,            85
10,   10.00us to   19.99us, <20.00us,        18044,      36596,           170
11,   20.00us to   29.99us, <30.00us,         1276,      13582,            33
12,   30.00us to   39.99us, <40.00us,          398,      12686,            18
13,   40.00us to   49.99us, <50.00us,          380,      21810,             4
14,   50.00us to   59.99us, <60.00us,          494,      21538,             2
15,   60.00us to   69.99us, <70.00us,         3331,      18957,            42
16,   70.00us to   79.99us, <80.00us,         4133,      18888,            51
17,   80.00us to   89.99us, <90.00us,         1024,      18724,             6
18,   90.00us to   99.99us, <100.00us,         342,      13935,             5
19,  100.00us to  109.99us, <110.00us,         166,      10342,             1
20,  110.00us to  119.99us, <120.00us,         148,       7835,             5
21,  120.00us to  129.99us, <130.00us,         140,       6033,             3
22,  130.00us to  139.99us, <140.00us,         108,       5208,             1
23,  140.00us to  149.99us, <150.00us,         153,       4587,             1
24,  150.00us to  159.99us, <160.00us,          93,       4089,             2
25,  160.00us to  169.99us, <170.00us,          80,       3504,             0
26,  170.00us to  179.99us, <180.00us,          73,       2993,             2
27,  180.00us to  189.99us, <190.00us,          49,       2649,             1
28,  190.00us to  199.99us, <200.00us,          40,       2332,             0
29,  200.00us to  209.99us, <210.00us,          37,       2102,             2
30,  210.00us to  219.99us, <220.00us,          45,       1988,             1
31,  220.00us to  229.99us, <230.00us,          30,       1816,             0
32,  230.00us to  239.99us, <240.00us,          30,       1606,             1
33,  240.00us to  249.99us, <250.00us,          28,       1515,             1
34,  250.00us to  259.99us, <260.00us,          27,       1309,             0
35,  260.00us to  269.99us, <270.00us,          22,       1294,             0
36,  270.00us to  279.99us, <280.00us,          23,       1146,             0
37,  280.00us to  289.99us, <290.00us,          12,       1108,             1
38,  290.00us to  299.99us, <300.00us,          13,       1020,             1
39,  300.00us to  309.99us, <310.00us,          17,        861,             0
40,  310.00us to  319.99us, <320.00us,          11,        875,             0
41,  320.00us to  329.99us, <330.00us,          13,        814,             1
42,  330.00us to  339.99us, <340.00us,          11,        754,             0
43,  340.00us to  349.99us, <350.00us,          10,        750,             0
44,  350.00us to  359.99us, <360.00us,           4,        669,             0
45,  360.00us to  369.99us, <370.00us,           3,        633,             0
46,  370.00us to  379.99us, <380.00us,           5,        623,             2
47,  380.00us to  389.99us, <390.00us,           2,        545,             1
48,  390.00us to  399.99us, <400.00us,           9,        500,             0
49,  400.00us to  409.99us, <410.00us,           5,        486,             0
50,  410.00us to  419.99us, <420.00us,          11,        465,             0
51,  420.00us to  429.99us, <430.00us,           4,        427,             0
52,  430.00us to  439.99us, <440.00us,          10,        419,             0
53,  440.00us to  449.99us, <450.00us,           5,        433,             0
54,  450.00us to  459.99us, <460.00us,           2,        360,             0
55,  460.00us to  469.99us, <470.00us,           3,        320,             0
56,  470.00us to  479.99us, <480.00us,           4,        321,             0
57,  480.00us to  489.99us, <490.00us,           4,        346,             0
58,  490.00us to  499.99us, <500.00us,           3,        272,             1
59,  500.00us to  509.99us, <510.00us,           2,        272,             0
60,  510.00us to  519.99us, <520.00us,           6,        288,             0
61,  520.00us to  529.99us, <530.00us,           4,        265,             0
62,  530.00us to  539.99us, <540.00us,           3,        267,             0
63,  540.00us to  549.99us, <550.00us,           3,        235,             0
64,  550.00us to  559.99us, <560.00us,           3,        237,             0
65,  560.00us to  569.99us, <570.00us,           4,        200,             0
66,  570.00us to  579.99us, <580.00us,           5,        212,             0
67,  580.00us to  589.99us, <590.00us,           2,        168,             0
68,  590.00us to  599.99us, <600.00us,           1,        187,             0
69,  600.00us to  609.99us, <610.00us,           1,        192,             0
70,  610.00us to  619.99us, <620.00us,           3,        183,             0
71,  620.00us to  629.99us, <630.00us,           1,        136,             0
72,  630.00us to  639.99us, <640.00us,           1,        160,             0
73,  640.00us to  649.99us, <650.00us,           0,        154,             0
74,  650.00us to  659.99us, <660.00us,           1,        108,             1
75,  660.00us to  669.99us, <670.00us,           4,        149,             0
76,  670.00us to  679.99us, <680.00us,           0,        145,             0
77,  680.00us to  689.99us, <690.00us,           3,        118,             0
78,  690.00us to  699.99us, <700.00us,           1,        110,             0
79,  700.00us to  709.99us, <710.00us,           1,        102,             1
80,  710.00us to  719.99us, <720.00us,           2,        106,             0
81,  720.00us to  729.99us, <730.00us,           1,         95,             0
82,  730.00us to  739.99us, <740.00us,           0,         92,             0
83,  740.00us to  749.99us, <750.00us,           0,        100,             0
84,  750.00us to  759.99us, <760.00us,           0,         95,             0
85,  760.00us to  769.99us, <770.00us,           1,         97,             0
86,  770.00us to  779.99us, <780.00us,           1,         71,             0
87,  780.00us to  789.99us, <790.00us,           1,         69,             0
88,  790.00us to  799.99us, <800.00us,           1,         90,             0
89,  800.00us to  809.99us, <810.00us,           0,         66,             0
90,  810.00us to  819.99us, <820.00us,           1,         64,             0
91,  820.00us to  829.99us, <830.00us,           3,         59,             0
92,  830.00us to  839.99us, <840.00us,           2,         69,             0
93,  840.00us to  849.99us, <850.00us,           1,         56,             0
94,  850.00us to  859.99us, <860.00us,           1,         57,             0
95,  860.00us to  869.99us, <870.00us,           2,         58,             0
96,  870.00us to  879.99us, <880.00us,           2,         58,             0
97,  880.00us to  889.99us, <890.00us,           2,         47,             0
98,  890.00us to  899.99us, <900.00us,           0,         53,             0
99,  900.00us to  909.99us, <910.00us,           0,         37,             0
100,  910.00us to  919.99us, <920.00us,           1,         44,             0
101,  920.00us to  929.99us, <930.00us,           0,         37,             0
102,  930.00us to  939.99us, <940.00us,           0,         37,             0
103,  940.00us to  949.99us, <950.00us,           0,         33,             0
104,  950.00us to  959.99us, <960.00us,           1,         40,             0
105,  960.00us to  969.99us, <970.00us,           2,         34,             0
106,  970.00us to  979.99us, <980.00us,           0,         36,             0
107,  980.00us to  989.99us, <990.00us,           1,         25,             0
108,  990.00us to  999.99us, <1000.00us,          1,         29,             0
109, 1000.00us to 1009.99us, <1010.00us,          1,         33,             0
110,    1.01ms to    1.02ms, <1.02ms,             0,         29,             0
111,    1.02ms to    1.03ms, <1.03ms,             2,         35,             0
112,    1.03ms to    1.04ms, <1.04ms,             2,         16,             0
113,    1.04ms to    1.05ms, <1.05ms,             0,         24,             0
114,    1.05ms to    1.06ms, <1.06ms,             1,         25,             0
115,    1.06ms to    1.07ms, <1.07ms,             0,         17,             0
116,    1.07ms to    1.08ms, <1.08ms,             0,         23,             0
117,    1.08ms to    1.09ms, <1.09ms,             2,         12,             0
118,    1.09ms to    1.10ms, <1.10ms,             2,         26,             0
119,    1.10ms to    1.11ms, <1.11ms,             0,         23,             0
120,    1.11ms to    1.12ms, <1.12ms,             1,         22,             0
121,    1.12ms to    1.13ms, <1.13ms,             0,         21,             0
122,    1.13ms to    1.14ms, <1.14ms,             3,         20,             0
123,    1.14ms to    1.15ms, <1.15ms,             1,         16,             0
124,    1.15ms to    1.16ms, <1.16ms,             0,         16,             0
125,    1.16ms to    1.17ms, <1.17ms,             0,         19,             0
126,    1.17ms to    1.18ms, <1.18ms,             1,         16,             0
127,    1.18ms to    1.19ms, <1.19ms,             0,         20,             0
128,    1.19ms to    1.20ms, <1.20ms,             1,         18,             0
129,    1.20ms to    1.21ms, <1.21ms,             0,         13,             0
130,    1.21ms to    1.22ms, <1.22ms,             0,         13,             0
131,    1.22ms to    1.23ms, <1.23ms,             0,         12,             0
132,    1.23ms to    1.24ms, <1.24ms,             0,         13,             0
133,    1.24ms to    1.25ms, <1.25ms,             2,         13,             0
134,    1.25ms to    1.26ms, <1.26ms,             0,         20,             0
135,    1.26ms to    1.27ms, <1.27ms,             0,         19,             0
136,    1.27ms to    1.28ms, <1.28ms,             0,         11,             0
137,    1.28ms to    1.29ms, <1.29ms,             0,          8,             0
138,    1.29ms to    1.30ms, <1.30ms,             0,         13,             0
139,    1.30ms to    1.31ms, <1.31ms,             1,         11,             0
140,    1.31ms to    1.32ms, <1.32ms,             0,          5,             0
141,    1.32ms to    1.33ms, <1.33ms,             0,         14,             0
142,    1.33ms to    1.34ms, <1.34ms,             1,          8,             0
143,    1.34ms to    1.35ms, <1.35ms,             1,         10,             0
144,    1.35ms to    1.36ms, <1.36ms,             0,         13,             0
145,    1.36ms to    1.37ms, <1.37ms,             0,         10,             0
146,    1.37ms to    1.38ms, <1.38ms,             0,         13,             0
147,    1.38ms to    1.39ms, <1.39ms,             0,          7,             0
148,    1.39ms to    1.40ms, <1.40ms,             0,         10,             0
149,    1.40ms to    1.41ms, <1.41ms,             0,          8,             0
150,    1.41ms to    1.42ms, <1.42ms,             1,          6,             0
151,    1.42ms to    1.43ms, <1.43ms,             0,         11,             0
152,    1.43ms to    1.44ms, <1.44ms,             0,         14,             0
153,    1.44ms to    1.45ms, <1.45ms,             0,          9,             0
154,    1.45ms to    1.46ms, <1.46ms,             0,          9,             0
155,    1.46ms to    1.47ms, <1.47ms,             0,          4,             0
156,    1.47ms to    1.48ms, <1.48ms,             0,          8,             0
157,    1.48ms to    1.49ms, <1.49ms,             0,          9,             1
158,    1.49ms to    1.50ms, <1.50ms,             0,          8,             0
159,    1.50ms to    1.51ms, <1.51ms,             0,          5,             0
160,    1.51ms to    1.52ms, <1.52ms,             0,          7,             0
161,    1.52ms to    1.53ms, <1.53ms,             0,          4,             0
162,    1.53ms to    1.54ms, <1.54ms,             0,          7,             0
163,    1.54ms to    1.55ms, <1.55ms,             0,          2,             0
164,    1.55ms to    1.56ms, <1.56ms,             1,          5,             0
165,    1.56ms to    1.57ms, <1.57ms,             1,          6,             0
166,    1.57ms to    1.58ms, <1.58ms,             2,          6,             0
167,    1.58ms to    1.59ms, <1.59ms,             0,          9,             0
168,    1.59ms to    1.60ms, <1.60ms,             0,          3,             0
169,    1.60ms to    1.61ms, <1.61ms,             0,          4,             0
170,    1.61ms to    1.62ms, <1.62ms,             1,         11,             0
171,    1.62ms to    1.63ms, <1.63ms,             0,          7,             0
172,    1.63ms to    1.64ms, <1.64ms,             1,          1,             0
173,    1.64ms to    1.65ms, <1.65ms,             0,          6,             0
174,    1.65ms to    1.66ms, <1.66ms,             0,          1,             0
175,    1.66ms to    1.67ms, <1.67ms,             0,          3,             0
176,    1.67ms to    1.68ms, <1.68ms,             1,          7,             0
177,    1.68ms to    1.69ms, <1.69ms,             0,          5,             0
178,    1.69ms to    1.70ms, <1.70ms,             0,          6,             0
179,    1.70ms to    1.71ms, <1.71ms,             0,          5,             0
180,    1.71ms to    1.72ms, <1.72ms,             2,          7,             0
181,    1.72ms to    1.73ms, <1.73ms,             0,          7,             0
182,    1.73ms to    1.74ms, <1.74ms,             0,          4,             0
183,    1.74ms to    1.75ms, <1.75ms,             0,          7,             0
185,    1.76ms to    1.77ms, <1.77ms,             0,          3,             0
186,    1.77ms to    1.78ms, <1.78ms,             0,          3,             0
187,    1.78ms to    1.79ms, <1.79ms,             0,          5,             0
188,    1.79ms to    1.80ms, <1.80ms,             0,          1,             0
189,    1.80ms to    1.81ms, <1.81ms,             0,          6,             0
190,    1.81ms to    1.82ms, <1.82ms,             3,          3,             0
191,    1.82ms to    1.83ms, <1.83ms,             1,          0,             0
192,    1.83ms to    1.84ms, <1.84ms,             0,          2,             0
193,    1.84ms to    1.85ms, <1.85ms,             0,          5,             0
194,    1.85ms to    1.86ms, <1.86ms,             0,          5,             0
195,    1.86ms to    1.87ms, <1.87ms,             0,          4,             0
197,    1.88ms to    1.89ms, <1.89ms,             0,          2,             0
198,    1.89ms to    1.90ms, <1.90ms,             1,          2,             0
199,    1.90ms to    1.91ms, <1.91ms,             0,          4,             0
200,    1.91ms to    1.92ms, <1.92ms,             2,          2,             0
201,    1.92ms to    1.93ms, <1.93ms,             0,          5,             0
202,    1.93ms to    1.94ms, <1.94ms,             0,          5,             0
203,    1.94ms to    1.95ms, <1.95ms,             0,          2,             0
205,    1.96ms to    1.97ms, <1.97ms,             1,          2,             0
206,    1.97ms to    1.98ms, <1.98ms,             0,          2,             0
207,    1.98ms to    1.99ms, <1.99ms,             0,          5,             0
208,    1.99ms to    2.00ms, <2.00ms,             0,          2,             0
209,    2.00ms to    2.01ms, <2.01ms,             2,          5,             0
210,    2.01ms to    2.02ms, <2.02ms,             0,          1,             0
211,    2.02ms to    2.03ms, <2.03ms,             1,          1,             0
212,    2.03ms to    2.04ms, <2.04ms,             0,          2,             0
213,    2.04ms to    2.05ms, <2.05ms,             1,          1,             0
214,    2.05ms to    2.06ms, <2.06ms,             0,          2,             0
216,    2.07ms to    2.08ms, <2.08ms,             0,          3,             0
217,    2.08ms to    2.09ms, <2.09ms,             0,          4,             0
218,    2.09ms to    2.10ms, <2.10ms,             0,          2,             0
219,    2.10ms to    2.11ms, <2.11ms,             0,          1,             0
220,    2.11ms to    2.12ms, <2.12ms,             0,          1,             0
221,    2.12ms to    2.13ms, <2.13ms,             0,          2,             0
223,    2.14ms to    2.15ms, <2.15ms,             0,          2,             0
224,    2.15ms to    2.16ms, <2.16ms,             0,          2,             0
225,    2.16ms to    2.17ms, <2.17ms,             0,          2,             0
226,    2.17ms to    2.18ms, <2.18ms,             0,          1,             0
227,    2.18ms to    2.19ms, <2.19ms,             0,          3,             0
228,    2.19ms to    2.20ms, <2.20ms,             0,          4,             0
229,    2.20ms to    2.21ms, <2.21ms,             0,          2,             0
230,    2.21ms to    2.22ms, <2.22ms,             0,          1,             0
231,    2.22ms to    2.23ms, <2.23ms,             0,          3,             0
232,    2.23ms to    2.24ms, <2.24ms,             0,          2,             0
233,    2.24ms to    2.25ms, <2.25ms,             0,          2,             0
234,    2.25ms to    2.26ms, <2.26ms,             0,          2,             0
235,    2.26ms to    2.27ms, <2.27ms,             0,          2,             0
236,    2.27ms to    2.28ms, <2.28ms,             0,          2,             0
237,    2.28ms to    2.29ms, <2.29ms,             0,          1,             0
238,    2.29ms to    2.30ms, <2.30ms,             0,          3,             0
239,    2.30ms to    2.31ms, <2.31ms,             0,          2,             0
240,    2.31ms to    2.32ms, <2.32ms,             0,          1,             0
241,    2.32ms to    2.33ms, <2.33ms,             0,          1,             0
244,    2.35ms to    2.36ms, <2.36ms,             2,          1,             0
245,    2.36ms to    2.37ms, <2.37ms,             0,          1,             0
246,    2.37ms to    2.38ms, <2.38ms,             0,          1,             0
247,    2.38ms to    2.39ms, <2.39ms,             0,          3,             0
248,    2.39ms to    2.40ms, <2.40ms,             1,          0,             0
250,    2.41ms to    2.42ms, <2.42ms,             0,          1,             0
251,    2.42ms to    2.43ms, <2.43ms,             0,          4,             0
252,    2.43ms to    2.44ms, <2.44ms,             0,          1,             0
254,    2.45ms to    2.46ms, <2.46ms,             0,          1,             0
257,    2.48ms to    2.49ms, <2.49ms,             0,          1,             0
258,    2.49ms to    2.50ms, <2.50ms,             0,          2,             0
259,    2.50ms to    2.51ms, <2.51ms,             0,          3,             0
260,    2.51ms to    2.52ms, <2.52ms,             0,          4,             0
263,    2.54ms to    2.55ms, <2.55ms,             0,          1,             0
264,    2.55ms to    2.56ms, <2.56ms,             0,          1,             0
265,    2.56ms to    2.57ms, <2.57ms,             0,          1,             0
266,    2.57ms to    2.58ms, <2.58ms,             0,          1,             0
267,    2.58ms to    2.59ms, <2.59ms,             0,          1,             0
268,    2.59ms to    2.60ms, <2.60ms,             0,          3,             0
269,    2.60ms to    2.61ms, <2.61ms,             0,          1,             0
271,    2.62ms to    2.63ms, <2.63ms,             0,          1,             0
273,    2.64ms to    2.65ms, <2.65ms,             0,          3,             0
274,    2.65ms to    2.66ms, <2.66ms,             0,          1,             0
275,    2.66ms to    2.67ms, <2.67ms,             0,          2,             0
276,    2.67ms to    2.68ms, <2.68ms,             0,          1,             0
277,    2.68ms to    2.69ms, <2.69ms,             1,          1,             0
281,    2.72ms to    2.73ms, <2.73ms,             0,          3,             0
284,    2.75ms to    2.76ms, <2.76ms,             0,          2,             0
285,    2.76ms to    2.77ms, <2.77ms,             0,          1,             0
287,    2.78ms to    2.79ms, <2.79ms,             0,          1,             0
290,    2.81ms to    2.82ms, <2.82ms,             0,          2,             0
294,    2.85ms to    2.86ms, <2.86ms,             0,          1,             0
295,    2.86ms to    2.87ms, <2.87ms,             1,          2,             0
297,    2.88ms to    2.89ms, <2.89ms,             1,          0,             0
301,    2.92ms to    2.93ms, <2.93ms,             0,          1,             0
306,    2.97ms to    2.98ms, <2.98ms,             0,          3,             0
307,    2.98ms to    2.99ms, <2.99ms,             0,          1,             0
309,    3.00ms to    3.01ms, <3.01ms,             0,          1,             0
310,    3.01ms to    3.02ms, <3.02ms,             0,          3,             0
313,    3.04ms to    3.05ms, <3.05ms,             0,          1,             0
314,    3.05ms to    3.06ms, <3.06ms,             0,          3,             0
315,    3.06ms to    3.07ms, <3.07ms,             0,          4,             0
319,    3.10ms to    3.11ms, <3.11ms,             0,          1,             0
321,    3.12ms to    3.13ms, <3.13ms,             0,          1,             0
322,    3.13ms to    3.14ms, <3.14ms,             0,          1,             0
323,    3.14ms to    3.15ms, <3.15ms,             0,          1,             0
324,    3.15ms to    3.16ms, <3.16ms,             0,          1,             0
325,    3.16ms to    3.17ms, <3.17ms,             0,          1,             0
326,    3.17ms to    3.18ms, <3.18ms,             0,          1,             0
327,    3.18ms to    3.19ms, <3.19ms,             0,          1,             0
329,    3.20ms to    3.21ms, <3.21ms,             0,          1,             0
334,    3.25ms to    3.26ms, <3.26ms,             0,          1,             0
335,    3.26ms to    3.27ms, <3.27ms,             0,          1,             0
336,    3.27ms to    3.28ms, <3.28ms,             0,          1,             0
337,    3.28ms to    3.29ms, <3.29ms,             0,          2,             0
340,    3.31ms to    3.32ms, <3.32ms,             0,          1,             0
343,    3.34ms to    3.35ms, <3.35ms,             0,          1,             0
347,    3.38ms to    3.39ms, <3.39ms,             0,          1,             0
348,    3.39ms to    3.40ms, <3.40ms,             0,          1,             0
349,    3.40ms to    3.41ms, <3.41ms,             0,          1,             0
351,    3.42ms to    3.43ms, <3.43ms,             0,          1,             0
352,    3.43ms to    3.44ms, <3.44ms,             0,          1,             0
355,    3.46ms to    3.47ms, <3.47ms,             0,          1,             0
357,    3.48ms to    3.49ms, <3.49ms,             0,          1,             0
358,    3.49ms to    3.50ms, <3.50ms,             0,          1,             0
359,    3.50ms to    3.51ms, <3.51ms,             0,          1,             0
360,    3.51ms to    3.52ms, <3.52ms,             0,          1,             0
363,    3.54ms to    3.55ms, <3.55ms,             0,          1,             0
366,    3.57ms to    3.58ms, <3.58ms,             0,          2,             0
368,    3.59ms to    3.60ms, <3.60ms,             0,          1,             0
370,    3.61ms to    3.62ms, <3.62ms,             0,          1,             0
373,    3.64ms to    3.65ms, <3.65ms,             0,          1,             0
377,    3.68ms to    3.69ms, <3.69ms,             0,          1,             0
378,    3.69ms to    3.70ms, <3.70ms,             0,          2,             0
389,    3.80ms to    3.81ms, <3.81ms,             0,          1,             0
392,    3.83ms to    3.84ms, <3.84ms,             0,          1,             0
393,    3.84ms to    3.85ms, <3.85ms,             0,          1,             0
394,    3.85ms to    3.86ms, <3.86ms,             0,          1,             0
395,    3.86ms to    3.87ms, <3.87ms,             0,          1,             0
401,    3.92ms to    3.93ms, <3.93ms,             0,          1,             0
408,    3.99ms to    4.00ms, <4.00ms,             1,          0,             0
409,    4.00ms to    4.01ms, <4.01ms,             0,          1,             0
414,    4.05ms to    4.06ms, <4.06ms,             0,          2,             0
416,    4.07ms to    4.08ms, <4.08ms,             0,          1,             0
417,    4.08ms to    4.09ms, <4.09ms,             0,          1,             0
420,    4.11ms to    4.12ms, <4.12ms,             0,          2,             0
421,    4.12ms to    4.13ms, <4.13ms,             0,          1,             0
432,    4.23ms to    4.24ms, <4.24ms,             0,          1,             0
435,    4.26ms to    4.27ms, <4.27ms,             0,          1,             0
439,    4.30ms to    4.31ms, <4.31ms,             0,          1,             0
441,    4.32ms to    4.33ms, <4.33ms,             0,          1,             0
451,    4.42ms to    4.43ms, <4.43ms,             1,          1,             0
457,    4.48ms to    4.49ms, <4.49ms,             0,          1,             0
458,    4.49ms to    4.50ms, <4.50ms,             0,          2,             0
462,    4.53ms to    4.54ms, <4.54ms,             0,          1,             0
467,    4.58ms to    4.59ms, <4.59ms,             0,          1,             0
473,    4.64ms to    4.65ms, <4.65ms,             0,          1,             0
484,    4.75ms to    4.76ms, <4.76ms,             0,          1,             0
498,    4.89ms to    4.90ms, <4.90ms,             0,          1,             0
506,    4.97ms to    4.98ms, <4.98ms,             0,          1,             0
510,    5.01ms to    5.02ms, <5.02ms,             0,          1,             0
512,    5.03ms to    5.04ms, <5.04ms,             0,          1,             0
513,    5.04ms to    5.05ms, <5.05ms,             0,          1,             0
524,    5.15ms to    5.16ms, <5.16ms,             0,          1,             0
525,    5.16ms to    5.17ms, <5.17ms,             0,          1,             0
536,    5.27ms to    5.28ms, <5.28ms,             0,          1,             0
538,    5.29ms to    5.30ms, <5.30ms,             0,          1,             0
544,    5.35ms to    5.36ms, <5.36ms,             0,          1,             0
547,    5.38ms to    5.39ms, <5.39ms,             0,          1,             0
551,    5.42ms to    5.43ms, <5.43ms,             0,          1,             0
560,    5.51ms to    5.52ms, <5.52ms,             0,          1,             0
581,    5.72ms to    5.73ms, <5.73ms,             0,          1,             0
586,    5.77ms to    5.78ms, <5.78ms,             0,          1,             0
597,    5.88ms to    5.89ms, <5.89ms,             0,          1,             0
623,    6.14ms to    6.15ms, <6.15ms,             0,          1,             0
663,    6.54ms to    6.55ms, <6.55ms,             0,          1,             0
671,    6.62ms to    6.63ms, <6.63ms,             0,          1,             0
690,    6.81ms to    6.82ms, <6.82ms,             0,          1,             0
713,    7.04ms to    7.05ms, <7.05ms,             1,          0,             0
733,    7.24ms to    7.25ms, <7.25ms,             0,          1,             0
770,    7.61ms to    7.62ms, <7.62ms,             0,          1,             0
873,    8.64ms to    8.65ms, <8.65ms,             0,          1,             0
894,    8.85ms to    8.86ms, <8.86ms,             0,          1,             0
942,    9.33ms to    9.34ms, <9.34ms,             0,          1,             0
1009,   10.00ms to   19.99ms, <20.00ms,            2,          3,             0
                              TOTALS,        474142,     467843,          4459