Is there a thread limitation on the networkDataPort for EPAgent (9.5.5)?

[Mike Gucciard]

I've encountered what appears to be a hard limit of 2 + number of configured plugins, but there isn't anything noted in documentation to confirm this. 

I did find a reference to this in a thread from 2011, but was rather hoping this information was out of date.

(EPAgent - sending metrics across sockets )

I need to determine if I should abandon using XML to networkDataPort or if this thread limit can be increased?  

I have roughly 20 nodes I need to feed a bare handful of metrics from, and networkDataPort seemed like an ideal low impact solution.

Then I ran into this seeming limitation. 

Can anyone confirm / deny what I'm seeing here?

 

Symptoms are:

Socket opens cleanly from client, data is sent, XML format is valid.

EPAgent accepts socket open, appears to accept data.

Investigator will show data appearing only for 2 + count of plugins configured (7, in my case).

The rest appears to simply be dropped.

 

I'd expect there to be some way to expand the thread count allocated to handle this kind of feed, but cannot find anything documented.

 

Suggestions?

[rogelio.dipasquale]

Hi,

 

I never heard about such limit. How are your 7 apps distributed?

I mean, all of them are hosted on different IP addresses? I can quickly make a lab to try to reproduce your situation.

 

What EM version are you using? Also 9.5.5?

 

 

Regards,

Roger

[Mike Gucciard]

EPAgent version is 9.5.5

Single application being hosted.

14 JVMs, running Java agent  - no problems here.

Single IP address.

5 plugins configured (3 shell, 2 perl scripts).   Two are "stateful" and run continuously, the rest run iteratively at intervals.

 

Poking at this, I discovered is that if I open more than (2+<plugin count>) socket sessions, the connections will connect cleanly to the port, but no data will ever show up in investigator for anything beyond that count. 

 

Further investigation determined that if I did an <open>  Pass data  <close> sequence, instead of <open> pass data (every 60 sec) <no close>

that the limit functionally disappears.  But that still suggests that there is a limit on how many active open sockets can be handled at once.

 

So I've worked around the problem, but still would like to confirm the limitation exists, as I may not be able to cycle ports like this in all cases.

I currently have 14 XML feeds clicking away - so Yay for experimentation?

 

-Mike

[rogelio.dipasquale]

Hi Mike,

 

I see the situation. So it seems that 9.5.5 does not let you have 2+ tcp connections open simultaneously. I'll check 3 tcp connections on 9.7 and 10.1, and will let you know.

 

Regards,

Roger

[Guenter DI Grossberger]

I just checked the code and as you guessed correctly the thread pool size = #plugins + networkServerPort (if configured) + httpServerPort (if configured).

 

You could try using the EPAgent REST Interface instead. It is based on an internal jetty server that should be using a different thread pool with up to 20 threads.

 

Ciao,

Guenter

[Mike Gucciard]

I take it there is no way to increase the thread pool serving networkDataPort and httpServerPort directly?

I mean I can configure a bunch of "pseudo-plugins" to fake up to the number of sessions needed, but that seems awkward.

REST interface will require 9.7, which while on our list, isn't going to be "standard" in company for a while yet.

 

Does the same limitation apply to 9.7  AND to 10.x?

[rogelio.dipasquale]

Hi Mike,

 

I guess if there is no way you migrate, you should run 2 or 3 EPAgents listening on different ports so you can support all your requests.

I'll try to get the lab done to check if 9.7 or 10.1 support more plugins simultaneously

 

 

Regards,

Roger

[rogelio.dipasquale]

Hi Mike!

 

I could run the lab:

 

I configured

• 4  stateless plugins
• 1 http rest connector
• 10 tcp ports open, sending metrics

 

As you can see in the image, Agent 10.1.0.15 recognized only 7 of the tcp ports sending metrics, so I guess it is sill limited to a total of 12 plugins.

 

 

Regards,

Roger

[Mike Gucciard]

Good to know, forewarned is forearmed, and all that.

I'll just have to do everything possible to eliminate the cases where having an open socket is more useful than cycling open / data / closed.

 

Thank you for taking the time to lab confirm my analysis.

Much appreciated.

-Mike Gucciard

-AT&T  Entertainment Group

[rogelio.dipasquale]

No problem at all.

 

I gues you can also try deploying a set of EPAgents listening on different ports to deal with this; sometimes opening/closing ports might create overhead on the app, when metric volume is high.

I'm coping with an application that sends hundreds of metrics per interval an I'm considering going from REST api to opening a single tcp connection because of metrics lost.

 

 

Regards,

Roger

[ocaom01]

Yes, unfortunately it also applies for 10.1 and 10.2

I ran into this problem a few months ago, and made myself a little patch to increase the number of thread pool.

I can now set the thread pool size in the EPAgent profile:

> introscope.epagent.config.networkserver.poolsize=6

If you are interested I can send you my patch, but please keep in mind it is a field-solution

 

Regards,

Omar