You ever have one of those days where you think that you're CA's least favorite customer?

[Garin Walsh]

I think that's happening to me today.

 

Sigh.

 

-Garin

[Anon Anon]

Sounds like you could use a hug today. Cheer up buddy. You're not the only one who has had one of those days.

[Anon Anon]

Would that feeling stem from the amount of support cases you have to open and/or the lack of attention those cases get?

[Garin Walsh]

I get plenty of attention, there's no issue there. 

 

On the other hand, regarding open cases, I was faced with the question from support, "do you want us to close the case or mark it as deferred to a future release" and I said close it because the number of open and deferred cases is so large currently, it makes finding one individual case difficult.

 

That and I was told that my installation is too big to run the new parts (and some of the old) in UIM 8.1 and that I'd need to wait for a later release that addresses the scalability issues in the current release version. 

 

CA really needs to stop trying to put new half baked features into this product. Instead, get what's there working right. I could go on for days about the new SNMP tool's uselessness but I can't configure it to prove myself wrong because the tool to do so relies on infrastructure pieces that don't work in an install the size of mine.

 

Maybe 8.2 will correct the situation, or 8.5. Certainly those will break something else substantial.

 

-Garin 

[Anon Anon]

CA really needs to stop trying to put new half baked features into this product - I work with RayF so I suspect you have seen our frustrations on that front.

 

With regards "collector" - yeah for the last year my days go something like this

 

Open support case on either (snmptoolkit, Interface_traffic, etc ) -> get reply from support saying "have you tried collector yet -> explain that collector does not or did not at the time do what we needed it to do  -> get another reply from support that says "can you try the new version.." of whatever probe I had opened my support case against -> download new version of that probe -> frustrate my ends users -> open new support case regarding the new bugs in that version. Repeat. 

[Legacy User]

> CA really needs to stop trying to put new half baked features into this product

Amen!

[Scott Stuessi]

Garin,

Any specifics on the scalability issues?

thank you,

Scott

[Garin Walsh]

Right now the scalability issues that are causing the most pain are related to service_host being unable to generate the list of hubs in my environment. Without the list of hubs, all you can do is look at the contents of the archive via the admin console but it's impossible to configure or deploy anything.

 

Support has looked at the configuration and is working with engineering to identify next steps.

 

My hopes are not especially high as it was suggested that maybe this will get fixed in a subsequent release.

 

Scott, I couldn't tell from your profile if you're a customer or not but if not, the existing case is 00153903 which has a bunch more detail.

 

-Garin

[Scott Stuessi]

Garin.

We are a customer, ill see if i can look throught the case history.

thanks for the response.

Scott

[Anon Anon]

I have noticed that service_host takes forever to start responding.   On initial startup it trolls through all available hubs and robots trying to find them all and cache or index them maybe?  MPSE and discovery may be involved as well as ppm.  Likewise, the whole thing seems a little less likely to work until discovery has made a good pass over the environment to populate a bunch of stuff.  If you don't use it for a while, then go to the interface, AC will re-crawl the hubs and itemize robots before responding.  The interface will proably timeout before the crawl completes, but it will eventually finish and I can use AC.

 

This may imply that some timers might be better lowered than raised since you don't want to block forever on a down hub or robot in a sequential crawl.  Also, more memory might be better due to the datomic database that caches records before executing local queries.

 

I also had a weird brief time when it came up with two of each hub, one for each path down redundant tunnels. Then it corrected itself.  I'm guessing it may have been a point when the cache was loaded but not consolidated?  Logs also complain about duplicate robotids if you have any at high debug levels.

 

Currently have 338 hubs.

 

There are some additional config files and options that are hidden or not well explained.

 

If you get adminconsole running, it lets you configure adminconsole and monitoring services.  The latter ends up in two places.  I believe mons.cfg propagates out into log4j.xml automatically so you probably only need to change one.  This is mine with some interesting things turned up to trace level logging.  Note the monitoring service log location as well and the <soLinger> tag.  This is described as "Socket Linger Timeout (seconds)" in the configuration interface.  Might be valuable to reduce startup time.

 

Ids services is not configurable, but has a config file with some interesting options.  It's also noteworthy that there are seven log levels available under service_host.cfg and it is set in two places.  I also disabled compression temporarily.  Don't know if that has an effect.

 

Pretty much all of our hubs are 7.61 or newer with 7.62 at the middle distribution tier as tunnel clients.  7.61 has an odd bug where it can terminate a connection if a large amount of data is being requested and it can traverse the tunnel from the remote hub faster than accross the local network (or vpn) to the requester.  This is documented in detail in the forum.  7.63 doesn't contain the release and 7.62 was never officially released. 

 

Ignore the cat.  We're on windows/sql/UIM-8.1, but I mount some stuff to a unix box so I don't have to windows as actively.  

 

It is possible to get a second admin_console running too.  The trick is the second data engine which support finally said is supported but will be taken away later and shouldn't be used to push data to the database.  I just connect it to a queue that collects a non-existent subject.   You can tell adminconsole to use a data_engine in a different location than the local hub, but it depends on mpse which requires a local data_engine, so it's no help as far as I can tell.  You'll also need mpse and distsrv and you will need to load you licenses on the distsrv or some stuff won't start correctly.  Then of course there are the other 7 or so undocumented probe dependencies to get it working that we've itemized elsewhere on here.

 

############################ cat ./catalinaBase/conf/MonitoringService/mons.cfg
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<configuration>
<webapps>
<monitoring_services>
<settings>
<logFileName>catalinaBase/logs/monitoring_services/MonitoringServiceProbe.log</logFileName>
<logFileSize>5242880</logFileSize>
<appenders>
<com>
<nimsoft>
<mons>trace<model>
<ctd>
<display>
<builders>trace</builders>
</display>
</ctd>
</model>
</mons>
<ids>
<ctd>trace</ctd>
</ids>
</nimsoft>
</com>
<org>
<springframework>warn<security>warn</security>
</springframework>
<apache>
<commons>
<beanutils>
<converters>warn</converters>
</beanutils>
</commons>
</apache>
</org>
<rootLog>trace</rootLog>
</appenders>
<soLinger>5</soLinger>
</settings>
</monitoring_services>
</webapps>
</configuration>

 

######################################./catalinaBase/webapps/monitoring_services/WEB-INF/classes/log4j.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="true" threshold="null">
<!-- our console appender -->
<appender class="org.apache.log4j.ConsoleAppender" name="Console">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{MMddyyyy HH:mm:ss,SSS} [%10.10t] %X{client} %-5p %c - %m%n"/>
</layout>
</appender>
<appender class="org.apache.log4j.RollingFileAppender" name="ProbeLogFile">
<param name="File" value="catalinaBase/logs/monitoring_services/MonitoringServiceProbe.log"/>
<param name="Append" value="true"/>
<param name="MaxFileSize" value="5242880"/>
<param name="MaxBackupIndex" value="5"/>
<param name="Encoding" value="UTF-8"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{MMddyyyy HH:mm:ss,SSS} [%10.10t] %X{client} %-5p %c - %m%n"/>
</layout>
</appender>
<category additivity="false" name="org.springframework">
<priority value="warn"/>
<appender-ref ref="ProbeLogFile"/>
</category>
<category additivity="false" name="com.nimsoft.mons">
<priority value="trace"/>
<appender-ref ref="ProbeLogFile"/>
</category>
<category additivity="false" name="org.springframework.security">
<priority value="warn"/>
<appender-ref ref="ProbeLogFile"/>
</category>
<category additivity="false" name="com.nimsoft.ids.ctd">
<priority value="trace"/>
<appender-ref ref="ProbeLogFile"/>
</category>
<category additivity="false" name="org.apache.commons.beanutils.converters">
<priority value="warn"/>
<appender-ref ref="ProbeLogFile"/>
</category>
<category additivity="false" name="com.nimsoft.mons.model.ctd.display.builders">
<priority value="trace"/>
<appender-ref ref="ProbeLogFile"/>
</category>
<root>
<priority value="trace"/>
<appender-ref ref="Console"/>
<appender-ref ref="ProbeLogFile"/>
</root>
</log4j:configuration>

 

###############################cat ./service_host.cfg
<setup>
loglevel = 7
logsize = 5120
hook_stdout = true
log_stdout = true
data_engine = /Berbee/h-nimroot-msn-3/h-nimroot-msn-3/data_engine
deploy_poll_ms = 6000
deploy_wait_ms = 15000
CATALINA_OPTS = -Xms1024m -Xmx4096m -Djava.library.path="../../../lib"
composite_ctd = false
<ssl>
self_signed_nbits = 2048
use_generated_keystore = true
enabled = false
</ssl>
<server_xml>
<server>
port = 8006
</server>
<http_connector>
port = 8080
connectionTimeout = 20000
maxThreads = 300
compression = off
useSendfile = false
compressableMimeType = text/html,text/xml,text/plain,text/css,application/json,application/javascript,application/xml
</http_connector>
<https_connector>
port = 8443
connectionTimeout = 20000
maxThreads = 200
protocol = org.apache.coyote.http11.Http11NioProtocol
keystoreFile =
keystoreType =
keyAlias =
compression = on
useSendfile = false
compressableMimeType = text/html,text/xml,text/plain,text/css,application/json,application/javascript,application/xml
</https_connector>
<host>
name =
</host>
</server_xml>
<loggers>
org.apache.http = SDK_LOW
</loggers>
</setup>
<startup>
options = -Xms128m -Xmx2048m
</startup>

 

####################################### cat ./catalinaBase/conf/ids_services/ids_services.cfx

<setup>
loglevel = 4
logsize = 5120

max_n_logfiles = 5
log_odata_entities = false

archive_url = http://archive.nimsoft.com/
support_url = http://support.nimsoft.com/

product_catalog_cache_time_ms = 3600000

primary_distsrv = scan
primary_distsrv_scan_seconds = 900

primary_license_manager = scan
primary_license_manager_scan_seconds = 900

deployment_engine = automated_deployment_engine

odata_max_results = 350

<loggers>
org.apache.http = WARN
</loggers>

ScannerPoolNCoreThreads = 7
ScannerPoolMaxPoolSize = 7
ScannerPoolKeepAliveTimeSec = 120

PackageDownloadPoolNThreads = 5

utf8_supported_probes = Ad_server|1.70,Hyperv|2.11,Logmon|3.42,Ntevl|4.00,Adevl|2.00
</setup>

[Garin Walsh]

Thank you for sharing - I'll try to pick through this today to see if there's anything that pops out as a cause.

 

For me, on a good day, I have 650 active hubs. 

 

If I restart service_host it takes between 45 minutes and 4 hours to get to the point that it will present the default landing page that allows you to select the installers. Another 20 minutes past that to get the admin console page to display - though almost always I'm rewarded with the "no hubs to display" result.

 

Right now the biggest change from the default is running the java VM and probe memory settings at 8 GB on as much related to service_host as possible. With the default memory settings, I only saw service_host successfully start once. Before changing it would fail at some point in the startup and then Tomcat wouldn't start its listener.

 

You'd think that someone would have recognized that pretty much any "central" probe maintains a list of hubs and it's in the database at least three separate times. Maybe instead of recreating the wheel a 9th time, they should have just used the get_hubs callback to the hub, or hit up discovery_server, or maybe data_engine? <sarcasm>Though maybe where each of these uses a different method of collecting the info, service_host has a better and more accurate way?</sarcasm>

 

-Garin

 

 

[Garin Walsh]

Ray, I went though your config and compared w/ mine - found some differences and was able by following your example to shorten the  time of service_host a little but I still get an admin console that is void of content.

 

I suspect that there's a functional limit in the code somewhere and I've exceeded that. Probably something that allows only 512 entries and they never figured that someone would exceed that. 

 

At this point support continues to tell me that they're working on it but the conversation has been suspiciously lacking of the typical requests for logs and config files. Makes me think that they know what's wrong but not sure how to share it. I'm supposed to learn more this week

 

-Garin

[Anon Anon]

Probably nothing to do with this - but my AC gives no content with DNS name, except after a long wait, but loads pretty fast with IP. That is AFTER login, login works just fine either way. Havent looked more into it yet as there are quite a few concerns when setting up stuff..

 

-jon