tomcat problems

[wartus]

Hi,

I have multiple servers that use tomcat 5.5

almost all of them work perfectly with hyperic, exept 2 of them.

they do appear in th HQ interface, but they are very instable. availability is around 26% etc

It looks as if they are timing out, but i cant put my finger on it why or what to check.

does anyone have an idea?


thanx in advance



her is a agent log example:

2007-07-24 06:59:38,175 INFO [AgentConnection] 127.0.0.1 -> agent:ping
2007-07-24 06:59:39,449 INFO [AgentCommandsServer] Agent commands loaded
2007-07-24 06:59:39,493 INFO [AgentDaemon] Agent starting up
2007-07-24 06:59:39,541 INFO [ProductPluginManager] Loading plugin: system-plugin.jar
2007-07-24 06:59:39,754 INFO [ProductPluginManager] Loading plugin: netservices-plugin.jar
2007-07-24 06:59:39,812 INFO [MeasurementPlugin] Register TCP Socket proxy for domain: socket
2007-07-24 06:59:39,812 INFO [MeasurementPlugin] Register HTTP proxy for domain: http
2007-07-24 06:59:39,812 INFO [MeasurementPlugin] Register HTTP proxy for domain: url.availability
2007-07-24 06:59:39,812 INFO [MeasurementPlugin] Register FTP proxy for domain: ftp
2007-07-24 06:59:39,815 INFO [ProductPluginManager] Loading plugin: sqlquery-plugin.jar
2007-07-24 06:59:39,829 INFO [SQLQueryMeasurementPlugin] Registered proxy for domain: sql
2007-07-24 06:59:39,830 INFO [ProductPluginManager] Loading plugin: bind-plugin.jar
2007-07-24 06:59:39,844 INFO [ProductPluginManager] Loading plugin: postgresql-plugin.jar
2007-07-24 06:59:39,876 INFO [ProductPluginManager] Loading plugin: weblogic-plugin.jar
2007-07-24 06:59:40,191 INFO [ProductPluginManager] Loading plugin: mqseries-plugin.jar
2007-07-24 06:59:40,211 INFO [ProductPluginManager] Loading plugin: vmware-plugin.jar
2007-07-24 06:59:40,237 INFO [ProductPluginManager] Loading plugin: geronimo-plugin.xml
2007-07-24 06:59:40,252 INFO [ProductPluginManager] Loading plugin: memcached-plugin.jar
2007-07-24 06:59:40,257 INFO [ProductPluginManager] Loading plugin: ntds-plugin.jar
2007-07-24 06:59:40,264 INFO [ProductPluginManager] Loading plugin: db2-plugin.jar
2007-07-24 06:59:40,293 INFO [ProductPluginManager] Loading plugin: servlet-plugin.jar
2007-07-24 06:59:40,357 INFO [ProductPluginManager] Loading plugin: mule-plugin.jar
2007-07-24 06:59:40,379 INFO [ProductPluginManager] Loading plugin: dotnet-plugin.jar
2007-07-24 06:59:40,396 INFO [ProductPluginManager] Loading plugin: apache-plugin.jar
2007-07-24 06:59:40,485 INFO [ProductPluginManager] Loading plugin: resin-plugin.xml
2007-07-24 06:59:40,496 INFO [ProductPluginManager] Loading plugin: netdevice-plugin.jar
2007-07-24 06:59:40,540 INFO [ProductPluginManager] Loading plugin: hqagent-plugin.jar
2007-07-24 06:59:40,549 INFO [ProductPluginManager] Loading plugin: sybase-plugin.jar
2007-07-24 06:59:40,565 INFO [ProductPluginManager] Loading plugin: websphere-plugin.jar
2007-07-24 06:59:40,639 INFO [ProductPluginManager] Loading plugin: activemq-plugin.xml
2007-07-24 06:59:40,651 INFO [ProductPluginManager] Loading plugin: postfix-plugin.jar
2007-07-24 06:59:40,659 INFO [ProductPluginManager] Loading plugin: mysql-plugin.jar
2007-07-24 06:59:40,688 INFO [ProductPluginManager] Loading plugin: exchange-plugin.jar
2007-07-24 06:59:40,714 INFO [ProductPluginManager] Loading plugin: mssql-plugin.jar
2007-07-24 06:59:40,759 INFO [ProductPluginManager] Loading plugin: ntp-plugin.jar
2007-07-24 06:59:40,766 INFO [ProductPluginManager] Loading plugin: jboss-plugin.jar
2007-07-24 06:59:40,818 INFO [ProductPluginManager] Loading plugin: jmx-plugin.xml
2007-07-24 06:59:40,830 INFO [ProductPluginManager] Loading plugin: nagios-plugin.jar
2007-07-24 06:59:40,838 INFO [ProductPluginManager] Loading plugin: iplanet-plugin.jar
2007-07-24 06:59:40,859 INFO [iPlanetMeasurementPlugin] Defaulting to 6.1, loading: webserv61.mib
2007-07-24 06:59:40,865 INFO [ProductPluginManager] Loading plugin: oracle-plugin.jar
2007-07-24 06:59:40,906 INFO [ProductPluginManager] Loading plugin: iis-plugin.jar
2007-07-24 06:59:40,958 INFO [AgentDaemon] Product Plugin Manager initalized
2007-07-24 06:59:40,958 INFO [AgentCommandsServer] Agent commands started up
2007-07-24 06:59:40,969 INFO [AutoinventoryCommandsServer] Autoinventory Commands Server started up
2007-07-24 06:59:40,971 INFO [ControlCommandsServer] Control Commands Server started up
2007-07-24 06:59:41,038 INFO [CommandsServer] Commands Server started up
2007-07-24 06:59:41,083 INFO [SenderThread] Maximum metric batch size set to 500
2007-07-24 06:59:41,110 INFO [MeasurementCommandsServer] Measurement Commands Server started up
2007-07-24 06:59:41,111 INFO [ConfigPopulateThread] Starting config populate thread
2007-07-24 06:59:41,182 INFO [AgentDaemon] Agent started successfully
2007-07-24 06:59:41,183 INFO [AgentConnection] 127.0.0.1 -> agent:ping
2007-07-24 06:59:41,343 INFO [AutoinventoryCommandsServer] Autoinventory report successfully sent to server.
2007-07-24 06:59:41,355 INFO [AgentConnection] 127.0.0.1 -> bizapp:getCAMServer
2007-07-24 06:59:51,358 INFO [RuntimeAutodiscoverer] Running runtime autodiscovery for FileServer
2007-07-24 06:59:51,365 INFO [RuntimeAutodiscoverer] FileServer discovery took 0.003
2007-07-24 06:59:51,367 INFO [RuntimeAutodiscoverer] Running runtime autodiscovery for Tomcat 5.5
2007-07-24 07:00:26,288 WARN [ScheduleThread] Collection of metric: 'Tomcat 5.5:hyperic-hq:type=ServerMeasurement:Uptime:jmxUrl=http%3A//localhost%3A8080,jmxUser=%jmxUser%,jmxPass=%jmxPass%' took: 45178ms
2007-07-24 07:01:11,302 INFO [RuntimeAutodiscoverer] Tomcat 5.5 discovery took 79.931
2007-07-24 07:01:11,302 INFO [RuntimeAutodiscoverer] Running runtime autodiscovery for ProcessServer
2007-07-24 07:01:11,310 INFO [RuntimeAutodiscoverer] ProcessServer discovery took 0.003
2007-07-24 07:01:11,310 INFO [RuntimeAutodiscoverer] Running runtime autodiscovery for NetworkServer
2007-07-24 07:01:11,316 INFO [RuntimeAutodiscoverer] NetworkServer discovery took 0.002
2007-07-24 07:01:11,316 INFO [RuntimeAutodiscoverer] Running runtime autodiscovery for Net Services
2007-07-24 07:01:11,320 INFO [RuntimeAutodiscoverer] Net Services discovery took 0.001
2007-07-24 07:01:56,295 WARN [ScheduleThread] Collection of metric: 'Tomcat 5.5 Connector:Catalina:type=GlobalRequestProcessor,name=http-8080:requestCount:jmxUrl=http%3A//localhost%3A8080,jmxUser=%jmxUser%,jmxPass=%jmxPass%__RATE__=1m' took: 90000ms
2007-07-24 07:02:41,439 WARN [ScheduleThread] Collection of metric: 'Tomcat 5.5 Connector:Catalina:type=GlobalRequestProcessor,name=http-8080:requestCount:jmxUrl=http%3A//localhost%3A8080,jmxUser=%jmxUser%,jmxPass=%jmxPass%' took: 45144ms
2007-07-24 07:03:26,586 WARN [ScheduleThread] Collection of metric: 'Tomcat 5.5 Connector:Catalina:type=GlobalRequestProcessor,name=http-8080:bytesSent:jmxUrl=http%3A//localhost%3A8080,jmxUser=%jmxUser%,jmxPass=%jmxPass%__RATE__=1m' took: 45147ms
2007-07-24 07:04:11,727 WARN [ScheduleThread] Collection of metric: 'Tomcat 5.5 Connector:Catalina:type=GlobalRequestProcessor,name=http-8080:bytesReceived:jmxUrl=http%3A//localhost%3A8080,jmxUser=%jmxUser%,jmxPass=%jmxPass%__RATE__=1m' took: 45141ms
2007-07-24 07:04:56,868 WARN [ScheduleThread] Collection of metric: 'Tomcat 5.5 Connector:Catalina:type=GlobalRequestProcessor,name=http-8080:bytesReceived:jmxUrl=http%3A//localhost%3A8080,jmxUser=%jmxUser%,jmxPass=%jmxPass%' took: 45141ms
2007-07-24 07:05:42,016 WARN [ScheduleThread] Collection of metric: 'Tomcat 5.5 Connector:Catalina:type=GlobalRequestProcessor,name=http-8080:bytesSent:jmxUrl=http%3A//localhost%3A8080,jmxUser=%jmxUser%,jmxPass=%jmxPass%' took: 45146ms
2007-07-24 07:06:11,335 ERROR [RuntimeAutodiscoverer] Error sending runtime report to server: org.hyperic.hq.bizapp.client.AgentCallbackClientException: IO error: java.net.SocketTimeoutException: Read timed out
2007-07-24 07:06:27,156 WARN [ScheduleThread] Collection of metric: 'Tomcat 5.5 Webapp:hyperic-hq:type=Context,name=/ROOT:TotalTime:jmxUrl=http%3A//localhost%3A8080,jmxUser=%jmxUser%,jmxPass=%jmxPass%' took: 45140ms
2007-07-24 07:07:12,305 WARN [ScheduleThread] Collection of metric: 'Tomcat 5.5 Webapp:hyperic-hq:type=Context,name=/ROOT:RequestCount:jmxUrl=http%3A//localhost%3A8080,jmxUser=%jmxUser%,jmxPass=%jmxPass%' took: 45149ms

[wartus]

still struggling with this...

http://my.domain.com:8080/hyperic-hq/jmx-attributes?dmp=hyperic-hq:*

Time: 1185791351294
Remote: 77.250.0.132
User: null
Server: Apache Tomcat/5.5.9

Name: hyperic-hq:type=RuntimeInfo
FreeMemory: 427183664
ActiveThreadGroupCount: 1
TotalMemory: 465174528
ActiveThreadCount: 73

Name: hyperic-hq:type=Context,name=/ROOT
BytesReceived: 0
BytesSent: 0
Available: 1
DocBase: /home/tomcat/webapps/ROOT/
SessionsCreated: 0
RequestCount: 2
AvgTime: 524
TotalTime: 1049
MinTime: 1
SessionsDestroyed: 0
ErrorCount: 0
ContextName: /ROOT
MaxTime: 1048

Name: hyperic-hq:type=ProcessInfo
CpuUsage: 0.0028888888888888888
MemShare: 12562432
MemVsize: 673435648
TimeUser: 6390
MemPageFaults: 28209
MemSize: 673435648
OpenFd: 20
MemResident: 102866944
TimeSys: 170

Name: hyperic-hq:type=ServerMeasurement
Available: 1
ServerInfo: Apache Tomcat/5.5.9
Uptime: 1486359

Name: hyperic-hq:type=Context,name=/hyperic-hq
BytesReceived: 0
BytesSent: 0
Available: 1
DocBase: /home/tomcat/webapps/hyperic-hq/
SessionsCreated: 0
RequestCount: 26
AvgTime: 45140
TotalTime: 1173654
MinTime: 45127
SessionsDestroyed: 0
ErrorCount: 0
ContextName: /hyperic-hq
MaxTime: 45234

Name: hyperic-hq:type=AllContextInfo
RequestCount: 28
TotalTime: 1174703




http://my.domain.com:8080/hyperic-hq/jmx-attributes?dmp=*

takes 65 seconds to load on the problem servers, only a few on a good server.

[wartus]

it was a mysql connection issue