We had a CA Health Check towards the end of December of 17 and have completed most of the recommendations they made. We stood up new Windows servers for WCC and EEM. WCC now sits on its own server and is no longer using the Derby database and instead going to an Oracle database on another host. EEM now sits on its own server as well.
We constantly are running into a problem where the login screen for WCC comes up just fine but once a user enters their credentials, it sits and spins and then times out. We have monitoring around this using DynaTrace which see's the issue and points right to tomcat_*.
Threads resources exhausted
Threads exhausted on tomcat_*
Looking through logs we find...
WARNING: The web application [wcc#quickview] appears to have started a thread named [pool-5-thread-3] but has failed to stop it. This is very likely to create a memory leak.
We are using WCC Release: 11.4.5.20170424-b146
Its on: Microsoft Windows Server 2012 R2 Standard
Version: 6.3.9600 Build 9600
Memory: 8GB
CPU: 4 Intel(R) Xeon(R) CPU E5-2670 0 @ 3.60GHz
Tomcat 32 wrapper.conf is as follows:
#********************************************************************
# Wrapper Properties
#********************************************************************
# Java Application
wrapper.java.command=../../jre_32/bin/java
wrapper.debug=TRUE
wrapper.java.initmemory=512
wrapper.java.maxmemory=1024
wrapper.ignore_sequence_gaps=TRUE
Tomcat wrapper.conf is as follows:
#********************************************************************
# Wrapper Properties
#********************************************************************
# Java Application
wrapper.java.command=../../jre/bin/java
wrapper.debug=false
wrapper.java.initmemory=2048
wrapper.java.maxmemory=4096
wrapper.ignore_sequence_gaps=TRUE
Here is a bit more information from the logs. We see this each time where the collector seems to bomb but we can not tell if this is the actual issue causing the threat exhaustion or is is a symptom of whatever is actually causing the thread exhaustion.
We see this in the collector.log
2018-01-16 13:47:32,879 @reporting-collector <reporting-collectors-loop> [] INFO #AutoSysJobRunExtractor # getLatestEventDateTime(SIT_AutoSys) returns Wed Jan 17 22:00:00 EST 2018
2018-01-16 13:52:33,392 @reporting-collector <reporting-collectors-loop> [] WARN #ConfigHttpInvokerRequestExecutor # Waiting for full startup of configuration services: Read timed out
2018-01-16 13:52:33,392 @reporting-collector <reporting-collectors-loop> [] ERROR #ConfigHttpInvokerRequestExecutor # Configuration Remote Interface: SocketTimeoutException encountered: Read timed out
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at org.springframework.remoting.httpinvoker.SimpleHttpInvokerRequestExecutor.validateResponse(SimpleHttpInvokerRequestExecutor.java:178)
at org.springframework.remoting.httpinvoker.SimpleHttpInvokerRequestExecutor.doExecuteRequest(SimpleHttpInvokerRequestExecutor.java:92)
at com.ca.wcc.config.remote.httpinvoker.ConfigHttpInvokerRequestExecutor.doExecuteRequest(ConfigHttpInvokerRequestExecutor.java:75)
at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:136)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:192)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:174)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:142)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy104.getPreference(Unknown Source)
at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at com.ca.wcc.advice.CacheAdvice.get(CacheAdvice.java:67)
at sun.reflect.GeneratedMethodAccessor221.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy115.getPreference(Unknown Source)
at com.ca.uejm.datawarehouse.AutoSysJobRunExtractorWithNewFilter.getMarginForMaxEventAge(AutoSysJobRunExtractorWithNewFilter.java:303)
at com.ca.uejm.datawarehouse.AutoSysJobRunExtractorWithNewFilter.getFilter(AutoSysJobRunExtractorWithNewFilter.java:212)
at com.ca.uejm.datawarehouse.AutoSysJobRunExtractor.extract(AutoSysJobRunExtractor.java:179)
at com.ca.uejm.datawarehouse.wcc.AutoSysJobRunExtractorWrapper.instanciateAndRun(AutoSysJobRunExtractorWrapper.java:56)
at com.ca.uejm.datawarehouse.wcc.BaseWrapperWithServerLoop.instanciateAndRun(BaseWrapperWithServerLoop.java:56)
at com.ca.uejm.datawarehouse.wcc.BaseWrapper.run(BaseWrapper.java:76)
at com.ca.uejm.datawarehouse.wcc.AutosysCollector.run(AutosysCollector.java:71)
at com.ca.uejm.datawarehouse.wcc.ReportingCollectorsLoopService.collect(ReportingCollectorsLoopService.java:380)
at com.ca.uejm.datawarehouse.wcc.ReportingCollectorsLoopService.doWork(ReportingCollectorsLoopService.java:231)
at com.ca.uejm.datawarehouse.wcc.BaseLoopImpl.work(BaseLoopImpl.java:230)
at com.ca.uejm.datawarehouse.wcc.LoopWithLock.workNoLock(LoopWithLock.java:71)
at com.ca.uejm.datawarehouse.wcc.LoopWithLock.work(LoopWithLock.java:99)
at com.ca.uejm.datawarehouse.wcc.BaseLoopImpl.runLoop(BaseLoopImpl.java:193)
at com.ca.uejm.datawarehouse.wcc.ReportingCollectorsLoopService.runLoop(ReportingCollectorsLoopService.java:199)
at com.ca.uejm.datawarehouse.wcc.BaseLoopImpl$1.run(BaseLoopImpl.java:141)
at java.lang.Thread.run(Thread.java:745)
2018-01-16 13:52:33,408 @reporting-collector <reporting-collectors-loop> [] ERROR #BaseWrapperWithServerLoop # for server SIT_AutoSys: Could not access HTTP invoker remote service at [http://localhost:10132/wcc-configuration/cfmremote/remoteGlobalPrefService]; nested exception is com.ca.wcc.config.api.exceptions.WCCException: Read timed out
org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [http://localhost:10132/wcc-configuration/cfmremote/remoteGlobalPrefService]; nested exception is com.ca.wcc.config.api.exceptions.WCCException: Read timed out
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:213)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:145)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy104.getPreference(Unknown Source)
at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at com.ca.wcc.advice.CacheAdvice.get(CacheAdvice.java:67)
at sun.reflect.GeneratedMethodAccessor221.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy115.getPreference(Unknown Source)
at com.ca.uejm.datawarehouse.AutoSysJobRunExtractorWithNewFilter.getMarginForMaxEventAge(AutoSysJobRunExtractorWithNewFilter.java:303)
at com.ca.uejm.datawarehouse.AutoSysJobRunExtractorWithNewFilter.getFilter(AutoSysJobRunExtractorWithNewFilter.java:212)
at com.ca.uejm.datawarehouse.AutoSysJobRunExtractor.extract(AutoSysJobRunExtractor.java:179)
at com.ca.uejm.datawarehouse.wcc.AutoSysJobRunExtractorWrapper.instanciateAndRun(AutoSysJobRunExtractorWrapper.java:56)
at com.ca.uejm.datawarehouse.wcc.BaseWrapperWithServerLoop.instanciateAndRun(BaseWrapperWithServerLoop.java:56)
at com.ca.uejm.datawarehouse.wcc.BaseWrapper.run(BaseWrapper.java:76)
at com.ca.uejm.datawarehouse.wcc.AutosysCollector.run(AutosysCollector.java:71)
at com.ca.uejm.datawarehouse.wcc.ReportingCollectorsLoopService.collect(ReportingCollectorsLoopService.java:380)
at com.ca.uejm.datawarehouse.wcc.ReportingCollectorsLoopService.doWork(ReportingCollectorsLoopService.java:231)
at com.ca.uejm.datawarehouse.wcc.BaseLoopImpl.work(BaseLoopImpl.java:230)
at com.ca.uejm.datawarehouse.wcc.LoopWithLock.workNoLock(LoopWithLock.java:71)
at com.ca.uejm.datawarehouse.wcc.LoopWithLock.work(LoopWithLock.java:99)
at com.ca.uejm.datawarehouse.wcc.BaseLoopImpl.runLoop(BaseLoopImpl.java:193)
at com.ca.uejm.datawarehouse.wcc.ReportingCollectorsLoopService.runLoop(ReportingCollectorsLoopService.java:199)
at com.ca.uejm.datawarehouse.wcc.BaseLoopImpl$1.run(BaseLoopImpl.java:141)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.ca.wcc.config.api.exceptions.WCCException: Read timed out
at com.ca.wcc.config.remote.httpinvoker.ConfigHttpInvokerRequestExecutor.doExecuteRequest(ConfigHttpInvokerRequestExecutor.java:108)
at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:136)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:192)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:174)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:142)
... 38 more
This then just keeps repeating in the log over and over until we conduct the restart of WCC.
We see this in the config-services.log...
2018-01-16 14:15:36,991 @configservices <clusterHealthChecker> [] ERROR #ClusterHealtAnalyzer # Error checking cluster
java.util.NoSuchElementException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1439)
at java.util.HashMap$KeyIterator.next(HashMap.java:1461)
at com.ca.wcc.health.ClusterHealtAnalyzer$ClusterSanityAnalyzerTask.run(ClusterHealtAnalyzer.java:68)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-01-16 14:16:36,981 @configservices <clusterHealthChecker> [] ERROR #ClusterHealtAnalyzer # Error checking cluster
java.util.NoSuchElementException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1439)
at java.util.HashMap$KeyIterator.next(HashMap.java:1461)
at com.ca.wcc.health.ClusterHealtAnalyzer$ClusterSanityAnalyzerTask.run(ClusterHealtAnalyzer.java:68)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-01-16 14:17:32,560 @configservices <http-nio-127.0.0.1-10132-exec-15> [] ERROR #DefaultUserPreferenceService # Exception caught setting user preference.
User[vqq4r], Key[config.user.timezone.offset.raw], Value[-18000000]
Exception: Hazelcast Instance is not active!
Again, it repeats over and over until we restart WCC.
We see this in the remoteservices.log...
2018-01-16 13:38:32,666 @remote <http-nio-127.0.0.1-10132-exec-14> [] ERROR #ASAPI # The AutoSys application server timed out.
com.ca.autosys.services.AsTimeoutException:
at com.ca.autosys.services.JResponseQ.dequeue(Native Method)
at com.ca.autosys.services.response.FilterRspSet.dequeueResponse(FilterRspSet.java:88)
at com.ca.autosys.services.response.ApiResponseSet.hasNext(ApiResponseSet.java:70)
at com.ca.uejm.adapter.autosys.command.ASAPI.getHistoryJobRuns(ASAPI.java:5650)
at com.ca.uejm.adapter.autosys.command.ASAPI.getHistoryJobRuns(ASAPI.java:5596)
at com.ca.wcc.service.adapter.AutoSysServiceImpl.getJobRuns(AutoSysServiceImpl.java:804)
at sun.reflect.GeneratedMethodAccessor352.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:77)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy36.getJobRuns(Unknown Source)
at sun.reflect.GeneratedMethodAccessor351.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.remoting.support.RemoteInvocation.invoke(RemoteInvocation.java:205)
at org.springframework.remoting.support.DefaultRemoteInvocationExecutor.invoke(DefaultRemoteInvocationExecutor.java:38)
at org.springframework.remoting.support.RemoteInvocationBasedExporter.invoke(RemoteInvocationBasedExporter.java:78)
at org.springframework.remoting.support.RemoteInvocationBasedExporter.invokeAndCreateResult(RemoteInvocationBasedExporter.java:114)
at org.springframework.remoting.httpinvoker.HttpInvokerServiceExporter.handleRequest(HttpInvokerServiceExporter.java:78)
at org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:49)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:853)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at com.ca.wcc.context.LazyWrappingServlet.service(LazyWrappingServlet.java:109)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:474)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:783)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:798)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1434)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
CA Support has had us adjust these java memory numbers numerous times. All it has done is slowed down how often the problem takes place. We went from 3 times a day to about once a week now. The only way to resolve it is to restart. We are really at a loss as to why we keep running into this issue and I am hoping somebody else out there has or is running into this same thing.
Other basic info
We do have the reporting enabled but not really used much at all in WCC which again now uses an Oracle database. EEM appears to be just fine and never gives us any issues. Same with Autosys.
Any help would be greatly appreciated!!!! Thank you
I would also like to know if anyone else is also experiencing this at all?