Hi bala,
It seems to be a Java heap memory issue. Do you have any scripts in your service or any test case reading & writing files?
Also, are you running all devtest components on single machine or in distributed environment?
Either try to increase the JVM memory or upgrading the configuration of your devtest server.
Thanks
------------------------------
Regards,
Vaibhav Jain
Capgemini
------------------------------
Original Message:
Sent: 09-04-2020 01:25 PM
From: Bala Subrahmanyam Tontepu
Subject: VSE in server is going down.
All,
We are seeing below error messages in logs.We are facing VSE issue continuously
VSE.log
2020-09-02 06:42:07,778Z (01:42) [SI-Read-Thread] ERROR System.err - Exception in thread "SI-Read-Thread" java.lang.OutOfMemoryError: GC overhead limit exceeded
2020-09-02 06:42:10,598Z (01:42) [ServerRequestResponder 356] WARN com.itko.lisa.net.MessageHandler - Whoa there. Big OUTBOUND message, 10580588 bytes: ActiveMQObjectMessage {commandId = 0, responseRequired = false, messageId = null, originalDestination = null, originalTransactionId = null, producerId = null, destination = null, transactionId = null, expiration = 0, timestamp = 0, arrival = 0, brokerInTime = 0, brokerOutTime = 0, correlationId = null, replyTo = null, persistent = false, type = null, priority = 0, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = com.itko.activemq.util.ByteSequence@1b103e38, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 10580588, properties = null, readOnlyProperties = false, readOnlyBody = false, droppable = false}
2020-09-02 06:42:10,599Z (01:42) [ServerRequestResponder 356] WARN com.itko.lisa.net.MessageHandler - Object msg, toString first 1024 chars: {TT_Account_JanRain_V2=com.itko.lisa.vse.VirtualServiceStatus@178c775f, TT_Accounts_V2=com.itko.lisa.vse.VirtualServiceStatus@4a147926, TT_Offers_OfferDetails=com.itko.lisa.vse.VirtualServiceStatus@10c515e, TT_GLS=com.itko.lisa.vse.VirtualServiceStatus@7a42e79, Loyalty_MW_Offers_V13=com.itko.lisa.vse.VirtualServiceStatus@6ced96d9, 9206=com.itko.lisa.vse.VirtualServiceStatus@f88acbb, Accounts_Identity=com.itko.lisa.vse.VirtualServiceStatus@29f99cd5, Accounts_Profile=com.itko.lisa.vse.VirtualServiceStatus@42d5f4b2, BlackPool=com.itko.lisa.vse.VirtualServiceStatus@420ef96f, CAM-9120_1=com.itko.lisa.vse.VirtualServiceStatus@8a978e1, cim=com.itko.lisa.vse.VirtualServiceStatus@6dd34215, TT_Menu_Order_V28=com.itko.lisa.vse.VirtualServiceStatus@4e4b79, Core_mar=com.itko.lisa.vse.VirtualServiceStatus@769307a8, CORES-5336=com.itko.lisa.vse.VirtualServiceStatus@1272085, Cores=com.itko.lisa.vse.VirtualServiceStatus@7340b65b, DD_Test=com.itko.lisa.vse.VirtualServiceStatus@36470552, Demo06.06=com.itko.lisa.vse.VirtualServi
We observed below error in Registry.logs
2020-09-03 12:39:30,001Z (07:39) [qtp1221664517-19560] INFO com.itko.lisa.invoke.resource.AclResource - pingUserSession: securitytoken = 6182AE47EDE111EA87070A93191A9410
2020-09-03 12:39:30,295Z (07:39) [qtp1221664517-19557] WARN com.ca.sv.invoke.controller.v2.VseServiceController - Returning NOT FOUND: Could not find VSE: gGVzeBm64mNeDiRdpe53PGJ7fFT6uOP
com.ca.sv.invoke.exception.VSEServerNotFoundException: Could not find VSE: gGVzeBm64mNeDiRdpe53PGJ7fFT6uOP
at sun.reflect.GeneratedConstructorAccessor418.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:250)
at com.ca.sv.invoke.service.v2.VseServerService.getServiceCountForVSE(VseServerService.groovy:104)
at com.ca.sv.invoke.service.v2.VseServerService$getServiceCountForVSE$3.call(Unknown Source)
at com.ca.sv.invoke.controller.v2.VseServiceController.getServices(VseServiceController.groovy:107)
at com.ca.sv.invoke.controller.v2.VseServiceController.getServices(VseServiceController.groovy)
at org.grails.core.DefaultGrailsControllerClass$MethodHandleInvoker.invoke(DefaultGrailsControllerClass.java:223)
at org.grails.core.DefaultGrailsControllerClass.invoke(DefaultGrailsControllerClass.java:188)
at org.grails.web.mapping.mvc.UrlMappingsInfoHandlerAdapter.handle(UrlMappingsInfoHandlerAdapter.groovy:90)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
at com.ca.devtest.http.XSSFilter.doFilter(XSSFilter.java:72)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at com.ca.devtest.acl.servlet.filters.RemoteAuthenticationFilter.doFilter(RemoteAuthenticationFilter.java:241)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:77)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:130)
at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:66)
at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:105)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:123)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:524)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745)
Original Message:
Sent: 09-01-2020 01:42 PM
From: Bala Subrahmanyam Tontepu
Subject: VSE in server is going down.
Hi Vaibhav,
1) vse.log is the latest file after that vse.log.1 is next latest file.
2) There are no sql queries and messages > 1MB
Original Message:
Sent: 09-01-2020 10:32 AM
From: Vaibhav Jain
Subject: VSE in server is going down.
Hi Saili,
Can you highlight the log file which was modified at last as for me all file timestamps are same.
Also, do you have large sql queries and messages >1 MB in your services?
Thanks
------------------------------
Regards,
Vaibhav Jain
Capgemini
Original Message:
Sent: 09-01-2020 08:34 AM
From: Saili Joshi
Subject: VSE in server is going down.
Hi Vaibhav,
It worked fine for one day and again we are facing same issue.
Attaching vse.log file.
Kindly check and let us know.
Thanks,
Saili Inamdar
Original Message:
Sent: 09-01-2020 02:50 AM
From: Vaibhav Jain
Subject: VSE in server is going down.
Cool. Thanks for the update.
------------------------------
Regards,
Vaibhav Jain
Capgemini
Original Message:
Sent: 08-31-2020 06:05 PM
From: Bala Subrahmanyam Tontepu
Subject: VSE in server is going down.
Thanks Vaibhav, its working fine now.We fixed the issue
Original Message:
Sent: 08-28-2020 01:20 PM
From: Vaibhav Jain
Subject: VSE in server is going down.
Hi Bala,
I can see error associated with service "VS_TT_Account_Identity_V8_Run", try stopping/removing that service and observe the VSE behavior.
If this solves your issue, then issue may be related to the script /configuration of that particular virtual service.
Thanks
------------------------------
Regards,
Vaibhav Jain
Capgemini
Original Message:
Sent: 08-28-2020 11:41 AM
From: Bala Subrahmanyam Tontepu
Subject: VSE in server is going down.
Hi Vaibhav,
Please find attached log files
Original Message:
Sent: 08-28-2020 09:36 AM
From: Vaibhav Jain
Subject: VSE in server is going down.
Hi Saili,
Could you verify the log file as it doesn't seem to be a correct one.
Thanks
------------------------------
Regards,
Vaibhav Jain
Capgemini
Original Message:
Sent: 08-28-2020 09:19 AM
From: Saili Joshi
Subject: VSE in server is going down.
Hi Sankar,
PFA vse.log file.
Regards,
Saili Inamdar
Original Message:
Sent: 08-28-2020 08:43 AM
From: Sankar Natarajan
Subject: VSE in server is going down.
HI Saili
Please share vse.log file for analysis.
--
Original Message:
Sent: 8/28/2020 8:33:00 AM
From: Saili Inamdar
Subject: RE: VSE in server is going down.
Hi Sankar & Vaibhav,
Thank you for your reply.
I increased the size in both the files you have mentioned above but still facing this VSE down issue.
The service automatically stops and shows in running state for some time and then shows stopped.
Even after starting, it takes long time to reflect on portal.
Kindly suggest.
Regards,
Saili Inamdar
Original Message:
Sent: 08-10-2020 11:48 PM
From: Vaibhav Jain
Subject: VSE in server is going down.
Hi Saili,
There is an option of increasing the Java Memory Heap size using -Xmx & -Xms options. However, this could also be limited because of the hardware available for the machine.
i assume you are using an external DB for your setup. Are you having high volume of transactions while recording? any specific errors visible in logs?
thanks.
------------------------------
Regards,
Vaibhav Jain
Capgemini
Original Message:
Sent: 08-10-2020 08:19 AM
From: Saili Joshi
Subject: VSE in server is going down.
Hello All,
I am facing issue of VSE going down when some recording is there in place.
I need to clear out files present under lisatmp and tmp folders and then i restart VSE.
Is there any other way like increasing size or something like that by which this issue will be solved permanently?
Please suggest.
Thanks, for your help in advance.
Regards,
Saili Inamdar