Service Virtualization

 View Only
Expand all | Collapse all

VSE in server is going down.

  • 1.  VSE in server is going down.

    Posted Aug 10, 2020 08:20 AM
    Hello All,

    I am facing issue of VSE going down when some recording is there in place.
    I need to clear out files present under lisatmp and tmp folders and then i restart VSE.

    Is there any other way like increasing size or something like that by which this issue will be solved permanently?
    Please suggest.

    Thanks, for your help in advance.

    Regards,
    Saili Inamdar


  • 2.  RE: VSE in server is going down.

    Broadcom Employee
    Posted Aug 10, 2020 08:24 AM
    Hi Saili
               VSE Server going down is very unusual. Did you see any errors in VSE log about heap memory reaching its limit ? Can I know the -Xmx value that has been set in vmoptions file for VSE service?

    --
    regards
    Sankar Natarajan





  • 3.  RE: VSE in server is going down.

    Posted Aug 10, 2020 08:49 AM
      |   view attached
    Hi Sankar,
    Thanks for your reply.
    Please check the attached file where there is size mentioned. 
    I checked for VSEManager-VMOmptions file present under bin folder.

    Let me know if i am using the correct one.

    Thanks,
    Saili Inamdar


  • 4.  RE: VSE in server is going down.

    Broadcom Employee
    Posted Aug 10, 2020 09:34 AM
    Hi Saili
              Thats a different file. Please check the following files:

    VirtualServiceEnvironment.vmoptions
    VirtualServiceEnvironmentService.vmoptions

    --
    regards
    Sankar Natarajan





  • 5.  RE: VSE in server is going down.

    Posted Aug 10, 2020 11:49 PM
    Hi Saili,

    There is an option of increasing the Java Memory Heap size using -Xmx & -Xms options. However, this could also be limited because of the hardware available for the machine.

    i assume you are using an external DB for your setup. Are you having high volume of transactions while recording? any specific errors visible in logs?

    thanks.

    ------------------------------
    Regards,
    Vaibhav Jain
    Capgemini
    ------------------------------



  • 6.  RE: VSE in server is going down.

    Posted Aug 28, 2020 08:33 AM
    Hi Sankar & Vaibhav,
    Thank you for your reply.
    I increased the size in both the files you have mentioned above but still facing this VSE down issue.
    The service automatically stops and shows in running state for some time and then shows stopped.
    Even after starting, it takes long time to reflect on portal.
    Kindly suggest.

    Regards,
    Saili Inamdar


  • 7.  RE: VSE in server is going down.

    Broadcom Employee
    Posted Aug 28, 2020 08:43 AM
    HI Saili
          Please share vse.log file for analysis.

    --
    regards
    Sankar Natarajan





  • 8.  RE: VSE in server is going down.

    Posted Aug 28, 2020 09:19 AM
      |   view attached
    Hi Sankar,
    PFA vse.log file.

    Regards,
    Saili Inamdar

    Attachment(s)

    txt
    vse.log.txt   1 KB 1 version


  • 9.  RE: VSE in server is going down.

    Posted Aug 28, 2020 09:36 AM
    Hi Saili,

    Could you verify the log file as it doesn't seem to be a correct one.

    Thanks

    ------------------------------
    Regards,
    Vaibhav Jain
    Capgemini
    ------------------------------



  • 10.  RE: VSE in server is going down.

    Posted Aug 28, 2020 11:42 AM
      |   view attached
    Hi Vaibhav,

    Please find attached log files

    Attachment(s)

    zip
    vse.log.zip   924 KB 1 version


  • 11.  RE: VSE in server is going down.

    Posted Aug 28, 2020 01:20 PM
    Hi Bala,

    I can see error associated with service "VS_TT_Account_Identity_V8_Run", try stopping/removing that service and observe the VSE behavior.

    If this solves your issue, then issue may be related to the script /configuration of that particular virtual service.

    Thanks

    ------------------------------
    Regards,
    Vaibhav Jain
    Capgemini
    ------------------------------



  • 12.  RE: VSE in server is going down.

    Posted Aug 31, 2020 06:05 PM
    Thanks Vaibhav, its working fine now.We fixed the issue


  • 13.  RE: VSE in server is going down.

    Posted Sep 01, 2020 02:50 AM
    Cool. Thanks for the update.

    ------------------------------
    Regards,
    Vaibhav Jain
    Capgemini
    ------------------------------



  • 14.  RE: VSE in server is going down.

    Posted Sep 01, 2020 08:35 AM
      |   view attached
    Hi Vaibhav,
    It worked fine for one day and again we are facing same issue.
    Attaching vse.log file.
    Kindly check and let us know.

    Thanks,
    Saili Inamdar

    Attachment(s)

    zip
    VSE-LogFiles.zip   2.27 MB 1 version


  • 15.  RE: VSE in server is going down.

    Posted Sep 01, 2020 10:32 AM
    Hi Saili,

    Can you highlight the log file which was modified at last as for me all file timestamps are same.

    Also, do you have large sql queries and messages >1 MB in your services?

    Thanks

    ------------------------------
    Regards,
    Vaibhav Jain
    Capgemini
    ------------------------------



  • 16.  RE: VSE in server is going down.

    Posted Sep 01, 2020 01:43 PM
    Hi Vaibhav,

    1) vse.log is the latest file after that vse.log.1 is next latest file.
    2) There are no sql queries and messages > 1MB


  • 17.  RE: VSE in server is going down.

    Posted Sep 04, 2020 01:25 PM
    Edited by Bala Subrahmanyam Tontepu Sep 04, 2020 01:26 PM
    All,
    We are seeing below error messages in logs.We are facing VSE issue continuously
    VSE.log

    2020-09-02 06:42:07,778Z (01:42) [SI-Read-Thread] ERROR System.err - Exception in thread "SI-Read-Thread" java.lang.OutOfMemoryError: GC overhead limit exceeded
    2020-09-02 06:42:10,598Z (01:42) [ServerRequestResponder 356] WARN com.itko.lisa.net.MessageHandler - Whoa there. Big OUTBOUND message, 10580588 bytes: ActiveMQObjectMessage {commandId = 0, responseRequired = false, messageId = null, originalDestination = null, originalTransactionId = null, producerId = null, destination = null, transactionId = null, expiration = 0, timestamp = 0, arrival = 0, brokerInTime = 0, brokerOutTime = 0, correlationId = null, replyTo = null, persistent = false, type = null, priority = 0, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = com.itko.activemq.util.ByteSequence@1b103e38, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 10580588, properties = null, readOnlyProperties = false, readOnlyBody = false, droppable = false}
    2020-09-02 06:42:10,599Z (01:42) [ServerRequestResponder 356] WARN com.itko.lisa.net.MessageHandler - Object msg, toString first 1024 chars: {TT_Account_JanRain_V2=com.itko.lisa.vse.VirtualServiceStatus@178c775f, TT_Accounts_V2=com.itko.lisa.vse.VirtualServiceStatus@4a147926, TT_Offers_OfferDetails=com.itko.lisa.vse.VirtualServiceStatus@10c515e, TT_GLS=com.itko.lisa.vse.VirtualServiceStatus@7a42e79, Loyalty_MW_Offers_V13=com.itko.lisa.vse.VirtualServiceStatus@6ced96d9, 9206=com.itko.lisa.vse.VirtualServiceStatus@f88acbb, Accounts_Identity=com.itko.lisa.vse.VirtualServiceStatus@29f99cd5, Accounts_Profile=com.itko.lisa.vse.VirtualServiceStatus@42d5f4b2, BlackPool=com.itko.lisa.vse.VirtualServiceStatus@420ef96f, CAM-9120_1=com.itko.lisa.vse.VirtualServiceStatus@8a978e1, cim=com.itko.lisa.vse.VirtualServiceStatus@6dd34215, TT_Menu_Order_V28=com.itko.lisa.vse.VirtualServiceStatus@4e4b79, Core_mar=com.itko.lisa.vse.VirtualServiceStatus@769307a8, CORES-5336=com.itko.lisa.vse.VirtualServiceStatus@1272085, Cores=com.itko.lisa.vse.VirtualServiceStatus@7340b65b, DD_Test=com.itko.lisa.vse.VirtualServiceStatus@36470552, Demo06.06=com.itko.lisa.vse.VirtualServi


    We observed below error in Registry.logs

    2020-09-03 12:39:30,001Z (07:39) [qtp1221664517-19560] INFO  com.itko.lisa.invoke.resource.AclResource - pingUserSession: securitytoken = 6182AE47EDE111EA87070A93191A9410
    2020-09-03 12:39:30,295Z (07:39) [qtp1221664517-19557] WARN  com.ca.sv.invoke.controller.v2.VseServiceController - Returning NOT FOUND: Could not find VSE: gGVzeBm64mNeDiRdpe53PGJ7fFT6uOP
    com.ca.sv.invoke.exception.VSEServerNotFoundException: Could not find VSE: gGVzeBm64mNeDiRdpe53PGJ7fFT6uOP
     at sun.reflect.GeneratedConstructorAccessor418.newInstance(Unknown Source)
     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
     at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
     at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
     at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:250)
     at com.ca.sv.invoke.service.v2.VseServerService.getServiceCountForVSE(VseServerService.groovy:104)
     at com.ca.sv.invoke.service.v2.VseServerService$getServiceCountForVSE$3.call(Unknown Source)
     at com.ca.sv.invoke.controller.v2.VseServiceController.getServices(VseServiceController.groovy:107)
     at com.ca.sv.invoke.controller.v2.VseServiceController.getServices(VseServiceController.groovy)
     at org.grails.core.DefaultGrailsControllerClass$MethodHandleInvoker.invoke(DefaultGrailsControllerClass.java:223)
     at org.grails.core.DefaultGrailsControllerClass.invoke(DefaultGrailsControllerClass.java:188)
     at org.grails.web.mapping.mvc.UrlMappingsInfoHandlerAdapter.handle(UrlMappingsInfoHandlerAdapter.groovy:90)
     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
     at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
     at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
     at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
     at com.ca.devtest.http.XSSFilter.doFilter(XSSFilter.java:72)
     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
     at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
     at com.ca.devtest.acl.servlet.filters.RemoteAuthenticationFilter.doFilter(RemoteAuthenticationFilter.java:241)
     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
     at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
     at org.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:77)
     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
     at org.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67)
     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
     at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
     at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
     at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103)
     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
     at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:130)
     at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:66)
     at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:105)
     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
     at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:123)
     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
     at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
     at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
     at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
     at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
     at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
     at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
     at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
     at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
     at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
     at org.eclipse.jetty.server.Server.handle(Server.java:524)
     at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
     at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
     at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
     at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
     at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
     at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
     at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
     at java.lang.Thread.run(Thread.java:745)



  • 18.  RE: VSE in server is going down.

    Posted Sep 04, 2020 01:59 PM
    Hi bala,

    It seems to be a Java heap memory issue. Do you have any scripts in your service or any test case reading & writing files?

    Also, are you running all devtest components on single machine or in distributed environment?

    Either try to increase the JVM memory or upgrading the configuration of your devtest server.

    Thanks

    ------------------------------
    Regards,
    Vaibhav Jain
    Capgemini
    ------------------------------



  • 19.  RE: VSE in server is going down.

    Posted Sep 04, 2020 02:13 PM
    Hi Vaibhav,

    Please find in-line my comments

    It seems to be a Java heap memory issue. Do you have any scripts in your service or any test case reading & writing files?

    - No we are not writing or reading any files

    Also, are you running all devtest components on single machine or in distributed environment?

    - Yes all are in single server

    Either try to increase the JVM memory or upgrading the configuration of your devtest server.

    Increased JVM memory (-Xmx2048m from -Xmx512m) in RegistryService.vmoptions

    Still we are facing issue


  • 20.  RE: VSE in server is going down.

    Posted Sep 04, 2020 02:22 PM
    Edited by Vaibhav Jain Sep 04, 2020 02:46 PM
    Hi Bala,

    Increasing memory in vmoption doesn't help always as it is still dependent on the machine's configuration. 

    As you are facing frequent failure, I would suggest using a distributed setup.

    Thanks

    ------------------------------
    Regards,
    Vaibhav Jain
    Capgemini
    ------------------------------