Layer7 API Management

 View Only
Expand all | Collapse all

High Front End Response Times

  • 1.  High Front End Response Times

    Posted Nov 29, 2018 09:28 AM

    Dear Community,

     

    We have two clusters which are load balanced externally via f5.

    Gateway Version : 9.1

     

     

    From our customers, I am sometimes getting that our services are not responding-unreachable.

     

    When I inspect the dashboard during blackouts, in some cases I can realize that incoming requests are decreasing(for all services) for a period of time. (I am assuming this might be a network problem)

    (for ex: in the day time we are receiving around 200 req/sec, however during these blackouts we get around 3-5 req/sec)

     

     

    I implemented the cath all policy for changing the default service not found response , as users have mentioned in the community.

     

    However when I inspect the dashboard, I am seeing some weird response times from gateway(front end).

    Would this be relevant to blackouts that I am getting?

     

    I will show you maximum front end reponse times for the catch all policy with screenshots.

     

    Any ideas or direction will be greatly appreciated.

     

    Warm Regards.

     

     

     

     

     

     

     



  • 2.  Re:  High Front End Response Times

    Broadcom Employee
    Posted Nov 29, 2018 11:20 AM

    What is your catchall policy doing that it is having routing failures?  



  • 3.  Re:  High Front End Response Times

    Posted Nov 30, 2018 01:39 AM

    Dear Barry,

    The policy is just returning a template response.

    It's just a lightweight xml message, there is no policy logic inside.

     



  • 4.  Re:  High Front End Response Times

    Broadcom Employee
    Posted Nov 29, 2018 05:15 PM

    Dear halitemresayilir ,

    You can enable debug  trace to show the latency of each assertion, then you can find out which one costs most of the time. (not sure if gw9.1 has it, I can confirm gw9.2 or above can print ${trace.assertion.latency.ms})

    Working with the Debug Trace Policy - CA API Gateway - 9.4 - CA Technologies Documentation 

     

    NOTE: The front end response time may not be accurate if you have encapsulated assertion, and it includes route via http(s) assertion.

     

    Regards,

    Mark



  • 5.  Re:  High Front End Response Times

    Posted Nov 30, 2018 01:50 AM

    Dear Zhijun,

     

    In 9.1 tracing is also available.

     

    I will give it a try and share the results.



  • 6.  Re:  High Front End Response Times

    Posted Dec 07, 2018 06:57 AM

    Update;

     

    I am seeing this behavior(high frontend response times) altough the backend latency is very low.

    The relevant policy contains only just one routing assertion.

     

    The backend service returns response less than one second, but gateway can not return response to the client in 120 seconds.(120 seconds is the read time out for this service)

     

     



  • 7.  Re:  High Front End Response Times

    Broadcom Employee
    Posted Dec 09, 2018 07:59 PM

    Did you find the assertion(s) in your policy which costs the max latency?



  • 8.  Re:  High Front End Response Times

    Posted Dec 10, 2018 01:38 AM

    Dear Zhijun,

     

    I couldn't solve the problem.

     

    Some days I am having blackouts in our production environment during day time.

     

     

    I investigated two services during these blackouts.

    { first only has one assertinon (route via http) and second only has also one assertion (return template) }

     

    In the first policy there is 1 second backend latency and 100 seconds frontend latency.(sometimes during daytime)

     

    In the second policy (return template to requestor), there is no backend latency , and fronted latency is sometimes around 100 seconds. (sometimes during daytime)

     

    These issues do not happen all the time(maybe 10 x 1 minute blackouts in 6 hours) but I am suspecting they might be related to the load on the gateway.

     

    I would be very glad if we could investigate this issue further.

    (p.s ; there are 2 machines clustered, both running 9.1)

     



  • 9.  Re:  High Front End Response Times
    Best Answer

    Broadcom Employee
    Posted Dec 12, 2018 05:14 PM

    Sorry for the late reply.

    Did you enable the trace log and find out which assertion has the max latency?

    We may need to review your policy, it may be better to open a support ticket for that.

     

    Regards,

    Mark



  • 10.  RE: Re:  High Front End Response Times

    Posted Apr 07, 2020 10:33 AM
    Were you able to solve the problem? We are facing the similar issue on our side.


  • 11.  RE: Re:  High Front End Response Times

    Broadcom Employee
    Posted Apr 07, 2020 12:46 PM
    Hello. Have you opened a support case for this issue already? If not, I would recommend doing that, as there are a lot of possibilities for what could cause this for your environment.

    ------------------------------
    Senior Technical Support Engineer
    Broadcom
    ------------------------------



  • 12.  RE: Re:  High Front End Response Times

    Posted Apr 20, 2020 04:28 AM
    Edited by System Jun 16, 2020 02:23 PM
    Hello,

    We solved the issue.

    There was a problem on the load balancer distributing traffic to the nodes.





  • 13.  RE: Re:  High Front End Response Times

    Posted Apr 20, 2020 04:32 AM
    Hello,

    We solved the issue.

    There was a problem on the load balancer distributing traffic to the nodes.

    ------------------------------
    Dev
    Ministry of Customs and Trade
    ------------------------------