Layer7 API Management

 View Only
  • 1.  Latency In One Of the Load Balanced CA API Gateway Server

    Posted Jan 20, 2020 10:18 AM
    Hi All,

    I have 2 CA API Gateway servers (similar in configuration) under 1 load balancer.

    Server 1 is showing latency (>15seconds) in receiving the response from the backend service.
    Latency is happening intermittently ( for example : every 5th or 6th request)
    But, Server 2 is not showing such latency in receiving the response from the same backend service.

    All the servers and service reside on AWS infrastructure.

    Looks like a networking issue, but not sure where is the lag as both servers have same settings and network rules ?

    Also, is it worth checking any gateway configuration which could be the cause ? What setting to check ?

    Please guide.


  • 2.  RE: Latency In One Of the Load Balanced CA API Gateway Server

    Broadcom Employee
    Posted Jan 20, 2020 01:25 PM
    First thought is run a sniffer and see how long it takes, and the handshake. 
    Is it network side or resource (cpu/memory) is the one with the lag the primary database host? Or does the database reside in rds so its not part of either? If it is local to 1 host and the same host is slower at times it may need more resources since its carrying the load of processing and DB. 
    And secondly is there dns involved and dns lookup? 
    If so is it a fixed ip you can add in the system /etc/hosts to rule out any sort of dns lag or modify the route to reference ip VS hostname?
    Thanks.


  • 3.  RE: Latency In One Of the Load Balanced CA API Gateway Server

    Posted Jan 22, 2020 01:05 PM
    Hi Charles,

    I ran the tcpdump and curl command as well on gateway 1.
    curl command gave immediate response, so the server is making calls fine.
    But, tcpdump shows interesting picture.
    for ex: gateway audit says http routing assertion triggered at 10:15:20 but tcpdump shows that server made the call at 10:15:30
    and received at 10:15:30.some milliseconds... but gateway audit says "message processed successfully" at 10:15:35
    so gateway audit details shows "Resp time" = 15 seconds

    Shall I change the audit level to - Fine, Finer,Finest, would it help ?

    Database is external on RDS.

    memory/cpu etc on server is very low.

    Not sure how to check dns lookup time, but since gateway 2 is on cluster with gateway 1 with same configurations, DNS look up issue would have reflected in gateway 2 as well.. but thats not the case. gateway 2 is fine.


  • 4.  RE: Latency In One Of the Load Balanced CA API Gateway Server
    Best Answer

    Broadcom Employee
    Posted Jan 22, 2020 01:19 PM

    First I would look at how complex the policy is and ensure that the route assertion is taking that long (there are variables you can audit post route to see what it thinks the time is). 
    You can also add audits to check after other assertions as here,
    https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-gateway/9-3/policy-assertions/assertion-latency.html

    If this is RDS related I have to wonder after thinking about it if you are having RDS connection issues. 
    You may want scope What size rds are you using? What are the max connections? What is the gateways io.coreconcurrency (And how many gateways) can the rds take all the connections you are loading at it? Is it the default max connections settings?

    Then I would probably go with policy debugger or policy traces OR finer log levels.
    Thats probably how I would try to do it.
    Charlie.




  • 5.  RE: Latency In One Of the Load Balanced CA API Gateway Server

    Posted Jan 27, 2020 02:01 AM
    Edited by Amit Jaiswal Jan 27, 2020 02:26 AM

    Hi,

     

    I did the tcpdump and figured out that the gateway is creating a new tcp connection after a delay of 15 seconds almost.

    Once a connection is open all the future request are not showing any latency.

     

    That is why the issue is intermittent because the new tcp connection open is taking time.

     

    Not sure, how to troubleshoot this further because we have not setup any connection maxconcurrency  values.

    And the gateway 2 (clustered server) works fine without any connection latency.

     




  • 6.  RE: Latency In One Of the Load Balanced CA API Gateway Server

    Posted Jan 27, 2020 02:03 AM

    Hi,

     

    I did the tcpdump and figured out that the gateway is creating a new tcp connection after a delay of 15 seconds almost.

    Once a connection is open all the future request are not showing any latency.

     

    That is why the issue is intermittent because the new tcp connection open is taking time.

     

    Not sure, how to troubleshoot this further because we have not setup any connection maxconcurrency  values.

    And the gateway 2 (clustered server) works fine without any connection latency.




  • 7.  RE: Latency In One Of the Load Balanced CA API Gateway Server

    Posted Jan 27, 2020 02:02 AM
    Edited by Amit Jaiswal Jan 27, 2020 02:26 AM