Increase timeout for web requests

[Luke Way]

This sounds very similar to my situation:

 

https://community.pivotal.io/s/question/0D50e00005QOM2qCAH/our-java-application-has-timed-out-60-seconds-when-running-the-stored-procedure-that-takes-time-to-processplease-tell-me-how-to-solve-this-problem-such-as-lengthening-the-timeout-value-setting

 

I'm running a Spring Boot application on PWS. There are a few web requests that I know take several minutes to run. Rewriting the endpoints to use the claim check approach may be the best solution in the long run, but I'm trying to migrate an application as-is.

 

After ~60 seconds, the web browser receives 504 GATEWAY_TIMEOUT. In application logs I see 499 response code, but the process continues to run (work happens in a separate thread pool), and eventually completes. However, the connection to the browser has long since been closed, of course.

 

Does this give enough information to determine which piece is timing out? If so, how do I configure it? 15 minutes would be fine for my purposes.

 

Thanks!

 

-Luke

[Daniel Mikusa]

It's hard to say exactly. There are a number of places where a timeout could occur. If you're seeing a `[RTR/xx]` log entry which indicates a response time of 499, that means the client upstream from Gorouter hung up before Gorouter could deliver the response. That could be either an upstream load balancer or the client itself. You said this is your web browser and that you're seeing a 504, so that would really point to a 60s timeout on your load balancer. That would be where I would start looking. If you do not have access to the load balancer for your environment, you'd need to reach out to your network or operations team and work with them to debug.

 

Hope that helps!

[Daniel Mikusa]

Oh, sorry. I missed that you were running on PWS. PWS is managed by Pivotal so you shouldn't need to worry about the load balancer config. It should be a full 900s before you get disconnected. I did a quick test with a Python app and sleep, and I was able to wait longer than a minute.

 

Are you by chance connecting to the Internet through a proxy? That could have timeouts. If possible, also try making a request with a client like `curl` from your machine to your app & see if you still see the 504. If you're behind a proxy, try something similar but from another network not behind a proxy, like your home Internet or your mobile network.

[Daniel Mikusa]

>I forgot about this, but I'm using the "Pivotal SSL" marketplace service. Could that have something to do with it?

 

That's possible. Let me look into this for you.

 

In the meantime, if you could repeat your test using a route mapped to something under the cfapps.io domain that would cause traffic to go through our main set of ELBs and will also help to confirm if this is specific to Pivotal SSL service.

 

Thanks!

[Luke Way]

Yes, it's an [RTR/xx] log entry showing 499 status code. And yes in my browser I'm initiating an HTTP PUT that returns 504 status code after about a minute.

 

What is the load balancer for PWS and how do I configure it? I don't believe I ever set anything up for that. I just created an account/org/space, ran cf push, and mapped a route.

[Luke Way]

I work from home, and I'm not behind a proxy as far as I know. The same endpoint works fine deployed to a bare VM hosted elsewhere. No fancy load balancers or routing, just a direct connection, so I don't believe it's my internet.

 

I set up a test endpoint using sleep so I could easily reproduce the issue. I tested via curl with the same results (504 returned to curl, 499 in logs).

 

I forgot about this, but I'm using the "Pivotal SSL" marketplace service. Could that have something to do with it? Same results with either HTTP or HTTPS, but there's only one CNAME for the domain so that probably doesn't make a difference.

[Luke Way]

OK, I tried with the cfapps.io domain and the timeout there is 5 minutes. Looks like the Pivotal SSL service has the one minute timeout. 15 minutes would still work best for my current situation if there's any way to configure that.

[Daniel Mikusa]

OK, definitely looks like it's the SSL Service that's at fault here. I have an issue opened with our Cloud Ops team and they are investigating. I will keep you posted.

[Luke Way]

Awesome, thank you!

[Daniel Mikusa]

@Luke Way​ - OK. I talked to our ops team and we can boost the timeout for your SSL Services. I will need the service instance guid for the SSL service. Not sure it would be good to specify that here, since this is a public forum. Please email [e-mail], reference this thread, include your service instance guid and mention my name. That'll log a ticket and I can pick it up. Let me know if you have any questions.

[Luke Way]

Sent. Thank you for the help!

[Minakshi Kumari]

@Daniel Mikusa - Tanzu Support​ Do we have a community document on "How to boost the timeout for PCF SSL Services"? If so, kindly share the link. Thanks in Advance!

[Daniel Mikusa]

@Minakshi Kumari​ Please email [e-mail], reference this thread, include your Pivotal SSL service instance guid. It's a manual process on our end, but we'll submit the request with our ops team and get the change made. You won't need to do anything beyond that. Thanks