How to configure properly health checks for static file buildpack?

[Marcin Miklasz]

Hi,

 

I have angular application deployed to PCF using Staticfile buildpack.

During application load test some of GUI/angular containers keep crashing with the below message:

 

2020-02-19 04:08:50.295 App instance exited with guid ebc255.... payload: {"instance"=>"83474caf...", "index"=>6, "cell_id"=>"71..", "reason"=>"CRASHED", "exit_description"=>"Instance became unhealthy: Failed to make TCP connection to port 8080: timed out after 1.00 seconds; process did not exit", "crash_count"=>4, "crash_timestamp"=>1582103333311489529, "version"=>"9957e4ff-53b2-4fab-9c2c-09f9f5dd7fbc"} 2020-02-19 04:08:46.000 Exit status 137

What would be the reason of this error?

What is the recommended configuration for Staticfile buildpack health checks?

Note that I did not specify any health checks for Staticfile application so it is using default one, but why would it call port 8080?

 

[Daniel Mikusa]

It looks like it is trying to make a TCP connection to the application, i.e. using "port" based health checks. The TCP connection is failing, so the health check fails. Does this happen after your app has been running successfully for a while? or does this happen right when you push?

 

As far as port 8080, that is because the app is being told to listen on port 8080 inside the container & health checks are run internally, also inside the container. That is how it uses localhost:8080 to talk to your app.

 

For what it's worth, using HTTP based health checks is generally recommended. They tend to more accurately describe the health of your application. That said, if it can't even pass the port based health check, it won't pass an HTTP health check.

 

https://docs.pivotal.io/platform/application-service/2-8/devguide/deploy-apps/healthchecks.html#setting_health_checks

 

Double check that you have not altered the Staticfile buildpack's Nginx configuration. The default configuration should tell it to listen on port 8080.

 

Hope that helps!

[Daniel Mikusa]

OK, if it is happening after a while I suspect that you are hitting some sort of limit. Perhaps Nginx isn't able to accept connections fast enough and the OS is backing up. If it's CPU bound, you might find that adding more memory helps. As you've pointed out Nginx doesn't need the additional memory, but CPU shares are tied to the memory limit you select, so the higher the memory limit the more CPU shares you will get. More CPU shares means you will get more time on the processor before your app is throttled (note, throttling only happens when there is CPU contention).

 

Rather than increase memory, I would try increasing the number of instances of your app. That should spread the requests out over more app instances which all have their own limits. I don't think it's too different than setting a larger memory limit, you end up with the same number of shares, but it also has the effect of distributing your app across more AZs, so you end up getting better HA for your app as a side effect.

 

Hope that helps!

 

 

[Marcin Miklasz]

Thanks for the reply.

This happens after a while, under heavy load from the performance/load test tool.

PCF metrics indicate CPU spikes on the container however CPU is not configurable directly - only workaround I dig out from the forums was to increase RAM, not sure how correct it was but on the other hand, nginx does not require much RAM so we would be wasting memory on GUI containers. The only nginx config modification I did was for custom location proxies.