ASM Checkpoint connection error: 503

[Legacy User]

Hi,

 

We are seeing an issue where the On Premise Station is not relaying status of our Internal Website to asm.ca.com. The issue started happening last week. We have verified with our Security Engineers and they don't see anything blocked. I am able to ping asm.ca.com from opms and also ping the internal websites. I see the below error message.

 

(-98) No checkpoint available for check type https/IPvANY
(-95) Checkpoint connection error: 503

 

The Opms station shows as Active when I login to asm.ca.com. I also restarted the opms station and that did not resolve the issue. Any help will be greatly appreciated.

 

Thanks,

 

Arnab

[Lynn Williams]

Hi Arnab,

This scenario may require a support case to analyse the opms logs in detail.

In the meantime have you tried command "monit restart all" to see if it will hep to resolve the problem?

If not resolved after that what is the output from the command "monit summary" ?

Troubleshooting - CA App Synthetic Monitor - CA Technologies Documentation 

 

Thanks

 

Lynn

[Legacy User]

Hi Lynn,

 

Thank you for your reply.

 

Unfortunately the restart did not resolve the issue. I will provide you the output of the monit summary.

 

Here is the bottom of the log file.

 

2018-02-06 09:56:28,113 client          INFO     [8967faac-e276-4989-a9e4-e630f6d1e763] cURL OK: (Status: 503) http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30 (3.026381) - Effective URL http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30

2018-02-06 09:56:30,316 client          INFO     [46b390a9-a375-47b5-a716-da46dc441aaa] cURL OK: (Status: 503) http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30 (1.537756) - Effective URL http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30

2018-02-06 09:56:40,381 optunnel        INFO     OnPremise PoP tunnel started with config: /etc/asm/optunnel.yaml

2018-02-06 09:56:40,410 client          INFO     opPoP client is initiating

2018-02-06 09:56:40,410 client          INFO     Using pid file: /opt/asm/var/run/optunnel-client.pid

2018-02-06 09:56:40,410 client          ERROR    PID file exists /opt/asm/var/run/optunnel-client.pid

2018-02-06 09:56:40,410 client          ERROR    Found stale PID file in /opt/asm/var/run/optunnel-client.pid, removing...

2018-02-06 09:56:40,410 client          INFO     Created PID file in /opt/asm/var/run/optunnel-client.pid

2018-02-06 09:56:43,749 client          INFO     Trying to connect at wss://opp.asm.ca.com:443/4a8317d4-21ab-11e6-aff1-782bcb56f074

2018-02-06 09:56:44,310 client          INFO     WebSocket client connected to server (with cid=4a8317d4-21ab-11e6-aff1-782bcb56f074)

2018-02-06 09:56:44,323 client          INFO     opPoP client is up & ready

2018-02-06 10:00:55,830 client          INFO     [99464b02-788d-4385-b4e1-f2ac7c52578a] cURL OK: (Status: 503) http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=rbm&direct=1&age=30 (3.01754) - Effective URL http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=rbm&direct=1&age=30

2018-02-06 10:01:28,444 client          INFO     [839bfcce-0cdc-4aa7-83a3-7e4b6c838f4a] cURL OK: (Status: 503) http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30 (1.431061) - Effective URL http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30

2018-02-06 10:01:29,945 client          INFO     [03522980-a546-43d7-a425-c30e33a09b8c] cURL OK: (Status: 503) http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30 (1.515884) - Effective URL http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30

[Lynn Williams]

Hi Arnab,

As well as the "monit summary" output please also provide the output of "cat /proc/loadavg" and advise the OPMS version being used.

Thanks

Lynn

[Legacy User]

Hi Lynn,

 

Below is the information from the opms station.

 

# cat /opt/asm/opms/version.txt

8.4.0.2

 

# cat /proc/loadavg

0.00 0.03 0.05 1/199 12235

 

 

# monit summary

The Monit daemon 5.9 uptime: 4d 18h 47m

 

Process 'Xvfb'                      Running

Process 'tunnel-client'             Running

System 'dotstollopms101'            Running

Process 'api'                       Not monitored

Process 'httpbroker'                Not monitored

Process 'assetsmanager'             Not monitored

Process 'resultbroker'              Not monitored

Process 'fpm'                       Not monitored

Process 'redis-server'              Not monitored

Process 'rbtm1'                     Not monitored

Process 'rbtm1-Xvfb'                Running

Process 'rbtm2'                     Not monitored

Process 'rbtm2-Xvfb'                Running

Process 'php-fpm'                   Running

Process 'nginx'                     Running

File 'jmeter-jail-proc'             Accessible

Filesystem 'rootfs_warning'         Accessible

Filesystem 'rootfs_critical'        Accessible

Filesystem 'devshmfs_warning'       Accessible

Filesystem 'devshmfs_critical'      Accessible

Process 'fcgiwrap'                  Running

Process 'bm_proxy'                  Running

 

[Lynn Williams]

Hi Arnab,

Thanks for the output.

The fact that "monit summary"  shows that several process are not running is a cause for concern.

To troubleshoot in more detail please login and create a support case on support.ca.com (https://comm.support.ca.com/csupport/CaseManagement/cases/new)

Please upload the log files from these directories to the case and one of our ASM SMEs will review the case logs and contact you.

/var/log/optunnel
/var/log/nginx
/var/log/redis
/var/log/puppet
/var/log/smartpop
Also /var/log/syslog*

 

Thanks

 

Lynn

[Legacy User]

Thanks Lynn, I opened up a case. I see the below error in the syslog file 

 

Does OPMS rely on puppet certificate?

 

Feb 7 21:35:01 XXXXXCRON[4727]: (root) CMD (/usr/bin/diff /etc/resolv.conf /opt/asm/jmeter/jail/etc/resolv.conf || /bin/cp /etc/resolv.conf /opt/asm/jmeter/jail/etc/resolv.conf >/dev/null)
Feb 7 21:35:43 XXXXX puppet-agent[886]: Could not request certificate: getaddrinfo: Name or service not known

[Lynn Williams]

Hi Arnab,

I see you are working with Mat via case 00957989 and the main problem seems to be with the "redis-server" process not running  which would prevent any monitors from running. Looks like Mat will be engaging Engineering to troubleshoot further.

Regards,

Lynn

[Legacy User]

Thanks Lynn, I see the below message in the log file. I will wait for Mat to respond. 

 

[20238] 08 Feb 13:03:44.445 # Server started, Redis version 2.8.17
[20238] 08 Feb 13:03:44.445 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
[20238] 08 Feb 13:03:44.962 # Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix <filename>