DX Application Performance Management

Expand all | Collapse all

ASM Checkpoint connection error: 503

Jump to Best Answer
  • 1.  ASM Checkpoint connection error: 503

    Posted 02-05-2018 10:03 PM

    Hi,

     

    We are seeing an issue where the On Premise Station is not relaying status of our Internal Website to asm.ca.com. The issue started happening last week. We have verified with our Security Engineers and they don't see anything blocked. I am able to ping asm.ca.com from opms and also ping the internal websites. I see the below error message.

     

    (-98) No checkpoint available for check type https/IPvANY
    (-95) Checkpoint connection error: 503

     

    The Opms station shows as Active when I login to asm.ca.com. I also restarted the opms station and that did not resolve the issue. Any help will be greatly appreciated.

     

    Thanks,

     

    Arnab



  • 2.  Re: ASM Checkpoint connection error: 503
    Best Answer

    Posted 02-06-2018 12:41 AM

    Hi Arnab,

    This scenario may require a support case to analyse the opms logs in detail.

    In the meantime have you tried command "monit restart all" to see if it will hep to resolve the problem?

    If not resolved after that what is the output from the command "monit summary" ?

    Troubleshooting - CA App Synthetic Monitor - CA Technologies Documentation 

     

    Thanks

     

    Lynn



  • 3.  Re: ASM Checkpoint connection error: 503

    Posted 02-06-2018 09:56 PM

    Hi Lynn,

     

    Thank you for your reply.

     

    Unfortunately the restart did not resolve the issue. I will provide you the output of the monit summary.

     

    Here is the bottom of the log file.

     

    2018-02-06 09:56:28,113 client          INFO     [8967faac-e276-4989-a9e4-e630f6d1e763] cURL OK: (Status: 503) http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30 (3.026381) - Effective URL http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30

    2018-02-06 09:56:30,316 client          INFO     [46b390a9-a375-47b5-a716-da46dc441aaa] cURL OK: (Status: 503) http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30 (1.537756) - Effective URL http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30

    2018-02-06 09:56:40,381 optunnel        INFO     OnPremise PoP tunnel started with config: /etc/asm/optunnel.yaml

    2018-02-06 09:56:40,410 client          INFO     opPoP client is initiating

    2018-02-06 09:56:40,410 client          INFO     Using pid file: /opt/asm/var/run/optunnel-client.pid

    2018-02-06 09:56:40,410 client          ERROR    PID file exists /opt/asm/var/run/optunnel-client.pid

    2018-02-06 09:56:40,410 client          ERROR    Found stale PID file in /opt/asm/var/run/optunnel-client.pid, removing...

    2018-02-06 09:56:40,410 client          INFO     Created PID file in /opt/asm/var/run/optunnel-client.pid

    2018-02-06 09:56:43,749 client          INFO     Trying to connect at wss://opp.asm.ca.com:443/4a8317d4-21ab-11e6-aff1-782bcb56f074

    2018-02-06 09:56:44,310 client          INFO     WebSocket client connected to server (with cid=4a8317d4-21ab-11e6-aff1-782bcb56f074)

    2018-02-06 09:56:44,323 client          INFO     opPoP client is up & ready

    2018-02-06 10:00:55,830 client          INFO     [99464b02-788d-4385-b4e1-f2ac7c52578a] cURL OK: (Status: 503) http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=rbm&direct=1&age=30 (3.01754) - Effective URL http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=rbm&direct=1&age=30

    2018-02-06 10:01:28,444 client          INFO     [839bfcce-0cdc-4aa7-83a3-7e4b6c838f4a] cURL OK: (Status: 503) http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30 (1.431061) - Effective URL http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30

    2018-02-06 10:01:29,945 client          INFO     [03522980-a546-43d7-a425-c30e33a09b8c] cURL OK: (Status: 503) http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30 (1.515884) - Effective URL http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30



  • 4.  Re: ASM Checkpoint connection error: 503

    Posted 02-06-2018 10:09 PM

    Hi Arnab,

    As well as the "monit summary" output please also provide the output of "cat /proc/loadavg" and advise the OPMS version being used.

    Thanks

    Lynn



  • 5.  Re: ASM Checkpoint connection error: 503

    Posted 02-07-2018 09:28 AM

    Hi Lynn,

     

    Below is the information from the opms station.

     

    # cat /opt/asm/opms/version.txt

    8.4.0.2

     

    # cat /proc/loadavg

    0.00 0.03 0.05 1/199 12235

     

     

    # monit summary

    The Monit daemon 5.9 uptime: 4d 18h 47m

     

    Process 'Xvfb'                      Running

    Process 'tunnel-client'             Running

    System 'dotstollopms101'            Running

    Process 'api'                       Not monitored

    Process 'httpbroker'                Not monitored

    Process 'assetsmanager'             Not monitored

    Process 'resultbroker'              Not monitored

    Process 'fpm'                       Not monitored

    Process 'redis-server'              Not monitored

    Process 'rbtm1'                     Not monitored

    Process 'rbtm1-Xvfb'                Running

    Process 'rbtm2'                     Not monitored

    Process 'rbtm2-Xvfb'                Running

    Process 'php-fpm'                   Running

    Process 'nginx'                     Running

    File 'jmeter-jail-proc'             Accessible

    Filesystem 'rootfs_warning'         Accessible

    Filesystem 'rootfs_critical'        Accessible

    Filesystem 'devshmfs_warning'       Accessible

    Filesystem 'devshmfs_critical'      Accessible

    Process 'fcgiwrap'                  Running

    Process 'bm_proxy'                  Running

     



  • 6.  Re: ASM Checkpoint connection error: 503

    Posted 02-07-2018 05:34 PM

    Hi Arnab,

    Thanks for the output.

    The fact that "monit summary"  shows that several process are not running is a cause for concern.

    To troubleshoot in more detail please login and create a support case on support.ca.com (https://comm.support.ca.com/csupport/CaseManagement/cases/new)

    Please upload the log files from these directories to the case and one of our ASM SMEs will review the case logs and contact you.

    /var/log/optunnel
    /var/log/nginx
    /var/log/redis
    /var/log/puppet
    /var/log/smartpop
    Also /var/log/syslog*

     

    Thanks

     

    Lynn



  • 7.  Re: ASM Checkpoint connection error: 503

    Posted 02-07-2018 10:04 PM

    Thanks Lynn, I opened up a case. I see the below error in the syslog file 

     

    Does OPMS rely on puppet certificate?

     

    Feb 7 21:35:01 XXXXXCRON[4727]: (root) CMD (/usr/bin/diff /etc/resolv.conf /opt/asm/jmeter/jail/etc/resolv.conf || /bin/cp /etc/resolv.conf /opt/asm/jmeter/jail/etc/resolv.conf >/dev/null)
    Feb 7 21:35:43 XXXXX puppet-agent[886]: Could not request certificate: getaddrinfo: Name or service not known



  • 8.  Re: ASM Checkpoint connection error: 503

    Posted 02-08-2018 04:15 PM

    Hi Arnab,

    I see you are working with Mat via case 00957989 and the main problem seems to be with the "redis-server" process not running  which would prevent any monitors from running. Looks like Mat will be engaging Engineering to troubleshoot further.

    Regards,

    Lynn



  • 9.  Re: ASM Checkpoint connection error: 503

    Posted 02-08-2018 04:36 PM

    Thanks Lynn, I see the below message in the log file. I will wait for Mat to respond. 

     

    [20238] 08 Feb 13:03:44.445 # Server started, Redis version 2.8.17
    [20238] 08 Feb 13:03:44.445 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
    [20238] 08 Feb 13:03:44.962 # Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix <filename>