Error in API Portal

[Pratyush Singh]

Hi Team,

Please suggest me with the error in API Portal 4.2:

Also i tried to get the docker logs:

-- Logs begin at Thu 2019-09-12 00:04:44 PDT. --
Sep 12 04:38:44 10.XX.X.X dockerd[28468]: time="2019-09-12T04:38:44.672158864-07:00" level=error msg="stream copy error: reading from a closed fifo"
Sep 12 04:38:44 10.XX.X.X dockerd[28468]: time="2019-09-12T04:38:44.672204158-07:00" level=error msg="stream copy error: reading from a closed fifo"
Sep 12 04:38:44 10.XX.X.X dockerd[28468]: time="2019-09-12T04:38:44-07:00" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/05bd7a7076b24bfb90d13e7b69bfc9b2c3965fe429a809c2309dbd93b73b90d8/shim.sock" debug=false module="containerd/tasks" pid=18402
Sep 12 04:38:44 10.XX.X.X[28468]: time="2019-09-12T04:38:44.695379349-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/delete type="*events.ContainerDelete"
Sep 12 04:38:44 10.XX.X.X[28468]: time="2019-09-12T04:38:44.698714356-07:00" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Sep 12 04:38:44 10.XX.X.X[28468]: time="2019-09-12T04:38:44-07:00" level=info msg="shim reaped" id=05bd7a7076b24bfb90d13e7b69bfc9b2c3965fe429a809c2309dbd93b73b90d8 module="containerd/tasks"
Sep 12 04:38:44 10.XX.X.X[28468]: time="2019-09-12T04:38:44.724390811-07:00" level=error msg="stream copy error: reading from a closed fifo"
Sep 12 04:38:44 F10.XX.X.X[28468]: time="2019-09-12T04:38:44.724623657-07:00" level=error msg="stream copy error: reading from a closed fifo"
Sep 12 04:38:44 10.XX.X.X[28468]: time="2019-09-12T04:38:44.733538647-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/delete type="*events.ContainerDelete"
Sep 12 04:38:44 10.XX.X.Xd[28468]: time="2019-09-12T04:38:44.777045247-07:00" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 0ba4b2fc7105bf241eaa6548d9e7f8990cf64f4f03cb2019b08b03b676554352 l2oreu4yveh59kkwakv0t0cbu err:cannot delete entry as the entry in table endpoint_table with network id l2oreu4yveh59kkwakv0t0cbu and key 0ba4b2fc7105bf241eaa6548d9e7f8990cf64f4f03cb2019b08b03b676554352 does not exist"
Sep 12 04:38:53 10.XX.X.X dockerd[28468]: time="2019-09-12T04:38:53.000778747-07:00" level=error msg="410d374f0b48c841cf2e5014a90571ad58c7f38be7c6963501410f027b60b171 cleanup: failed to delete container from containerd: no such container"
Sep 12 04:38:53 10.XX.X.X[28468]: time="2019-09-12T04:38:53.000825533-07:00" level=error msg="fatal task error" error="starting container failed: OCI runtime create failed: container_linux.go:296: starting container process caused \"process_linux.go:279: applying cgroup configuration for process caused \\\"mkdir /sys/fs/cgroup/memory/docker/410d374f0b48c841cf2e5014a90571ad58c7f38be7c6963501410f027b60b171: no space left on device\\\"\": unknown" module=node/agent/taskmanager node.id=kfom3nx57emakucz0ken92qn5 service.id=gbvdvx9pmuz30scsn93us6mae task.id=k2sdtezcajeeqo2nxz8b0y1ql
Sep 12 04:38:53 10.XX.X.X[28468]: time="2019-09-12T04:38:53.067844765-07:00" level=error msg="Failed to deserialize netlink ndmsg: invalid argument"
Sep 12 04:38:53 10.XX.X.X[28468]: time="2019-09-12T04:38:53.072987900-07:00" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Sep 12 04:38:53 10.XX.X.X[28468]: time="2019-09-12T04:38:53.073032225-07:00" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Sep 12 04:38:53 10.XX.X.X[28468]: time="2019-09-12T04:38:53.133146173-07:00" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Sep 12 04:38:53 10.XX.X.X[28468]: time="2019-09-12T04:38:53.238260557-07:00" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Sep 12 04:38:53 10.XX.X.X[28468]: time="2019-09-12T04:38:53.285100491-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
Sep 12 04:38:53 10.XX.X.X[28468]: time="2019-09-12T04:38:53-07:00" level=info msg="shim docker-containerd-shim started" address="/contain


Please help.

Thanks
Prat

[Leandro Dantas]

Hi.

Some services are dependent on each other before portal becomes available.
Use  docker ps to monitor which container is failing. Then use docker service logs -f <service_name>  to follow the log messages on failing service.

your image shows even database is starting.  So use docker service logs -f portal_portaldb to see why postgres is not starting.

You can try to clean up and restart.
To clean up:
docker stack rm portal ( wait for all containers to be removed - check with docker ps)
docker container prune (to remove older , dead containers )
docker network prune (sometimes removing the stack does not remove related networks) 

then execute portal.sh again.

https://docops.ca.com/ca-api-developer-portal-enhanced-experience/4-2/en/install-configure-and-upgrade/troubleshoot-the-installation#TroubleshoottheInstallation-DeploymentFailure

------------------------------
Sr. Consultant Services
HCL Enterprise Studio
------------------------------

[Pratyush Singh]

Hi Leandro,

Thank you so much for your quick response.

One clarification: As per the statement : "You can try to clean up and restart." i just wanted to know that will it clean the complete setup of Portal from server because its a PROD server so i am just clarifying to get the details.

Many Thanks in advance.

Regards
Prat

[Leandro Dantas]

Hi.

Of course a backup is always recommended but these commands do note delete any docker volume (where persistent data is stored).

To backup, if you have restrictions on snapshotting the vm, try to make a copy of the volumes as described here: https://docops.ca.com/ca-api-developer-portal-enhanced-experience/4-3/en/set-up-and-maintenance/maintain-the-api-developer-portal/migrate-portal-to-a-different-machine.

------------------------------
Sr. Consultant Services
HCL Enterprise Studio
------------------------------