We are experiencing an issue with our 11.1.2 software gateway cluster.
Last week, the ssg.service on one of the gateway nodes quit starting automatically on reboot, despite systemctl showing it as active and it having functioned correctly for months.
When manually restarting ssg.service using systemctl start ssg.service, the gateway process did eventually start, but it took approximately 12 minutes for the sspc controller to acquire a port and for the gateway startup to complete.
During this extended startup, the sspc log repeatedly shows messages like: com.l7tech.server.processcontroller.ProcessController: Getting API port from /opt/SecureSpan/Gateway/node/default/var/processControllerPort. This delay is consistent across all nodes in our 4-node cluster, although other nodes typically start within 2-3 minutes and do not have the reboot startup issue. (other nodes will start ssg on reboot.)
Further investigation into /var/log/messages and journalctl revealed SELinux alerts. A recurring log entry indicates SELinux is preventing systemd from unlink access on the /opt/SecureSpan/Gateway/node/default/var/ssg.pid file, with a setroubleshoot message suggesting potential FILE_TYPE values for semanage fcontext. (a very lengthy list of file_types)
We attempted a few SELinux policy adjustments, but without success in resolving the core problem.
Initially, we focused on the reboot startup failure. After modifying the /etc/systemd/system/ssg.service unit file, the service now initiates on boot. However, the significant 8-10 minute delay during startup persists, whether on reboot or when manually starting the service. it did seem to go from about 11 - 12 minutes to 8 - 10 minutes so there was a little improvement.
We are seeking assistance to resolve both the prolonged startup delay and the underlying SELinux issue that may be contributing to it."
Any thoughts or ideas?
I thought I would post here prior to creating a support ticket.
------------------------------
Mark Stewart
Infrastructure Engineer Sr.
USAA
972-523-5059
------------------------------