SA Tomcat Services, Will Not Launch

[Jordan Reich]

Installation of SA and use of SA within the test environment worked fine. Installed SA onto production. Installation process via pdm_configure went fine. Edited the server.properties file. Edited the server.xml file for _SA.

 

Netstat shows that neither the HTTP or HTTPS port is listening. In this case 9000 and 9010. The SSL certification is valid it is also used identically on the BASE tomcat. I have also gotten the ports to fire up once for SA but have been unable to get them to fire up again. 

 

Log files show the following.

 

LOG4J_TOMCAT-SA

 

03 Feb 2018 09:11:23,673 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server The SDM server is running in AA mode
03 Feb 2018 09:11:23,673 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server Updating sa_server table...
03 Feb 2018 09:11:23,673 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server The Server Host Name = WPDHSWEBL35
03 Feb 2018 09:11:23,704 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server Hostname is not existing in DB, inserting new row for the server :WPDHSWEBL35
03 Feb 2018 09:11:23,751 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server Setting all other servers as non-main (i.e. sa_server.is_main=0)
03 Feb 2018 09:11:23,766 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server done
03 Feb 2018 09:11:23,766 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server MT Option is : OFF
03 Feb 2018 09:11:23,766 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server ******System Initialization Complete
03 Feb 2018 09:11:23,782 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server Server mode:1
03 Feb 2018 10:03:25,206 ERROR TCP_port-Read:Slump_nxd com.ca.ServicePlus.slump.TCP_port ReadInt Socketexception (did slump die?)java.net.SocketException: Connection reset
03 Feb 2018 10:03:25,206 ERROR TCP_port-Read:Slump_nxd com.ca.ServicePlus.slump.TCP_port Call to read_msg failed.java.net.SocketException: Connection reset
03 Feb 2018 10:03:25,222 ERROR TCP_port-Read:bpvirtdb_srvr com.ca.ServicePlus.slump.TCP_port ReadInt EOFException (did slump die?)java.io.EOFException
03 Feb 2018 10:03:25,222 ERROR TCP_port-Read:bpvirtdb_srvr com.ca.ServicePlus.slump.TCP_port Call to read_msg failed.java.net.SocketException
03 Feb 2018 10:03:25,222 ERROR TCP_port-Read:domsrvr:sa com.ca.ServicePlus.slump.TCP_port ReadInt EOFException (did slump die?)java.io.EOFException
03 Feb 2018 10:03:25,222 ERROR TCP_port-Read:domsrvr:sa com.ca.ServicePlus.slump.TCP_port Call to read_msg failed.java.net.SocketException
03 Feb 2018 12:15:58,415 INFO localhost-startStop-1 com.controlf1.baseservice.system.FrameworkInitServlet Apache log4j started.
03 Feb 2018 12:15:58,525 ERROR localhost-startStop-1 org.apache.axis.configuration.EngineConfigurationFactoryServlet Unable to find config file. Creating new servlet engine config file: /WEB-INF/server-config.wsdd
03 Feb 2018 12:15:58,915 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server This server is operating as the PRIMARY server...
03 Feb 2018 12:15:58,962 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server Initializing server...
03 Feb 2018 12:16:29,056 ERROR CF1 Server Initialization Thread com.ca.ServicePlus.slump.SLUMP Timed out attempting to connect fast channel to domsrvr:sa
03 Feb 2018 12:16:29,056 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server Initializing Property Manager ...
03 Feb 2018 12:21:18,918 ERROR TCP_port-Read:Slump_nxd com.ca.ServicePlus.slump.TCP_port ReadInt Socketexception (did slump die?)java.net.SocketException: Connection reset
03 Feb 2018 12:21:18,918 ERROR TCP_port-Read:bpvirtdb_srvr com.ca.ServicePlus.slump.TCP_port ReadInt EOFException (did slump die?)java.io.EOFException
03 Feb 2018 12:21:18,918 ERROR TCP_port-Read:Slump_nxd com.ca.ServicePlus.slump.TCP_port Call to read_msg failed.java.net.SocketException: Connection reset
03 Feb 2018 12:21:18,918 ERROR TCP_port-Read:bpvirtdb_srvr com.ca.ServicePlus.slump.TCP_port Call to read_msg failed.java.net.SocketException

 

PDM_TOMCAT_SA

 

INFO: Deploying web application directory C:\Program Files (x86)\CA\Service Desk Manager\bopcfg\www\CATALINA_BASE_SA\webapps\SupportAutomation
03 Feb 2018 12:15:58,415 INFO localhost-startStop-1 com.controlf1.baseservice.system.FrameworkInitServlet Apache log4j started.
03 Feb 2018 12:15:58,525 ERROR localhost-startStop-1 org.apache.axis.configuration.EngineConfigurationFactoryServlet Unable to find config file. Creating new servlet engine config file: /WEB-INF/server-config.wsdd
03 Feb 2018 12:15:58,915 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server This server is operating as the PRIMARY server...
03 Feb 2018 12:15:58,962 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server Initializing server...
Feb 03, 2018 12:15:59 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory C:\Program Files (x86)\CA\Service Desk Manager\bopcfg\www\CATALINA_BASE_SA\webapps\SupportAutomation has finished in 5,719 ms
Feb 03, 2018 12:15:59 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler [http-nio-9000]
Feb 03, 2018 12:15:59 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler [https-jsse-nio-9010]
Feb 03, 2018 12:15:59 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 5872 ms
03 Feb 2018 12:16:29,056 ERROR CF1 Server Initialization Thread com.ca.ServicePlus.slump.SLUMP Timed out attempting to connect fast channel to domsrvr:sa
03 Feb 2018 12:16:29,056 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server Initializing Property Manager ...
03 Feb 2018 12:21:18,918 ERROR TCP_port-Read:Slump_nxd com.ca.ServicePlus.slump.TCP_port ReadInt Socketexception (did slump die?)java.net.SocketException: Connection reset
03 Feb 2018 12:21:18,918 ERROR TCP_port-Read:bpvirtdb_srvr com.ca.ServicePlus.slump.TCP_port ReadInt EOFException (did slump die?)java.io.EOFException
03 Feb 2018 12:21:18,918 ERROR TCP_port-Read:Slump_nxd com.ca.ServicePlus.slump.TCP_port Call to read_msg failed.java.net.SocketException: Connection reset
03 Feb 2018 12:21:18,918 ERROR TCP_port-Read:bpvirtdb_srvr com.ca.ServicePlus.slump.TCP_port Call to read_msg failed.java.net.SocketException

 

Does anyone have any thoughts as to what may be causing this issue?

 

Thanks!

[Jordan Reich]

This has also been recorded into CA case 00954485.

However, I am hoping to potentially discover a solution before CA opens on Monday.

[Raghu Rudraraju]

Jordan,

 

Does this give you any results?

 

 slstat | findstr /i  domsrvr:sa   

 

THe error's indicating that SA Tomcat was not able to connect to the Support Automation domsrvr (domsrvr:sa).

 

_R

[Jordan Reich]

Yes, it returned the following:

 

[Raghu Rudraraju]

thx Jordan.

I can see from the tomcat log that the slump connection that SA Tomcat attempted into the above domsrvr timed out.

Is there any firewall by any chance between these nodes? 

 

_R

[Jordan Reich]

The domsrvr should be on the same machine. Thus local. Either way, there is no firewall present in the CA systems period. It is all internal facing.

[Alexander Perretti]

Jordan,

Nothing obvious is jumping out to me, but can you confirm the architecture? Is Support Automation only installed on the Background server at this point? Can you also confirm if all three of the Support Automation options manager settings are installed?

[Jordan Reich]

Hi Alex,

 

The production environment was setup in the exact manner in which the test environment installation was setup. For which is working. But to answer your first question, no.

 

Support Automation server has been installed onto both the background, and standby.

Proxy server was installed onto one of the application environments for SA.

Messaging server was installed onto the other application environment for SA.

 

Yes, all three options within the options manager are installed. As per the options list.

domsrvr:sa is installed

primary_domsrvr is installed and YES

supportautomation_url is installed and set appropriately

 

I have verified the NX on the background environment as well.

 

# Slump name of domsrvr reserved for Support Automation
@NX_SA_DOMSRVR=domsrvr:sa

@NX_SUPPORTAUTOMATION_URL=https://service... ETC

 

I did notice the following upon inspection of the background environment...

! @NX_SA_SERVER_ON_PRIMARY=Deinstalled Sun Aug 27 08:55:38 2017

 

Could this be causing further issues? This setup is exactly the same on the stand by environment as well.

 

Thanks!

[Alexander Perretti]

Jordan,

 

I do believe that's likely the cause of your issue.

 

 

Set the NX.env on the Background and Standby to:

@NX_SA_SERVER_ON_PRIMARY=Yes

 

 

Let us know if that helps.

[Jordan Reich]

Hi Alex,

 

So developing issues around the primary symptom. 

 

I was able to adjust the options within the options manager in SDM. Simply uninstalled and reinstalled the variable. I was able to check the STANDBY server and see that the value had changed and has remained changed:

 

@NX_SA_PRIMARY_DOMSRVR=Yes

 

But the background server still has the value as DEINSTALLED. I attempted to adjust the NX manually and cycle the service. It reverted back to deinstalled. Also did a manual adjustment. Failed over and did a complete reboot. Still reverted back to the default value. 

 

I do not appear to be able to change this value currently?

[Alexander Perretti]

Jordan,

 

If you haven't tried it already, I'd recommend a full stop of all Service Desk servers, then updating the NX.env files. Then start the Background, and finally all servers after it.

[Jordan Reich]

Can proceed with this attempt. Just not during the working day as this is on our production environment. So I'll have to get back to you on the results of that.

[Jordan Reich]

This resolved the problem, thank you!