CA Service Management

Expand all | Collapse all

SA Tomcat Services, Will Not Launch

Jump to Best Answer
  • 1.  SA Tomcat Services, Will Not Launch

    Posted 02-03-2018 04:03 PM

    Installation of SA and use of SA within the test environment worked fine. Installed SA onto production. Installation process via pdm_configure went fine. Edited the server.properties file. Edited the server.xml file for _SA.

     

    Netstat shows that neither the HTTP or HTTPS port is listening. In this case 9000 and 9010. The SSL certification is valid it is also used identically on the BASE tomcat. I have also gotten the ports to fire up once for SA but have been unable to get them to fire up again. 

     

    Log files show the following.

     

    LOG4J_TOMCAT-SA

     

    03 Feb 2018 09:11:23,673 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server The SDM server is running in AA mode
    03 Feb 2018 09:11:23,673 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server Updating sa_server table...
    03 Feb 2018 09:11:23,673 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server The Server Host Name = WPDHSWEBL35
    03 Feb 2018 09:11:23,704 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server Hostname is not existing in DB, inserting new row for the server :WPDHSWEBL35
    03 Feb 2018 09:11:23,751 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server Setting all other servers as non-main (i.e. sa_server.is_main=0)
    03 Feb 2018 09:11:23,766 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server done
    03 Feb 2018 09:11:23,766 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server MT Option is : OFF
    03 Feb 2018 09:11:23,766 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server ******System Initialization Complete
    03 Feb 2018 09:11:23,782 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server Server mode:1
    03 Feb 2018 10:03:25,206 ERROR TCP_port-Read:Slump_nxd com.ca.ServicePlus.slump.TCP_port ReadInt Socketexception (did slump die?)java.net.SocketException: Connection reset
    03 Feb 2018 10:03:25,206 ERROR TCP_port-Read:Slump_nxd com.ca.ServicePlus.slump.TCP_port Call to read_msg failed.java.net.SocketException: Connection reset
    03 Feb 2018 10:03:25,222 ERROR TCP_port-Read:bpvirtdb_srvr com.ca.ServicePlus.slump.TCP_port ReadInt EOFException (did slump die?)java.io.EOFException
    03 Feb 2018 10:03:25,222 ERROR TCP_port-Read:bpvirtdb_srvr com.ca.ServicePlus.slump.TCP_port Call to read_msg failed.java.net.SocketException
    03 Feb 2018 10:03:25,222 ERROR TCP_port-Read:domsrvr:sa com.ca.ServicePlus.slump.TCP_port ReadInt EOFException (did slump die?)java.io.EOFException
    03 Feb 2018 10:03:25,222 ERROR TCP_port-Read:domsrvr:sa com.ca.ServicePlus.slump.TCP_port Call to read_msg failed.java.net.SocketException
    03 Feb 2018 12:15:58,415 INFO localhost-startStop-1 com.controlf1.baseservice.system.FrameworkInitServlet Apache log4j started.
    03 Feb 2018 12:15:58,525 ERROR localhost-startStop-1 org.apache.axis.configuration.EngineConfigurationFactoryServlet Unable to find config file. Creating new servlet engine config file: /WEB-INF/server-config.wsdd
    03 Feb 2018 12:15:58,915 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server This server is operating as the PRIMARY server...
    03 Feb 2018 12:15:58,962 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server Initializing server...
    03 Feb 2018 12:16:29,056 ERROR CF1 Server Initialization Thread com.ca.ServicePlus.slump.SLUMP Timed out attempting to connect fast channel to domsrvr:sa
    03 Feb 2018 12:16:29,056 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server Initializing Property Manager ...
    03 Feb 2018 12:21:18,918 ERROR TCP_port-Read:Slump_nxd com.ca.ServicePlus.slump.TCP_port ReadInt Socketexception (did slump die?)java.net.SocketException: Connection reset
    03 Feb 2018 12:21:18,918 ERROR TCP_port-Read:bpvirtdb_srvr com.ca.ServicePlus.slump.TCP_port ReadInt EOFException (did slump die?)java.io.EOFException
    03 Feb 2018 12:21:18,918 ERROR TCP_port-Read:Slump_nxd com.ca.ServicePlus.slump.TCP_port Call to read_msg failed.java.net.SocketException: Connection reset
    03 Feb 2018 12:21:18,918 ERROR TCP_port-Read:bpvirtdb_srvr com.ca.ServicePlus.slump.TCP_port Call to read_msg failed.java.net.SocketException

     

    PDM_TOMCAT_SA

     

    INFO: Deploying web application directory C:\Program Files (x86)\CA\Service Desk Manager\bopcfg\www\CATALINA_BASE_SA\webapps\SupportAutomation
    03 Feb 2018 12:15:58,415 INFO localhost-startStop-1 com.controlf1.baseservice.system.FrameworkInitServlet Apache log4j started.
    03 Feb 2018 12:15:58,525 ERROR localhost-startStop-1 org.apache.axis.configuration.EngineConfigurationFactoryServlet Unable to find config file. Creating new servlet engine config file: /WEB-INF/server-config.wsdd
    03 Feb 2018 12:15:58,915 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server This server is operating as the PRIMARY server...
    03 Feb 2018 12:15:58,962 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server Initializing server...
    Feb 03, 2018 12:15:59 PM org.apache.catalina.startup.HostConfig deployDirectory
    INFO: Deployment of web application directory C:\Program Files (x86)\CA\Service Desk Manager\bopcfg\www\CATALINA_BASE_SA\webapps\SupportAutomation has finished in 5,719 ms
    Feb 03, 2018 12:15:59 PM org.apache.coyote.AbstractProtocol start
    INFO: Starting ProtocolHandler [http-nio-9000]
    Feb 03, 2018 12:15:59 PM org.apache.coyote.AbstractProtocol start
    INFO: Starting ProtocolHandler [https-jsse-nio-9010]
    Feb 03, 2018 12:15:59 PM org.apache.catalina.startup.Catalina start
    INFO: Server startup in 5872 ms
    03 Feb 2018 12:16:29,056 ERROR CF1 Server Initialization Thread com.ca.ServicePlus.slump.SLUMP Timed out attempting to connect fast channel to domsrvr:sa
    03 Feb 2018 12:16:29,056 INFO CF1 Server Initialization Thread com.controlf1.baseservice.system.Server Initializing Property Manager ...
    03 Feb 2018 12:21:18,918 ERROR TCP_port-Read:Slump_nxd com.ca.ServicePlus.slump.TCP_port ReadInt Socketexception (did slump die?)java.net.SocketException: Connection reset
    03 Feb 2018 12:21:18,918 ERROR TCP_port-Read:bpvirtdb_srvr com.ca.ServicePlus.slump.TCP_port ReadInt EOFException (did slump die?)java.io.EOFException
    03 Feb 2018 12:21:18,918 ERROR TCP_port-Read:Slump_nxd com.ca.ServicePlus.slump.TCP_port Call to read_msg failed.java.net.SocketException: Connection reset
    03 Feb 2018 12:21:18,918 ERROR TCP_port-Read:bpvirtdb_srvr com.ca.ServicePlus.slump.TCP_port Call to read_msg failed.java.net.SocketException

     

    Does anyone have any thoughts as to what may be causing this issue?

     

    Thanks!



  • 2.  Re: SA Tomcat Services, Will Not Launch

    Posted 02-03-2018 04:04 PM

    This has also been recorded into CA case 00954485.

    However, I am hoping to potentially discover a solution before CA opens on Monday.



  • 3.  Re: SA Tomcat Services, Will Not Launch

    Broadcom Employee
    Posted 02-03-2018 04:12 PM

    Jordan,

     

    Does this give you any results?

     

     slstat | findstr /i  domsrvr:sa   

     

    THe error's indicating that SA Tomcat was not able to connect to the Support Automation domsrvr (domsrvr:sa).

     

    _R



  • 4.  Re: SA Tomcat Services, Will Not Launch

    Posted 02-03-2018 05:09 PM

    Yes, it returned the following:

     



  • 5.  Re: SA Tomcat Services, Will Not Launch

    Broadcom Employee
    Posted 02-03-2018 06:51 PM

    thx Jordan.


    I can see from the tomcat log that the slump connection that SA Tomcat attempted into the above domsrvr timed out.


    Is there any firewall by any chance between these nodes? 

     

    _R



  • 6.  Re: SA Tomcat Services, Will Not Launch

    Posted 02-03-2018 08:09 PM

    The domsrvr should be on the same machine. Thus local. Either way, there is no firewall present in the CA systems period. It is all internal facing.



  • 7.  Re: SA Tomcat Services, Will Not Launch

    Broadcom Employee
    Posted 02-05-2018 11:24 AM

    Jordan,

    Nothing obvious is jumping out to me, but can you confirm the architecture? Is Support Automation only installed on the Background server at this point? Can you also confirm if all three of the Support Automation options manager settings are installed?



  • 8.  Re: SA Tomcat Services, Will Not Launch

    Posted 02-05-2018 11:31 AM

    Hi Alex,

     

    The production environment was setup in the exact manner in which the test environment installation was setup. For which is working. But to answer your first question, no.

     

    Support Automation server has been installed onto both the background, and standby.

    Proxy server was installed onto one of the application environments for SA.

    Messaging server was installed onto the other application environment for SA.

     

    Yes, all three options within the options manager are installed. As per the options list.

    domsrvr:sa is installed

    primary_domsrvr is installed and YES

    supportautomation_url is installed and set appropriately

     

    I have verified the NX on the background environment as well.

     

    # Slump name of domsrvr reserved for Support Automation
    @NX_SA_DOMSRVR=domsrvr:sa

    @NX_SUPPORTAUTOMATION_URL=https://service... ETC

     

    I did notice the following upon inspection of the background environment...

    ! @NX_SA_SERVER_ON_PRIMARY=Deinstalled Sun Aug 27 08:55:38 2017

     

    Could this be causing further issues? This setup is exactly the same on the stand by environment as well.

     

    Thanks!



  • 9.  Re: SA Tomcat Services, Will Not Launch
    Best Answer

    Broadcom Employee
    Posted 02-05-2018 11:38 AM

    Jordan,

     

    I do believe that's likely the cause of your issue.

     

     

    Set the NX.env on the Background and Standby to:

    @NX_SA_SERVER_ON_PRIMARY=Yes

     

     

    Let us know if that helps.



  • 10.  Re: SA Tomcat Services, Will Not Launch

    Posted 02-05-2018 12:47 PM

    Hi Alex,

     

    So developing issues around the primary symptom. 

     

    I was able to adjust the options within the options manager in SDM. Simply uninstalled and reinstalled the variable. I was able to check the STANDBY server and see that the value had changed and has remained changed:

     

    @NX_SA_PRIMARY_DOMSRVR=Yes

     

    But the background server still has the value as DEINSTALLED. I attempted to adjust the NX manually and cycle the service. It reverted back to deinstalled. Also did a manual adjustment. Failed over and did a complete reboot. Still reverted back to the default value. 

     

    I do not appear to be able to change this value currently?



  • 11.  Re: SA Tomcat Services, Will Not Launch

    Broadcom Employee
    Posted 02-05-2018 12:58 PM

    Jordan,

     

    If you haven't tried it already, I'd recommend a full stop of all Service Desk servers, then updating the NX.env files. Then start the Background, and finally all servers after it.



  • 12.  Re: SA Tomcat Services, Will Not Launch

    Posted 02-05-2018 02:10 PM

    Can proceed with this attempt. Just not during the working day as this is on our production environment. So I'll have to get back to you on the results of that.



  • 13.  Re: SA Tomcat Services, Will Not Launch

    Posted 02-06-2018 03:42 PM

    This resolved the problem, thank you!