DX NetOps

Expand all | Collapse all

Automate the Start/Stop of  Spectro Server thorugh Script on system reboot

  • 1.  Automate the Start/Stop of  Spectro Server thorugh Script on system reboot

    Posted 07-22-2015 08:18 AM

     

     

    Automate the Start/Stop of  Spectro Server thorugh Script on system reboot:

     

    The restart process of spectro server should not be handled by Processd as recommended by CA, as this may lead to ssdb corruption.

     

    Please follow the following steps to automate this proces through perl scripts.

     

    Step 1. Create two batch files to excute  startSS.pl and stop.SS script in the following spectrum installation directory :

    $spectrumroot$/SS/bin

    Step 2. Add the batch file in startup and shutdown group policy.

    Step 3.Add the startSS.bat to to the startup section.

    Step 4. Add the stopSS in the shutdown section.

    Step3. Now it will trigger the spectro server to stop as soon as it detects the shutdown, signal the stopSS.pl script will stop the spectro server first.

     

    Similarly, the spectro server will not start automatically after reboot  as recommended by CA, it will be started by the startSS.pl.

     

     

     

     



  • 2.  Re: Automate the Start/Stop of  Spectro Server thorugh Script on system reboot

    Posted 07-22-2015 08:26 AM

    Could you elaborate on how using processd may lead to SSdb corruption?

     

    Joe



  • 3.  Re: Automate the Start/Stop of  Spectro Server thorugh Script on system reboot

    Posted 07-22-2015 08:53 AM

    We are using the process daemon and haven't had any issues. If your OS is set up correctly then you shouldn't have issues unless your device is switched off at the power button (i.e. immediate shutdown vs controlled shutdown.)

     

    We have enabled the SpectroSERVER's to start on boot (AUTOBOOTSTART;y;) but don't use AUTORESTART (which restarts the SpectroSERVER process if processd detects it dies!).

     

    in $SPECROOT//lib/SDPM/partslist/

     

    SS.idb:

     

    # Processd Install Ticket for SpectroSERVER Daemon.
    PARTNAME;SS;
    APPNAME;SpectroSERVER Daemon;
    WORKPATH;$SPECROOT/SS;
    LOGNAMEPATH;$WORKPATH/VNM.OUT;
    ADMINPRIVS;y;
    #AUTORESTART;N;
    AUTOBOOTSTART;Y;
    STATEBASED;y;
    NUMPROCS;1; // unlimited
    RETRYTIMEOUT;0; // seconds
    TICKETUSER;spectrum;
    RETRYMAX;0;     // retries
    STARTPRIORITY;20;
    SERVERPROCESS;Y;
    #ENV;<var>=<value>;
    ARGV;$SPECROOT/SS/SpectroSERVER<CSEXE>; //
    
    
    

     

    This has worked fine for many years for me. The only issue you get with corruption of DB is when you just power-off the server without giving it time to shut down cleanly.

     

    Using this method starts up the SpectroSERVER process and also kills it when the Linux OS gets the signal to shutdown. You just need to give it time to shutdown.

     

    I've not had anyone from CA recommend that we not use processd to do this.



  • 4.  Re: Automate the Start/Stop of  Spectro Server thorugh Script on system reboot

    Posted 07-22-2015 10:11 AM

    Hi all - you would use the processd start/stop - as this does not depend on Service grant timers at OS-level. Most significant detail is, that processd waits truly until SpectroSERVER did finish with the stop down - and then correctly stop down the other services like location-server and mysqld. BUT - consider - use the "./lib/SPDM/processd.pl stop" command - and not (i.e. in case of Windows) - to stop the Windows-Service -- which is most likely affected by a "application/service stopping timer" .. and once skipped it will kill the application. This could cause a SpectroSERVER-db corruption. Cheers, Joerg



  • 5.  Re: Automate the Start/Stop of  Spectro Server thorugh Script on system reboot

    Posted 07-22-2015 10:39 AM

    This is true, I'm not sure about Windows, I haven't installed Spectrum on a Windows host for over 7 or 8 years and always suggest Linux due to it's easier integration / automation capability.

     

    You can resolve the issue you mention by setting the STARTPRIORITY.

     

    Give the processes which rely on others a higher priority: e.g. SpectroServer should start after LOCSERV and MYSQL so would be a higher STARTPRIORITY than these two.

    Also Archivemanager would be even higher to start after SpectroServer is up.

     

    Shutdown makes sure all 30's are shut before 20's and then 10's etc.

     

    To be fair, if people don't shut down Spectrum properly then you should expect corruption. This is part of why you have a shutdown procedure which people should follow. If someone shuts down e.g.: MySQL without stopping the SpectroSERVER process, that is looking for trouble

     

    I've found processd a very useful tool and use it to often start and stop other non-spectrum processes and it's always just worked for me.



  • 6.  Re: Automate the Start/Stop of  Spectro Server thorugh Script on system reboot

    Posted 07-22-2015 10:44 AM

    We have had many reports from Spectrum users stating the server administrators shutdown the Windows server to apply a patch but did not gracefully shutdown the SpectroSERVER first which resulted in SpectroSERVER database corruption. The reason this happens is the system will only wait 20 second by default for a service to shutdown. The system tries to shutdown processd. processd is trying to shutdown the Spectrum processes including the SpectroSERVER. The SpectroSERVER takes longer than 20 seconds to shutdown so the system kills it.

     

    To resolve this on Windows, please reference the "SpectroSERVER Shutdown in a Windows Environment" section in the "Distributed SpectroSERVER Administrator Guide" for more guidance concerning modifying the "WaitToKillServiceTimeout" registry setting.

     

    Although rare, this has also occurred on Linux and Solaris. Out of the box processd waits 20 seconds to shutdown the Spectrum processes. To resolve this on Linux and Solaris, please reference the "SpectroSERVER Shutdown in a Solaris Environment" section in the "Distributed SpectroSERVER Administrator Guide" for more guidance concerning modifying the "PROCESSD_SHUTDOWN_TIMEOUT" environment variable.

     

    Also reference the "Tuesday Tip: SpectroSERVER shutdown best practices"  https://communities.ca.com/message/98574287#98574287

     

    Joe



  • 7.  Re: Automate the Start/Stop of  Spectro Server thorugh Script on system reboot

    Posted 07-23-2015 12:16 AM

    Thank you all for sharing all your experiences with spectro server and processd. I had just posted the simple steps to use the perl scripts to start and stop the spectro server. actually We faced the similar kind of issue, which Joseph has discussed the window server patching,  by which ssdb was corrupted multiple time, so we chose this way of script..we also tried by changing the "AUTOBOOT parameter to "y" in to restart the spectro server on boot.which dint work in my case.This may be due to the shutdown process, which dint wait for the SS to stop completely.

    In a Windows environment, processd waits until all sub-processes have shut down before it shuts down completely. However, Windows has a registry setting called WaitToKillServiceTimeout, which specifies the length of time (in milliseconds) that Windows will wait for all services to stop after a Windows shutdown is initiated. If a service (such as processd) is still running after the amount of time specified by WaitToKillServiceTimeout, Windows will terminate this service. The default value is only 20 seconds, which may not give the SpectroSERVER enough time to shut down completely when the system is shut down.

    To increase this time period, open the Registry Editor and locate HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Control. Click the Control key, and in the right pane, double-click the WaitToKillServiceTimeout value. In the window, change the value to any value up to 600,000 milliseconds (10 minutes). Click OK

    we are using spectrum scripts since long and never faced issue since then.

    cheers keep posting!!

    Ajay.