Automic Workload Automation

Expand all | Collapse all

SMC file - command to start a service in a specific start mode

  • 1.  SMC file - command to start a service in a specific start mode

    Posted 08-02-2020 12:02 PM
    The Automic Service Manager uses two files to define and start services:
    • SMD file: defines services, e.g., Automation Engine server processes like work processes and communication processes.
    • SMC file: starts services defined in the SMD file (CREATE keyword), usually with delays between each one (WAIT keyword).
    In the SMD file, it is common to define AE work processes with multiple start modes. This makes it possible to, for instance, start the AE in coldstart mode by passing the option -parm"StartMode=Cold" to the started AE server process.

    This morning we had to start up an AE server in Coldstart mode. We normally start the AE server via the Service Manager, and the SMgr automatically starts the AE server processes as soon as it starts. So this was the sequence of events:
    1. Start up the Service Manager.
    2. In the SMgr GUI, manually stop all of the AE server processes that begin to start automatically.
    3. Manually start one AE server process in Coldstart mode.
    4. Manually start the other AE server processes one-by-one, waiting a few seconds between each one.
    Is there a way to define an SMC file that starts the first AE server process using a specific start mode, such as Coldstart? (I'm thinking there might be a STARTMODE option for the CREATE keyword, or something like that.)

    If that is not possible, I will probably solve the problem like this: define an alternate destination in the ucybsmgr.ini file that has the same SMD file, but no SMC file (or a blank SMC file). This way, if we need to start the server in Coldstart mode, we can start the SMgr using this alternate destination. It will start up the Service Manager, but not automatically start any server processes. Then can manually start the first AE server process in Coldstart mode, and start the other server processes normally.

    (We already do something similar for troubleshooting. We have a minimal destination in the SMgr INI file that uses the regular SMD file, but whose SMC file starts only one AE server process of each type.)


  • 2.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-03-2020 12:50 AM
    Edited by Marcin Uracz 08-03-2020 12:53 AM
    And why not just change the ucsrv.ini before you start your ServiceManager?
    There is the StartMode parameter.

    ucsrv.ini

    This would then start your system cold and keep the standard startup sequence intact.  

    If nothing changed in the meantime through the versions (have not tested this with 12.3.2 yet), after the first WP starts it changes this entry in the ucsrv.ini automatically to normal so the cold start is only done once.


    ------------------------------
    Cheers,
    Marcin
    ------------------------------



  • 3.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-03-2020 05:27 AM
    +1


  • 4.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-03-2020 05:51 AM
    Edited by Michael Lowry 08-03-2020 05:51 AM
    Thanks, but I'm asking about a way to specify the start mode via the the SMC file, not the ucsrv.ini file.
    ​​


  • 5.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-03-2020 07:27 AM
    Well then other then what Dejan proposed I am out of ideas.

    Maybe apart from a little bit dirty solution.  Giving the first  WP1 some 10 second wait in the SMC before it starts and then starting the SMGR this way:

    systemctl start [your_smgr_service] && sleep 2 &&  ./UCYBSMCl -c START_PROCESS -s WP1 -sm Coldstart -h localhost -n V12.3

    This way the command  to start the WP1 manually with cold start should be faster then the normal boot up process.

    There is a little bit of gabling involved but hey, let's live a little ;) ;)


    ------------------------------
    Cheers,
    Marcin
    ------------------------------



  • 6.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-03-2020 05:38 AM
    I did NOT test ... but I'm pretty sure this would work ...

    If you really want to "... define an SMC file that starts the first AE server process using a specific start mode, such as Coldstart?...", try this:

    1. Add the following line to your SMD file
      • DEFINE WP1_COLD;*WP_STARTCMD_COLD;*SRV_STARTPATH;START1=(Coldstart,*WP_STARTCMD_COLD);START2=(Systemstop,*WP_STARTCMD_STOP);START3=(Coldstart with Systemstop,*WP_STARTCMD_COLDSTOP)
      • This record is NOT used in the regular SMC file.
    2. cp    uc4.smc    uc4_COLDSTART.smc
    3. replace CREATE WP1 with CREATE WP1_COLD in uc4_COLDSTART.smc




  • 7.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-03-2020 08:42 AM
    Edited by Michael Lowry 08-04-2020 11:16 AM
    Yeah, I thought about that too. But I don't want to maintain a separate SMD file that differs in only one line from the ordinary SMD file. Here's what I ended up doing:
    [Destination AE_server]
    deffile=/opt/ae/SMgr/uc4s.smd
    cmdfile=/opt/ae/SMgr/uc4s.smc

    [Destination AE_server_nostart]
    deffile=/opt/ae/SMgr/uc4s.smd
    cmdfile=/opt/ae/SMgr/uc4s_nostart.smc

    [Destination AE_server_minimal]
    deffile=/opt/ae/SMgr/uc4s.smd
    cmdfile=/opt/ae/SMgr/uc4s_minimal.smc

    The three destinations in the SMgr INI file work as follows:
    • AE_server Start the AE server normally, with tens of AE server processes.
    • AE_server_nostart Start the Service Manager, but do not start any AE server processes.
    • AE_server_minimal Start the AE server in minimal mode, with only one AE server process of each type.

    If we start the SMgr and specify the destination AE_server_nostart, it starts up the Service Manager, but does not automatically start any server processes.
    /opt/ae/SMgr/ucybsmgr -iucybsmgr.ini AE_server_nostart
    Then we can manually start the first AE server process in Coldstart mode, and start the other server processes normally.
    ​​


  • 8.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-03-2020 10:02 AM
    That sounds good!

    "... But I don't want to maintain a separate SMD file that differs in only one line from the ordinary SMD file. ..:"
    There was no "separate SMD file" in my suggestion. Just one additional line in your existing SMD file and (a separate) "SMC file that starts the first AE server process using a specific start mode, such as Coldstart"
    With this suggestion you would:

    • start one (first) AE server process in Coldstart mode
    • start all other process automatically/normally, without any manual interaction


    Anyway, your solution works fine as long as you don't depend on Destination name "AE_server".

    BTW, I thought DESTINATION (ServiceManager) has to match SYSTEM (from AE-INI) and SYSTEM (from Agent-INI). And there's a comment in ucsrv.ini:

    ; system: AE system name

    ; The name can consist of a maximum of 8 characters. Allowed are the upper-case letters A to Z, numbers and "_".

    Is my assumption wrong? Does it work with DESTINATION/SYSTEM longer than 8 chars?





  • 9.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-03-2020 11:03 AM
    Edited by Michael Lowry 08-04-2020 05:49 AM
    In my experience, Service Manager destination names do not have to match AE system names. Perhaps some have gotten this idea because the default destination, if none is specified, is 'uc4s'. The Service Manager can also be used to start anything . Some people use it to start just agents, or a server & agents together.

    We have successfully used destination names that contain uppercase letters, lowercase letters, numbers, dashes (-) and underscores (_). I do not know what the maximum number of characters is, but it is more than 8.



  • 10.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-04-2020 04:49 AM
    Edited by Christian Böck 08-04-2020 04:51 AM

    Hi Michael,

     

    AFAIK, the phrase can be contain a maximum of 16 characters.

     

    rgds

    Christian

     






  • 11.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-04-2020 05:52 AM
    Edited by Michael Lowry 08-04-2020 05:52 AM
    Thanks, @Christian Boeck.

    FWIW, both AE_server_nostart and AE_server_minimal contain 17 characters, and both work fine.

    ​​


  • 12.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-04-2020 06:06 AM

    Strange, currently the awi shows 16

     

     

    In my previous documentation i noticed 18

     

    Well, something between 16 and 18 seems to work ��

     

    rgds

    Christian

     






  • 13.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-04-2020 11:14 AM
    Edited by Michael Lowry 08-04-2020 11:45 AM
    @Christian Boeck: I ran some quick tests and discovered the following:
    • The Service Manager works fine with destination names of up to 31 characters.
    • Destination names of 32 characters or longer cause problems.

    Problems with 32-character destination names

    First, any INI file passed to ucybsmgr using the -i option is not read successfully. Instead, the program attempts to read the INI using the default name ucybsmgr.ini.
    20200804/163754.427 - U00022024 The ServiceM anager was started with INI-file '/opt/uc4/SMgr/ucybsmgr.ini'
    If a valid INI file called ucybsmgr.ini does exist, things get a little bit further. The program writes to the log correctly for instance. However, the destination name also gets corrupted, and the default INI file path gets tacked onto the end. Because the corrupted name is not found in the INI file, the program adds this destination to the file with default SMD and SMC file names.
    [Destination 12345678901234567890123456789012/opt/uc4/SMgr/ucybsmgr.ini]
    deffile=/opt/uc4/SMgr/uc4.smd
    cmdfile=/opt/uc4/SMgr/uc4.smc

    Then, when ucybsmgr forks the ucybsmgr-listener child process, this corrupted destination name is passed to the child process. Here is an example comparing the processes started with 31- and 32-character destination names:

    31

    uc4 23706 1 0 16:20 pts/4 00:00:00 /opt/uc4/SMgr/ucybsmgr -i/opt/uc4/SMgr/ucybsmgr_server_1.ini 1234567890123456789012345678901
    uc4 23710 23706 0 16:20 pts/4 00:00:00 ucybsmgr-listener 1234567890123456789012345678901
    32
    uc4 22287 1 0 16:16 pts/4 00:00:00 /opt/uc4/SMgr/ucybsmgr -i/opt/uc4/SMgr/ucybsmgr_server_1.ini 12345678901234567890123456789012
    uc4 22291 22287 0 16:16 pts/4 00:00:00 ucybsmgr-listener 12345678901234567890123456789012/opt/uc4/SMgr/ucybsmgr.ini
    In this case, the the ucybsmgr-listener child process does open a listening socket, but it does not respond to connection attempts from the SMGr GUI.

    If a valid INI file called ucybsmgr.ini does not exist, this leads to additional problems.
    20200804/163754.427 - U00022028 Unable to open file '/opt/uc4/servicemanager-server/ucybsmgr.ini', error code: '2'
    20200804/163754.427 - No such file or directory
    In this case, the program does not write a log; instead it writes messages to stdout. And when usybsmgr-listener starts, it does not even open a listening socket.

    In both cases (whether a valid INI file called ucybsmgr.ini exists or not), the ucybsmgr process does not start any services.​



  • 14.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-04-2020 11:37 AM
    Woow!
    Thanks a lot for sharing!


  • 15.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-05-2020 04:36 AM
    Hi @Michael A. Lowry,

    yep you are right, but I ran also some tests now and discovered the following:

    it seems, that the usage of the phrase is not the same between smgr commandline and awi admin perspective.

    If you want to control your agents ( start/stop ) via awi admin perspective you can not use a phrase higher than 16 characters, otherwise you got the following error

    Stop works - Start doesn't work

    my conclusion is, if you want to use the awi for stop and start you can not use a phrase higher than 16 characters, otherwise you can.

    rgds
    Christian

    ​​

    ------------------------------
    Thx & rgds
    Christian
    ------------------------------



  • 16.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-05-2020 11:46 AM
    Edited by Michael Lowry 08-05-2020 11:46 AM
    @Christian Boeck: Agreed. We will switch to names that are 16 characters or fewer. I will also modify our AE startup script to perform validity checks since the SMgr is not doing this.

    Obviously:
    • The SMgr and AWI should work consistently and enforced the same limits.
    • The SMgr should perform basic range checks to prevent overflow bugs. ​



  • 17.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-06-2020 02:21 AM
    Edited by Michael Lowry 08-06-2020 05:38 AM
    Here is an excerpt from the AE startup script I wrote. This contains most of the validation & checking steps.
    • Validate destination name (allowed characters, number of characters).
    • Check for existence of INI file.
    • Find destination name in INI file.
    • Find & validate SMD and SMC specifications of destination in INI file.
    • Check for existence of SMD and SMC files.
    • Find & validate SMgr port number in INI file.
    • Check for running AE server service manager processes.
    • Check for TIME_WAIT/CLOSE_WAIT sockets on the SMgr port; wait & retry several times if such sockets are found.
    • Warn if destination name exceeds the maximum 16 characters that the AWI supports.
    If all of these checks pass, the complete script then starts the Service Manager for the AE Server. The complete script also performs a few additional steps specific to our environment, but the steps included in this excerpt are the ones relevant to this conversation.

    It should be mostly self-explanatory. Several variables that must be set in the script:
    • INI_FILE: the ucybsmgr INI file
    • DESTINATION: the destination name in the INI file
    • UC4_SERVER_SMGR_HOME: the path to the service manager used to start the AE server

    # Validate destination name
    re='^[A-Za-z0-9\_\-]+$'
    if [[ ! "${DESTINATION}" =~ $re ]]; then
    echo "ERROR: Destination name contains invalid characters. Valid characters: A-Z, a-z, 0-9, -, _"
    exit 1
    elif [[ ${#DESTINATION} -gt 31 ]]; then
    echo "ERROR: Destination name must be 1-31 characters long (1-16 characters for use in the AWI)."
    exit 1
    fi

    # Check for existence of INI file.
    if [[ ! -s "${INI_FILE}" ]]; then
    echo "ERROR: INI file ${INI_FILE} not found."
    exit 1
    fi

    # Find destination name in INI file.
    DEST_HIT_COUNT=$(cat ${INI_FILE} | grep "\S" | grep -v '^;' | grep "\[Destination .*$DESTINATION\]" | wc -l)
    case "${DEST_HIT_COUNT}" in
    0)
    echo "ERROR: Destination ${DESTINATION} not found in INI file."
    exit 1
    ;;
    1)
    echo "Destination : $DESTINATION"
    ;;
    *)
    echo "ERROR: Destination ${DESTINATION} found more than once in INI file."
    exit 1
    ;;
    esac

    # Get SMD & SMC file paths from INI file
    SMD_FILE=$(cat ${INI_FILE} | grep "\S" | grep -v '^;' | grep -A2 "\[Destination .*$DESTINATION\]" | awk -F= '$1=="deffile" {print $2}')
    echo "SMD file : $SMD_FILE"
    SMC_FILE=$(cat ${INI_FILE} | grep "\S" | grep -v '^;' | grep -A2 "\[Destination .*$DESTINATION\]" | awk -F= '$1=="cmdfile" {print $2}')
    echo "SMC file : $SMC_FILE"

    error="false"
    # Check for SMD file in destination specification.
    if [[ "$SMD_FILE" == "" ]]; then
    echo "ERROR: SMD file not specified in destination."
    error="true"
    fi
    # Check for SMC file in destination specification.
    if [[ "$SMC_FILE" == "" ]]; then
    echo "ERROR: SMC file not specified in destination."
    error="true"
    fi
    if [[ $error == "true" ]]; then
    exit 1
    fi

    # Check for existence of SMD file.
    if [[ ! -s "${SMD_FILE}" ]]; then
    echo "ERROR: Specified SMD file does not exist."
    error="true"
    fi
    # Check for existence of SMC file.
    if [[ ! -s "${SMC_FILE}" ]]; then
    echo "ERROR: Specified SMC file does not exist."
    error="true"
    fi
    if [[ $error == "true" ]]; then
    exit 1
    fi

    # Find SMgr port number in INI file.
    SMgr_Port=$(awk -F= '$1 ~ /port/ {print $2}' ${INI_FILE})
    echo "SMgr port number : $SMgr_Port"

    # Validate SMgr port number
    re='^[0-9]+$'
    if [[ ! "${SMgr_Port}" =~ $re ]] || [[ "${SMgr_Port}" -gt 65535 ]]; then
    echo "ERROR: Invalid SMgr port number found in INI file."
    exit 1
    fi

    # Check for AE Server Service Manager(s) already running.
    SMgr_PID_count=$( ps -ef | grep "${UC4_SERVER_SMGR_HOME}" | awk -v smgrhome="${UC4_SERVER_SMGR_HOME}" '$8 ~ smgrhome && $8 !~ /grep/ {print $2}' | wc -l )
    if [[ "${SMgr_PID_count}" -ne 0 ]]; then
    if [[ "${SMgr_PID_count}" -gt 1 ]]; then
    PLURAL_SUFFIX="s"
    NUMBER_PREFIX="${SMgr_PID_count} "
    fi
    echo
    echo "${NUMBER_PREFIX}AE Server Service Manager${PLURAL_SUFFIX} already running!"
    ps -ef | grep "${UC4_SERVER_SMGR_HOME}" | grep -v grep
    exit 1
    fi

    # Check for CLOSE_WAIT or TIME_WAIT sockets on port used by SMgr
    counter=0
    while [[ $counter -le $max_retries ]]; do
    (( counter++ ))
    netstat -an | grep WAIT | grep -l ${SMgr_Port} >/dev/null 2>&1
    RC=$?
    if [[ $RC -eq 0 ]]; then
    if [[ $counter -ge $max_retries ]]; then
    echo "ERROR: Socket still found on port ${SMgr_Port} after ${max_retries} retries. Giving up."
    exit 1
    else
    if [[ $counter -eq 1 ]]; then
    echo "WARNING: CLOSE_WAIT or TIME_WAIT socket found on port ${SMgr_Port}. Will retry ${max_retries} times at ${retry_delay}-second intervals."
    else
    echo " Socket still found on port ${SMgr_Port}. Waiting ${retry_delay} seconds."
    fi
    sleep ${retry_delay}
    fi
    else
    break
    fi
    done
    echo
    if [[ ${#DESTINATION} -gt 16 ]] && [[ ${#DESTINATION} -lt 31 ]]; then
    echo "WARNING: Destination name is valid, but it will not work in the AWI. To work in the AWI, the destination name must be between 1 and 16 characters long."
    echo
    fi

    I hope this is helpful to someone. Enjoy.



  • 18.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-07-2020 10:26 AM
    Edited by Michael Lowry 08-07-2020 10:27 AM
    There are three pieces of information displayed in the Service column of the SMgr GUI ('Service Manager Dialog') that are not displayed in the output of the equivalent command line program ucybsmcl.

    1. AE Process type & role, e.g., UC4 PWP-Server
    2. Assigned AE process names, e.g., AE_PROD#WP055
    3. Connection counts, e.g., 74 Connections
    Sadly, although the Service Manager is obviously able to provide these pieces of information , the ucybsmcl program is completely incapable of retrieving them. Does anyone know a command line approach to obtaining these pieces of information from the SMgr?

    if I have time, I might run a packet trace of the SMgr GUIucybsmgr communication to see if I can learn anything.



  • 19.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-04-2020 11:20 AM
    Edited by Michael Lowry 08-04-2020 11:20 AM
    I learned that if a destination lacks a cmdfile specification, ucybsmgr will add the default uc4s.cmd to the destination in the INI file. To ensure that no services are started with the AE_server_nostart destination, I created a SMD file for this destination. It contains just a single WAIT 1 statement. I updated the example above.





  • 20.  RE: SMC file - command to start a service in a specific start mode

    Posted 08-05-2020 06:13 AM

    Hello,

    I dont believe that you will be able to acheive the desired by playing only with SMD and SMC file.

    From the top of my head I believe that the better solution would be :

    - Have the standard definition of the SMD and SMC file. Do not enable the auto-start of the processes.

    - Within the standard definition you already have the required parameters

    VAR SRV_STARTPATH;*OWN/../../server/bin
    VAR CP_STARTCMD;*SRV_STARTPATH/ucsrvcp -I*SRV_STARTPATH/ucsrv.ini -svc%port%
    VAR WP_STARTCMD;*SRV_STARTPATH/ucsrvwp -I*SRV_STARTPATH/ucsrv.ini -svc%port%
    VAR JWP_STARTCMD;/usr/bin/java -d64 -Xmx2G -jar ucsrvjp.jar -I*SRV_STARTPATH/ucsrv.ini -svc%port%
    VAR JCP_STARTCMD;/usr/bin/java -d64 -Xmx2G -jar ucsrvjp.jar -I*SRV_STARTPATH/ucsrv.ini -svc%port% -rest
    VAR WP_STARTCMD_COLD;*WP_STARTCMD -parm"StartMode=Cold"
    VAR WP_STARTCMD_STOP;*WP_STARTCMD -parm"SystemStop=Yes"
    VAR WP_STARTCMD_COLDSTOP;*WP_STARTCMD -parm"StartMode=Cold;SystemStop=Yes"
    !!!
    DEFINE UC4WP1;*WP_STARTCMD;*SRV_STARTPATH;START1=(Coldstart,*WP_STARTCMD_COLD);START2=(Systemstop,*WP_STARTCMD_STOP);START3=(Coldstart with Systemstop,*WP_STARTCMD_COLDSTOP)
    DEFINE UC4WP2;*WP_STARTCMD;*SRV_STARTPATH;START1=(Coldstart,*WP_STARTCMD_COLD);START2=(Systemstop,*WP_STARTCMD_STOP);START3=(Coldstart with Systemstop,*WP_STARTCMD_COLDSTOP)
    DEFINE UC4WP3;*WP_STARTCMD;*SRV_STARTPATH;START1=(Coldstart,*WP_STARTCMD_COLD);START2=(Systemstop,*WP_STARTCMD_STOP);START3=(Coldstart with Systemstop,*WP_STARTCMD_COLDSTOP)

    - You can call your variables whatever you like
    - Create a script (sh, ksh, or any other). Or multiple scripts for all the situation.
    - Within the script put the actual startup of the SMGR

    ( nohup ./ucybsmgr PHRASE > foo.out 2>&1 & )

    Then utilize the command line interface from within the script. You can add some sleep inbetween, or even checks 

    ucybsmcl -c START_PROCESS -h hostname:port -n phrase -s UC4WP1
    ucybsmcl -c START_PROCESS -h hostname:port -n phrase -s UC4WP2
    ucybsmcl -c START_PROCESS -h hostname:port -n phrase -s UC4WP3
    ....
    (for normal start)

    ucybsmcl -c START_PROCESS -h hostname:port -n phrase -s UC4WP1 -sm Coldstart
    ucybsmcl -c START_PROCESS -h hostname:port -n phrase -s UC4WP2
    ucybsmcl -c START_PROCESS -h hostname:port -n phrase -s UC4WP3
    (for cold start, as you need only first WP to be with that mode.)

    You can add checks using 

    ucybsmcl -c GET_PROCESS_LIST -h hostname:port -n phrase

    and have some logic of remediation within.


    This script can be afterwards started manually or remotely.

    That way you will not have to maintain several smd/smc files for different occasions, but simply start the required script (or start it with the required param).

    My personal opinion is that the AutoStart of uc4 server processes should be avoided, especially for production systems.

    Best Regards,

    Krum Ganev