How to invoke custom Java class from GEL script

[Pavan Bugata]

Hi,

I have a use case - a sequence of three shell scripts to be executed on Linux box. I got the java jar deployed under clarity/lib which accepts the parameters. So I need to pass DB password from properties.xml and shell script name/s (as an ArrayList). Is there a way to invoke the Java class with parameters from GEL script? 

[Edited]
My goal is to run the shell scripts in sequence on Linux box. If we cannot invoke Java class from GEL, please let me know if we can run shell scripts from GEL directly.

Thanks,
PB

[Nick Darlington]

Yes you can call java from GEL, and you can either choose to do that directly with your jar - since it's placed in either clarity/lib or clarity/customlib it is normally automatically added to the classpath when the app/bg services start.

What you do with it would not be supported; it's your own code after all and nobody elses, and generally speaking whilst things like this might work in the SaaS environments, calling scripts would be deeply frowned upon and discouraged.

Since you put the jar files in clarity/lib though, I will assume you're on-premises and not SaaS.

I don't have Clarity installed on Linux currently to demo right now, but I can do it from Windows and it's pretty much the same deal in other OS's so long as you use appropriate script names and contents.  It should go without saying that these ought to be non-interactive scripts that cannot stop and prompt for any inputs, or else they may get stuck in limbo and cause the process engine to time out over the long-running (stuck) script, which will then bring down the process engine for everything.

With the disclaimers out of the way, if I have a script like this called external_script.bat (and we assume for now it's placed in the clarity/bin folder although I'd suggest putting it some place else):

@echo off
echo Called by a gel script > logfile.txt
echo Shhh, don't tell anyone the %1 >> logfile.txt
echo Do I talk too much?  Well you said it was ok right? %2 >> logfile.txt​

If I create a GEL script like this and run it:

<gel:script 
  xmlns:core="jelly:core"
  xmlns:gel="jelly:com.niku.union.gel.GELTagLibrary"
>

<!-- example script name and parameters -->
<core:set var="dbPwd" value="secret" />
<core:set var="scriptName" value="external_script.bat" />
<core:set var="verbosity" value="--verbose" />

<!-- make a 1D array of objects needed for invoke(obj, string...) call -->
<core:invokeStatic var="stringClass" className="java.lang.Class" method="forName">
  <core:arg type="java.lang.String" value="java.lang.String"/>
</core:invokeStatic>

<core:invokeStatic var="stringArray" className="java.lang.reflect.Array" method="newInstance">
  <core:arg type="java.lang.Class" value="${stringClass}"/>
  <core:arg type="int" value="${3}"/>
</core:invokeStatic>

<!-- assign the array items -->
<core:invokeStatic className="java.lang.reflect.Array" method="set">
  <core:arg type="java.lang.Object" value="${stringArray}" />
  <core:arg type="int" value="${0}" />
  <core:arg type="java.lang.Object" value="${scriptName}" />
</core:invokeStatic>

<core:invokeStatic className="java.lang.reflect.Array" method="set">
  <core:arg type="java.lang.Object" value="${stringArray}" />
  <core:arg type="int" value="${1}" />
  <core:arg type="java.lang.Object" value="${dbPwd}" />
</core:invokeStatic>

<core:invokeStatic className="java.lang.reflect.Array" method="set">
  <core:arg type="java.lang.Object" value="${stringArray}" />
  <core:arg type="int" value="${2}" />
  <core:arg type="java.lang.Object" value="${verbosity}" />
</core:invokeStatic>

<!-- create builder -->
<core:new className="java.lang.ProcessBuilder" var="builder">
  <core:arg value="${stringArray}" />
</core:new>

<!-- start the script -->
<core:invoke on="${builder}" method="start" var="proc" />

</gel:script>
​

I get a logfile.txt created with the following contents:

Called by a gel script
Shhh, don't tell anyone the secret
Do I talk too much?  Well you said it was ok right? --verbose​

The tricky part of this, if there is to be one, is that ProcessBuilder's constructor takes either a List<String> or String... args list; and constructing those in GEL as you can see takes a disproportionate amount of effort for the results (compared to plain java equivalents).
Reference: https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/ProcessBuilder.html

Other packages and classes can be more simply called, where the <core:arg> value can either be accepted directly from a GEL variable or maybe just coerced by setting the type="..." attribute.

A lot (but not all) of the jelly commons core library tags work as documented, so for more variations such as needing to call static methods or similar, check it out.
Reference: http://commons.apache.org/proper/commons-jelly/tags.html

Make sure that all the files (scripts, jars, etc.) have the appropriate permissions set, so that the OS user running the bg services where the process engine is running will be able to find and execute what it needs.

[Nick Darlington]

Of course, you may be able to 'cut out the middle man' here if your scripts are just going to call your .jar file, as I said by using core:new / core:invoke / core:invokeStatic directly against the .jar.

However depending on what it does, you may not want to, as it will be loaded within the same thread (by default) and process that the bg service is using to execute the gel script, and that might mean using memory or other resources in a way that aren't advisable within a GEL script.  In that case, having it carried out as an external call via the shell script (or even calling java -jar blahblah directly without a script) so that it is in its own process may be the better way to go.

Just something to think about and consider.

[Eugene Greiff]

Take note what Nick is saying.  In my early experiences with pushing GEL to do more complex things, I ran smack into what he is pointing out.

Since then, I have setup an external web helper application on an external server.  On the web helper, I deploy soap (some REST for use in html portlets) web services to perform any heavy lift process (creating complex spreadsheets/access databases, update internal systems (we are OnDemand), populating external data stores…).

With the combination of using this web helper and being able to invoke process on CA PPM, I can kick off process from either an external process or via an event in CA PPM.  Using this approach, I can support request via an html portlet or a process.

For example, A user request for some heavy lift processing say from a html portlet. The request is received via the web helper and the user in informed the process is queued and will be notified when complete. The web helper processes the request let's say for a finance spreadsheet (we call them checkbooks).  The service builds the checkbook and uploads it to our CA PPM (on demand) server.  The web helper request service then kicks off a CA PPM process to pull that spreadsheet from the FTP file location and attach it to the user's project.  The process closes by letting the user know that their request is available.

On a side note, you will run into some issues, seems to be mostly around Factory style methods using the core tags.

V/r,

Gene

[Pavan Bugata]

Thanks @Eugene Greiff for the response. Yes, I don't want to handle complex things using GEL. So password encryption/decryption is being done in the shell script itself and using Java class to invoke the shell script​. However, as I said in my earlier response there is some issue with the BG which I could not figure out why the process was successfully executing the shell script after restarting the BG. Do you have any thoughts on this?

~PB

[Eugene Greiff]

So what is triggering the GEL script to run the three shell scripts?  What method are you using for the encryption/decryption?  What environment variables are you looking for to export to the bash shell?



V/r,
Gene

[Pavan Bugata]

Thanks for the response @Nick Darlington. Sorry for my late reply.
Yes, you were right ​our application is on-prem and not SaaS. Instead of calling shell scripts in sequence - I have consolidated into one single shell script and a process has been created that contains Java Job as Action Item that accepts input string parameter (here shell script name). Everything was working fine if I run shell script manually from Linux box but if the shell script is executed by the process sometimes it couldn't get the environment variables and throw following errors on some nodes
-  line 30: sqlplus: command not found 
some other nodes throw as CLARITY_HOME path as NULL

In the above cases, environment variables are properly set and exported in .bash_profile. These errors are caused by BG and when I restart the BG on that particular node - the process works fine. Couldn't identify the issue what actually BG does and why it was not able to pick up the environment variables sometimes. Is it because of cache or something? It would be great if this issue is resolved as I'm stuck here

Thanks,
PB


​

[Nick Darlington]

I wouldn't expect intermittent behavior in the availability of an environment variable within the background service.  Are you sure the variable was set and exported in the profile (of the OS user that is specifically running the bg service) BEFORE the bg services were started?  If the service was running under a different user account and/or it was set after the service was started, that might explain it.

However, you haven't shared how you are going about getting the environment variable to use it.  Was it exactly like this (aside from the variable name to hold the result), or something else?

 <core:invokeStatic className="java.lang.System" method="getenv" var="clarityHome">
  <core:arg value="CLARITY_HOME"/>
 </core:invokeStatic>

[Pavan Bugata]

Yes @Nick Darlington the variable was set and exported in the profile BEFORE the bg services started (profile last update date was 08/11 and bg.status file (apps/clarity/bin) was on 08/26) and service is running on the same user "clarity" where my shell scripts exist (apps/clarity/lib)

I am not using GEL script to execute the shell scripts. Created a process that kickoffs the Job of Java Type that accepts parameters. Parameter holds the shell script name and passed to the java job.

I'm pretty sure the issue is environment variables are not being picked up in one of the nodes in UAT. Below is the example where I could see the sqlplus path printed in the logs on node 1(success case)

The below is the Null Env variables logs from the other node

Here are the bash_profile env variables

Here is the shell script which is being used:

Is there any way I can set the env variables in the shell if the values are NULL?

Thanks,
PB

[Nick Darlington]

I get that GEL may not be directly invoking things, but GEL has - in my experience - not been inconsistent about the presence of those environment variable values; I can't say the same for the contexts under which other processes it launches may be invoked with.

So if you are finding inconsistencies within the other processes and shell scripts, perhaps GET the values in the GEL script and pass them TO your other processes, if that would give you the consistency you seek.

Otherwise for env variables set and exported in a user profile that is running the bg services, I can't see how that would ever have those kinds of inconsistencies.

Unfortunately once you get outside of Clarity and into the host environment to run other things, I'm not able to offer any suggestion really.  As you're witnessing, that seems to be fairly environment and code specific.  But if you can rework it to read the env vars more reliably in the GEL and store/pass them over to the other external code to use when it executes, I hope that will get you to workaround those environmental issues.

[Eugene Greiff]

No sure this would help but since you are on premise, I have build a set of jobs options to support running cmd, gel, java type jobs from PPM.  You could extend it to support any language or modify the cmdprocess class to put the needed environment information and passing it out.  The other nice thing is it supports fire and forget in order to release the BG thread.  Let me know if you would like the code. You would need to recompile it as it was developed for V13.

V/r,
Gene