Unleash your workload with loops and advanced variable passing

If you chose to read this article, you are probably looking for a smart way to arrange workload definitions in your enterprise and make your workflows more dynamic. In fact, requirements become higher day by day and the more you go deep in the workload automation the most your workflows became complex, high connected and hard to maintain.
​
In this article, we will show you some methodologies you can apply to get rid of headache and make your life easier.

Nowadays, the applications need to be pluggable, extensible and often require modifications or adjustments based on dynamic inputs or specific scenarios. This can lead to changes to already defined jobs and job streams or additional creation of similar definitions with known consequences such as: redundancy, complexity of interaction, and error prone procedures in case of changes.
In Workload Automation, the most “popular” way to deal with the just mentioned problems that is followed by beginners to avoid definition redundancy, consists of using static variables contained in variable tables. This can satisfy some of the easier scenarios, but workload automation can do much more …
In the next sections, we will describe how to leverage features like:

• Variable passing
  • stdlist
  • Job properties
  • Jobprop
• Variable tables
• Variable table jobs

 
Variable passing between jobs – stdlist
Let’s suppose you need to run several jobs and the output of each one should be provided as input to the successor one, just like a Unix pipe.  You can perform this task in several ways, but Workload Automation offers you a standard and easy way to complete it.
In this example, we will show you how to catch the output (joblog/stdilist) of a job from another one within the same job stream.
​
First, we create a job stream containing two jobs:

• JOB1 sets a variable file name (a timestamp for example)

date +%s

• JOB2 reads this output from JOB1 and uses this to run a script that copies a file composed by a conventional name + the timestamp, to another path.

 
This is JOB1, a Unix job that runs a “date +%s” command.

​Below the output of JOB1: 

​This is JOB2, a Unix job that copies a file from a path to a new one using the “passed” timestamp as part of the filename.

As you can see in this example, to catch the output of the previous job we used the following statement:
${job:<JOB_NAME>.stdlist}

​Note: the stdlist variable contains all the joblog output and adds a blank line at the end.

Variable passing between jobs – Job properties
 
Another common scenario is to manage variables that are exposed by the advanced Workload Automation job types, such as database job, restful jobs and so on... In fact, these jobs expose automatically some variables when they are executed (for example, the number of rows retrieved by an sql query, the value of a query, and so on).
In this example, we want to take the json output of a Rest call and pass it through the workload. Using a RestFul job, you can make a call to a service and “save” the properties of the returned json in a variable named “JSON result”.
So, we create a RESTFUL_JS job stream containing two jobs:

• JOB1 runs on an agent connected to the internet and executes a call to a rest service to get the current exchange rates of dollar/euro.  Json result variable (${job:<JOB_NAME>.JSONResult}) of this job contains the exchange rate.
• JOB2 reads the JSONResult property of JOB1 (${job:exchange_rate.JSONResult}) and uses it to run a script that updates all the internal employees’ refunds.

 
This is the graphical view

​This is JOB1:

​In the JSON object result query field you can specify the property you want to manage.
 
This is JOB2:

​Note:  You must check the “variable resolution at runtime” checkbox.

As you can see in this example, to catch the output of the previous job we used the following statement:
${job:<JOB_NAME>.JSONResult}

In the same way, we could retrieve any variable exposed by a job in the same job stream. The job properties exposed by each type of job (database, Datastage, and so on) are listed in the Workload Automation documentation.
 
Variable passing between jobs –  Jobprop
As we have seen in the previous section, we could access a variable exposed by another job with the following syntax:
${job:<JOB_NAME>. <JOB-VARIABLE-NAME>}
This syntax can also be used to retrieve custom variables set and exposed at runtime by a job using a utility provided by the tool: Jobprop.

The Jobprop utility is an installed program present on dynamic agents that allows to set variables and their values in a job and pass the variables to the successive job in the same job stream instance, just like we have seen for job properties.

You can use this command on native or executable job to set variable values you can pass in a successive job in the same job stream. The values are set at run time.
Syntax to set variable: jobprop <VAR-NAME> <value>
Syntax to get variable: ${job:<JOB_NAME>.<VAR-NAME>}
 
Where JOB_NAME is the name of the job present in the same jobstream and VAR-NAME is the key to get the variable value.
Once the variable has been set with jobprop, it can be used by other jobs just like any job property.

This is an example of how to use jobprop between two jobs.

Example
JOB_A
<jsdle:executable interactive="false">
<jsdle:script>#!/bin/sh
. /home/ITAuser/TWA/TWS/tws_env.sh
jobprop VAR1 value1
</jsdle:script>
</jsdle:executable>
JOB_B
<jsdl:application name="executable">
<jsdle:executable interactive="false">
<jsdle:script>
echo VAR1=${job:joba.VAR1}
</jsdle:script>
</jsdle:executable>
 
As you can see, we used an executable job to run an inline script calling the jobprop utility. The JOB_B as we have seen before, takes the output of the JOB_A with the syntax: ${job:JOB_NAME.VAR-NAME}.
 
Variabletable job and Variable Table (a more complex scenario)
Sometimes you need to share variables between jobs in different job steams. To do this, it’s required to use another powerful Workload Automation tool: the Variable Table job. With this job type, present in Workload Automation from version 9.4 on, you can use the Variable Table job type to add or modify a variable in a specified variable table at runtime. The Variable Table job type enables variable passing from one job to another, in the same job stream or in a different job stream.

Let’s see the following example. In our company, we have an application that needs to run on a demand basis but in a different way depending on the day of the week. Thus, for each day of the week there is a job stream called APPLICATION and when the application is required to run the correct version should be triggered.
In this scenario, we can’t neither schedule each application’s job stream nor use a unique version of it picking variable values from different variable tables, because each version is very different in terms of dependencies, job run and so on.

Let’s see how we can achieve this goal.

• First, we define a job stream called “RUN APPLICATION” which uses a variable table called “APPLICATION_DATA_TABLE”:

​The only job present in this job stream is a “Job Stream submission” type job called “SUBMIT_JOBSTREAM”, that allow us to submit the application job stream for the specific day of the week using a variable present in the variable table assigned to the job stream, in this way: APPLICATION_^DAY_OF_THE_WEEK^ 

To implement the dynamic selection of the job stream, we need to figure out a way to dynamically update the value of the variable table.

We can define a job stream called “CALC_DAY” with 2 jobs. The first one calculates the day of the week and exposes the variable with the “jobprop” utility as explained in the previous section.

​The second job is of type “Variable Table” and it is used to automatically update the variable in the “APPLICATION_DATA_TABLE” based on the jobprop variable exposed by the previous job.

As shown in the above picture, in the “variable table” job we can specify the name of an existing variable in a variable table and assign a new value to it. The value chosen in this case is a variable exposed with JobProp by the predecessor job with the syntax already shown in the previous sections.

Finally, we set up dependencies to have the “CALC_DAY” job stream run before the “RUN_APPLICATION” job stream. 

​This is the final design of the 2 job streams:

So, whenever the application is required to run, the CALC_DAY job is released by the user starting the flow, updating the variable table with the day of execution, and finally submitting the correct job stream accordingly to the day of execution.
 
With this overview on the variable passing features offered by Workload Automation, we hope we gave some hints to “get rid of headache”.

If you need any clarification, feel free to leave a message in the message box below.

​Good work with Workload Automation and see you soon!
 
Giacomo & Riccardo
​

​Giacomo del Vecchio
Workload Automation Consultant at HCL Technologies.
Giacomo provides services as Workload Automation SME. He designs and implements solutions to satisfy specific customer's requirements, supports and participates in pre-sales activities (RFx, POC), interacts with customers on a daily basis both remote and on-site (EU, US, India).

Riccardo Belloni
Senior Workload Automation Consultant at HCL Technologies. Riccardo is part of the HCL Workload Automation services team. He is a Workload Automation expert who provides technical support to the customer to manage and create the desired solutions. Develops and provides technical education. His ability to change on various IT aspects was based on +7 years of IT work and consulting. He loves freediving and spearfishing, these passions took him, since 2014, to become a freediving instructor and two years ago a rescue (and bls) instructor.