WORKLOAD AUTOMATION COMMUNITY
  • Home
  • Blogs
  • Forum
  • Resources
  • Events
  • About
  • Contact
  • What's new

Unleash your workload with loops and advanced variable passing

4/3/2019

1 Comment

 
Picture
If you chose to read this article, you are probably looking for a smart way to arrange workload definitions in your enterprise and make your workflows more dynamic. In fact, requirements become higher day by day and the more you go deep in the workload automation the most your workflows became complex, high connected and hard to maintain.
​
In this article, we will show you some methodologies you can apply to get rid of headache and make your life easier.
Nowadays, the applications need to be pluggable, extensible and often require modifications or adjustments based on dynamic inputs or specific scenarios. This can lead to changes to already defined jobs and job streams or additional creation of similar definitions with known consequences such as: redundancy, complexity of interaction, and error prone procedures in case of changes.
In Workload Automation, the most “popular” way to deal with the just mentioned problems that is followed by beginners to avoid definition redundancy, consists of using static variables contained in variable tables. This can satisfy some of the easier scenarios, but workload automation can do much more …
In the next sections, we will describe how to leverage features like:
  • Variable passing
    • stdlist
    • Job properties
    • Jobprop
  • Variable tables
  • Variable table jobs
 
Variable passing between jobs – stdlist
Let’s suppose you need to run several jobs and the output of each one should be provided as input to the successor one, just like a Unix pipe.  You can perform this task in several ways, but Workload Automation offers you a standard and easy way to complete it.
In this example, we will show you how to catch the output (joblog/stdilist) of a job from another one within the same job stream.
​
First, we create a job stream containing two jobs:
  • JOB1 sets a variable file name (a timestamp for example)
date +%s
  • JOB2 reads this output from JOB1 and uses this to run a script that copies a file composed by a conventional name + the timestamp, to another path.
 
This is JOB1, a Unix job that runs a “date +%s” command.
Picture
Picture
​Below the output of JOB1: 
Picture
​This is JOB2, a Unix job that copies a file from a path to a new one using the “passed” timestamp as part of the filename.
Picture
As you can see in this example, to catch the output of the previous job we used the following statement:
${job:<JOB_NAME>.stdlist}

​Note: the stdlist variable contains all the joblog output and adds a blank line at the end.
Picture
Variable passing between jobs – Job properties
 
Another common scenario is to manage variables that are exposed by the advanced Workload Automation job types, such as database job, restful jobs and so on... In fact, these jobs expose automatically some variables when they are executed (for example, the number of rows retrieved by an sql query, the value of a query, and so on).
In this example, we want to take the json output of a Rest call and pass it through the workload. Using a RestFul job, you can make a call to a service and “save” the properties of the returned json in a variable named “JSON result”.
So, we create a RESTFUL_JS job stream containing two jobs:
  • JOB1 runs on an agent connected to the internet and executes a call to a rest service to get the current exchange rates of dollar/euro.  Json result variable (${job:<JOB_NAME>.JSONResult}) of this job contains the exchange rate.
  • JOB2 reads the JSONResult property of JOB1 (${job:exchange_rate.JSONResult}) and uses it to run a script that updates all the internal employees’ refunds.
 
This is the graphical view
Picture
​This is JOB1:
Picture
Picture
Picture
​In the JSON object result query field you can specify the property you want to manage.
 
This is JOB2:
Picture
​Note:  You must check the “variable resolution at runtime” checkbox.
Picture
As you can see in this example, to catch the output of the previous job we used the following statement:
${job:<JOB_NAME>.JSONResult}

In the same way, we could retrieve any variable exposed by a job in the same job stream. The job properties exposed by each type of job (database, Datastage, and so on) are listed in the Workload Automation documentation.
 
Variable passing between jobs –  Jobprop
As we have seen in the previous section, we could access a variable exposed by another job with the following syntax:
${job:<JOB_NAME>. <JOB-VARIABLE-NAME>}
This syntax can also be used to retrieve custom variables set and exposed at runtime by a job using a utility provided by the tool: Jobprop.

The Jobprop utility is an installed program present on dynamic agents that allows to set variables and their values in a job and pass the variables to the successive job in the same job stream instance, just like we have seen for job properties.

You can use this command on native or executable job to set variable values you can pass in a successive job in the same job stream. The values are set at run time.
Syntax to set variable: jobprop <VAR-NAME> <value>
Syntax to get variable: ${job:<JOB_NAME>.<VAR-NAME>}
 
Where JOB_NAME is the name of the job present in the same jobstream and VAR-NAME is the key to get the variable value.
Once the variable has been set with jobprop, it can be used by other jobs just like any job property.

This is an example of how to use jobprop between two jobs.

Example

JOB_A
<jsdle:executable interactive="false">
<jsdle:script>#!/bin/sh
. /home/ITAuser/TWA/TWS/tws_env.sh
jobprop VAR1 value1
</jsdle:script>
</jsdle:executable>
JOB_B
<jsdl:application name="executable">
<jsdle:executable interactive="false">
<jsdle:script>
echo VAR1=${job:joba.VAR1}
</jsdle:script>
</jsdle:executable>
 
As you can see, we used an executable job to run an inline script calling the jobprop utility. The JOB_B as we have seen before, takes the output of the JOB_A with the syntax: ${job:JOB_NAME.VAR-NAME}.
 
Variabletable job and Variable Table (a more complex scenario)
Sometimes you need to share variables between jobs in different job steams. To do this, it’s required to use another powerful Workload Automation tool: the Variable Table job. With this job type, present in Workload Automation from version 9.4 on, you can use the Variable Table job type to add or modify a variable in a specified variable table at runtime. The Variable Table job type enables variable passing from one job to another, in the same job stream or in a different job stream.

Let’s see the following example. In our company, we have an application that needs to run on a demand basis but in a different way depending on the day of the week. Thus, for each day of the week there is a job stream called APPLICATION and when the application is required to run the correct version should be triggered.
In this scenario, we can’t neither schedule each application’s job stream nor use a unique version of it picking variable values from different variable tables, because each version is very different in terms of dependencies, job run and so on.

Let’s see how we can achieve this goal.
  • First, we define a job stream called “RUN APPLICATION” which uses a variable table called “APPLICATION_DATA_TABLE”:
Picture

​The only job present in this job stream is a “Job Stream submission” type job called “SUBMIT_JOBSTREAM”, that allow us to submit the application job stream for the specific day of the week using a variable present in the variable table assigned to the job stream, in this way: APPLICATION_^DAY_OF_THE_WEEK^ 
Picture
To implement the dynamic selection of the job stream, we need to figure out a way to dynamically update the value of the variable table.

We can define a job stream called “CALC_DAY” with 2 jobs. The first one calculates the day of the week and exposes the variable with the “jobprop” utility as explained in the previous section.

​The second job is of type “Variable Table” and it is used to automatically update the variable in the “APPLICATION_DATA_TABLE” based on the jobprop variable exposed by the previous job.
Picture
As shown in the above picture, in the “variable table” job we can specify the name of an existing variable in a variable table and assign a new value to it. The value chosen in this case is a variable exposed with JobProp by the predecessor job with the syntax already shown in the previous sections.

Finally, we set up dependencies to have the “CALC_DAY” job stream run before the “RUN_APPLICATION” job stream. 

​This is the final design of the 2 job streams:
Picture
So, whenever the application is required to run, the CALC_DAY job is released by the user starting the flow, updating the variable table with the day of execution, and finally submitting the correct job stream accordingly to the day of execution.
 
With this overview on the variable passing features offered by Workload Automation, we hope we gave some hints to “get rid of headache”.

If you need any clarification, feel free to leave a message in the message box below.

​Good work with Workload Automation and see you soon!
 
Giacomo & Riccardo
​
Picture
​Giacomo del Vecchio
Workload Automation Consultant at HCL Technologies.
Giacomo provides services as Workload Automation SME. He designs and implements solutions to satisfy specific customer's requirements, supports and participates in pre-sales activities (RFx, POC), interacts with customers on a daily basis both remote and on-site (EU, US, India).
Picture
Riccardo Belloni
Senior Workload Automation Consultant at HCL Technologies. Riccardo is part of the HCL Workload Automation services team. He is a Workload Automation expert who provides technical support to the customer to manage and create the desired solutions. Develops and provides technical education. His ability to change on various IT aspects was based on +7 years of IT work and consulting. He loves freediving and spearfishing, these passions took him, since 2014, to become a freediving instructor and two years ago a rescue (and bls) instructor.
1 Comment
Allta
8/6/2019 10:46:33 am

Hello,

This is very interesting.
I am trying to do something similar with no success..

You talked about loo, I assume it's looping over jobs inside a jobstream ?


I am trying to write a job with a for loop inside.

And inside this for loop, I am trying to call a new job using a different alias everytime like this :


for product in $(cat /tmp/ListeProduct)
do
echo $product
/opt/IBM/TWA/TWS/bin/jobprop ProductName ${prodcut}
conman sbj "INTE01#EXPCONSOLOG01_LD_UNIX_ARCHIVAGE;alias=ARCHIVAGE_TEST_${product}"
done

But only the last iteration job is executed. The variable with jobprop works well but not the submit job.

Reply

Your comment will be posted after it is approved.


Leave a Reply.

    Archives

    March 2025
    February 2025
    January 2025
    December 2024
    November 2024
    October 2024
    September 2024
    August 2024
    July 2024
    June 2024
    May 2024
    April 2024
    March 2024
    February 2024
    January 2024
    October 2023
    August 2023
    July 2023
    June 2023
    May 2023
    April 2023
    March 2023
    February 2023
    January 2023
    December 2022
    September 2022
    August 2022
    July 2022
    June 2022
    May 2022
    April 2022
    March 2022
    February 2022
    January 2022
    December 2021
    October 2021
    September 2021
    August 2021
    July 2021
    June 2021
    May 2021
    April 2021
    March 2021
    February 2021
    January 2021
    December 2020
    November 2020
    October 2020
    September 2020
    August 2020
    July 2020
    June 2020
    May 2020
    April 2020
    March 2020
    January 2020
    December 2019
    November 2019
    October 2019
    August 2019
    July 2019
    June 2019
    May 2019
    April 2019
    March 2019
    February 2019
    January 2019
    December 2018
    November 2018
    October 2018
    September 2018
    August 2018
    July 2018
    June 2018
    May 2018
    April 2018
    March 2018
    February 2018
    January 2018
    December 2017
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    June 2017
    May 2017

    Categories

    All
    Analytics
    Azure
    Business Applications
    Cloud
    Data Storage
    DevOps
    Monitoring & Reporting

    RSS Feed

www.hcltechsw.com
About HCL Software 
HCL Software is a division of HCL Technologies (HCL) that operates its primary software business. It develops, markets, sells, and supports over 20 product families in the areas of DevSecOps, Automation, Digital Solutions, Data Management, Marketing and Commerce, and Mainframes. HCL Software has offices and labs around the world to serve thousands of customers. Its mission is to drive ultimate customer success with their IT investments through relentless innovation of its products. For more information, To know more  please visit www.hcltechsw.com.  Copyright © 2024 HCL Technologies Limited
  • Home
  • Blogs
  • Forum
  • Resources
  • Events
  • About
  • Contact
  • What's new