WORKLOAD AUTOMATION COMMUNITY
  • Home
  • Blogs
  • Forum
  • Resources
  • Events
    • IWA 9.5 Roadshows
  • About
  • Contact
  • What's new

Creating a File Dependency for a Job StreamĀ using Start Condition

10/15/2020

0 Comments

 
Picture
Waiting for a file to arrive before a job can start processing it is the most common and quintessential requirement for any workload automation tool.  Historically, this was done by creating a File dependency using OPENS and Unix tricks to manage wild cards and multiple matching files.  Then Event Driven Workload Automation was introduced where an Event Rule could monitor a file with wild cards and when the condition was satisfied the dependent Job Stream was submitted.  With the advent of the Start Condition feature introduced in version 9.4 Fix Pack 1, the event rule file monitor capability is integrated into a Job Stream using the utility, filemonitor.  Another important aspect of this requirement is to pass the name of the matching file to another job in the workflow to process the data contained in the file.   ​
In order to eliminate the need to use composer and conman commands currently used on Fault-Tolerant Agents (FTA), the following new job types are also explored to perform the same activities on Dynamic Agents.
  1. Variable Table Update
  2. Job Stream Submission

Creating a File Dependency for a Job Stream using Start Condition
This section explores the feature to use a Start Condition in a Job Stream and also uses an internal variable, ListOfFilesFound, and an output file to pass the name of the matching file to a successor job in the same Job Stream.

How to use the Start Condition feature with File Created 
​

Follow the steps below to create a Start Condition for a Job Stream 
  1. Login to the Dynamic Workload Console  
  2. Navigate to Design à Manage Workload Definitions 
  3. Select Create New à Job Stream 
  4. In the General tab, fill in the Job Stream name and Workstation 
  5. In the Start condition tab, fill in the values in the fields as shown below: 
  • Condition: File created An existing file with a matching name will not satisfy the condition, a new file with a matching name must be created. 
  • Workstation: EEL 
  • File: /tmp/start.cond.*.txt 
  • Output file:/tmp/start.cond.out.txt 
  • Job name:FILE_CREATED  A file monitor job with this name is auto generated. This job iteratively keeps monitoring the file and when satisfied, submits  a new instance of the Job Stream. ​
Picture
How to use the Name of the File via a Variable and an Output File

​
Follow the steps below to create a Start Condition for a Job Stream

1. Select Create New à Job Definition à UNIX
2. In the General tab, fill in the Job name, Workstation, and Login
3. In the Task tab, enter the following 
​Script name: /bin/echo "File Monitored: ${job:FILE_CREATED.ListOfFilesFound}"
Picture
4. Click on Save
5. Select Create New à Job Definition à UNIX
6. In the General tab, fill in the Job name, Workstation, and Login
7. In the Task tab, enter the following 
​Script name: cat /tmp/start.cond.out.txt
Picture
8. Click on Save
9. Add the two jobs created earlier to the Job Stream
10. Click on Save
11. The following is the Job Stream definition is composer format.
SCHEDULE EEL#START_COND_JS 
STARTCOND FILECREATED EEL#"/tmp/start.cond.*.txt" INTERVAL 60 
( ALIAS FILE_CREATED RERUN OUTFILE "/tmp/start.cond.out.txt" ) 
:
EEL#FILE_NAME_FROM_VARIABLE
 SCRIPTNAME "/bin/echo \"File Monitored: ${job:FILE_CREATED.ListOfFilesFound}\""
 STREAMLOGON iwadmin
 DESCRIPTION "Retrieve file name in Start Cond from an internal variable, ListOfFilesFound"
 TASKTYPE UNIX
 RECOVERY STOP
 
EEL#FILE_NAME_FROM_OUTPUT_FILE
 SCRIPTNAME "cat /tmp/start.cond.out.txt"
 STREAMLOGON iwadmin
 DESCRIPTION "Retrieve file name in Start Cond from the Output file field"
 TASKTYPE UNIX
 RECOVERY STOP
 FOLLOWS FILE_NAME_FROM_VARIABLE
 
END
12. ​Select the Job Stream Name and choose Select an Action à Submit Job Stream into Current Plan

Monitor the Status of Job Stream with the Start Condition and File Created

​Follow the steps below to monitor the status of the Job Stream
  1. Login to the Dynamic Workload Console 
  2. Navigate to Monitoring and Reporting à Monitor Workload
  3. Choose Job Stream for Object Type and click on Run
  4. In the Filter window, enter Start_cond and press Enter
  5. In the resulting view, Right Click on the Job Stream, START_COND_JS, and choose Jobs
  6. Notice that there are 4 jobs in the Job Stream
          FILE_CREATED                                      Auto generated job to keep monitoring the file
          RESTART_STARTCOND                       Auto generated job to submit the Job Stream
          FILE_NAME_FROM_VARIABLE           Job to retrieve the matching file from a variable
          FILE_NAME_FROM_OUTPUT_FILE    Job to retrieve the matching file from an output file
Picture
7. Left click on the Job, FILE_CREATED, to view its properties.  Click on Extra Properties tab to view the internal variable that stores the name of the matching file.
Picture
8. ​Right click on the Job, FILE_NAME_FROM_VARIABLE, and choose Job Log.  Notice that the name of the matching file is retrieved from the internal variable, ListOfFilesFound, set by the job,
FILE_CREATED, ${job:FILE_CREATED.ListOfFilesFound}.
​File Monitored: /tmp/start.cond.092920201352.txt
Picture
9. Right click on the Job, FILE_NAME_FROM_OUTPUT_FILE, and choose Job Log.  Notice that the name of the matching file is retrieved from the file name specified in the Output file field, 
​cat /tmp/start.cond.out.txt
/tmp/start.cond.092920201352.txt
Picture
How to use the Start Condition feature with File Modified

Follow the steps below to create a Start Condition for a Job Stream
  1. Login to the Dynamic Workload Console 
  2. Navigate to Design à Manage Workload Definitions
  3. Select Create New à Job Stream
  4. In the General tab, fill in the Job Stream name and Workstation
  5. In the Start condition tab, fill in the values in the fields as shown below:
  • Condition:         File modified         An existing file with a matching name will not satisfy the condition, the file with a matching name must be modified.
  • Workstation:     EEL
  • File:                  /tmp/start.cond.*.txt
  • Output file:       /tmp/startcond.out      Must have a different pattern than the file being monitored.
  • Additional Parameters:  -modificationCompletedTime 120       When a file is modified, the event is not sent immediately, but only after the interval of time specified by -modificationCompletedTime <seconds> has elapsed and during which no additional changes were made to the file.  For other parameters, refer to Chapter 16, Using utility commands, in the IBM Workload Scheduler version 9.5 User’s Guide and Reference.
  • Job name:     FILE_MODIFIED     A file monitor job with this name is auto generated.  This job iteratively keeps monitoring the file and when satisfied, submits a new instance of the Job Stream
Picture
​6. Select Create New à Job Definition à UNIX
7. In the General tab, fill in the Job name, Workstation, and Login
8. In the Task tab, enter the following 
​    Script name: /bin/echo "File Monitored: ${job:FILE_MODIFIED.ListOfFilesFound}"
Picture
9. Click on Save
10. Select Create New à Job Definition à UNIX
11. In the General tab, fill in the Job name, Workstation, and Login
12. In the Task tab, enter the following 
​     Script name: cat /tmp/startcond.out
Picture
13. Click on Save
14. Add the two jobs created earlier to the Job Stream
15. Click on Save
16. The following is the Job Stream definition is composer format.
SCHEDULE EEL#START_COND_JS 
STARTCOND FILEMODIFIED EEL#"/tmp/start.cond.*.txt" INTERVAL 60 
( ALIAS FILE_MODIFIED RERUN PARAMS "-modificationCompletedTime 120" OUTFILE "/tmp/startcond.out" ) 
:
EEL#FILE_NAME_FROM_VARIABLE
 SCRIPTNAME "/bin/echo \"File Monitored: ${job:FILE_MODIFIED.ListOfFilesFound}\""
 STREAMLOGON iwadmin
 DESCRIPTION "Retrieve file name in Start Cond from an internal variable, ListOfFilesFound"
 TASKTYPE UNIX
 RECOVERY STOP
 
EEL#FILE_NAME_FROM_OUTPUT_FILE
 SCRIPTNAME "cat /tmp/startcond.out"
 STREAMLOGON iwadmin
 DESCRIPTION "Retrieve file name in Start Cond from the Output file field"
 TASKTYPE UNIX
 RECOVERY STOP
 FOLLOWS FILE_NAME_FROM_VARIABLE
 
END
​17. Select the Job Stream Name and choose Select an Action à Submit Job Stream into Current Plan

Monitor the Status of Job Stream with the Start Condition with File Modified

Follow the steps below to monitor the status of the Job Stream
  1. Login to the Dynamic Workload Console 
  2. Navigate to Monitoring and Reporting à Monitor Workload
  3. Choose Job Stream for Object Type and click on Run
  4. In the Filter window, enter Start_cond and press Enter
  5. In the resulting view, Right Click on the Job Stream, START_COND_JS, and choose Jobs
  6. Notice that there are 4 jobs in the Job Stream
          FILE_MODIFIED                               Auto generated job to keep monitoring the file
          RESTART_STARTCOND                     Auto generated job to submit the Job Stream
          FILE_NAME_FROM_VARIABLE         Job to retrieve the matching file from a variable
          FILE_NAME_FROM_OUTPUT_FILE    Job to retrieve the matching file from an output file
Picture
7. Left click on the Job, FILE_CREATED, to view its properties.  Click on Extra Properties tab to view the internal variable that stores the name of the matching file.
Picture
8. ​Right click on the Job, FILE_NAME_FROM_VARIABLE, and choose Job Log.  Notice that the name of the matching file is retrieved from the internal variable, ListOfFilesFound, set by the job,
FILE_CREATED, ${job:FILE_MODIFIED.ListOfFilesFound}.
​File Monitored: /tmp/start.cond.092920201352.txt
Picture
9. Right click on the Job, FILE_NAME_FROM_OUTPUT_FILE, and choose Job Log.  Notice that the name of the matching file is retrieved from the file name specified in the Output file field, 
cat /tmp/start.cond.out.txt
/tmp/start.cond.092920201352.txt
Picture
Create a Variable Table job type to update a Variable with the name of the file monitored
​

Follow the steps below to create a job to update a Variable in a Variable Table with the name of the File that was monitored via the Start Condition.
  1. Login to the Dynamic Workload Console 
  2. Click on Design  Manage Workload Definitions
  3. When the Workload Designer window opens, click on Create New and enter variable in the search window, click on the Variable Table job type in the result.​
Picture
4. Fill in the Name, Workstation, Description, and click on Variable resolution at runtime fields as shown below.
Picture
5. Click on the Connection tab and fill in the information to connect to the MDM as shown below.
Picture
6. Click on Test Connection to verify the URL, Port, and credentials for the MDM.
7. Click on the Action tab and fill in the fields as shown below.  

​Variable Table:                         The name of the Variable Table where the variable needs to be updated
                                                   CLC_CLC2ORA_TABLE
Name:                                        The name of the Variable to be updated
                                                   SOURCEFILE
Value:                                        The contents of the output file from the Start Condition
                                                  $(cat /tmp/startcond.out)
                                                  Note that the variable, ${job:FILE_MODIFIED.ListOfFilesFound}, can’t be 
                                                  used in this scenario as the value contains double quotes around the file name, which results in
                                                  two sets of double quotes,  which is not supported in a Variable definition.
Picture
8. Click on Save.


Create a Submit Job Stream job type to submit a Job Stream that retrieves the name of the file monitored
​

Follow the steps below to create a job to update a Variable in a Variable Table with the name of the File that was monitored via the Start Condition.
  1. Login to the Dynamic Workload Console 
  2. Click on Design  Manage Workload Definitions
  3. When the Workload Designer window opens, click on Create New and enter job stream in the search window, click on the Job Stream Submission job type in the result.
Picture
4. Fill in the Name, Workstation, Description, and click on Variable resolution at runtime fields as shown below.
Picture
5. Click on the Connection tab and fill in the information to connect to the MDM as shown below.
Picture
6. Click on Test Connection to verify the URL, Port, and credentials for the MDM.
7. Click on the Action tab and fill in the fields as shown below.  
​Workstation:         The name of the Workstation where the Job Stream is defined
Job Stream:          The name of the Job Stream that uses the Variable set earlier
Select:                    Validates the name of the Job Stream entered
Picture
8. Click on Save.

​Want to learn more? Write me at Sajjad.Kabir@hcl.com or schedule a demo.

Author's Bio
Picture
​Sajjad Kabir, Solutions Architect, HCL Software
 
Sajjad Kabir is an Information Technology professional with over 25 years of diverse industry experience with primary emphasis in IT Architecture and Service Management. Extensive experiences in management, architecture, design, development, implementation, and systems integration solutions in multiple industries, platforms, and network environments. 
View my profile on LinkedIn
0 Comments

Your comment will be posted after it is approved.


Leave a Reply.

    Archives

    March 2023
    February 2023
    January 2023
    December 2022
    September 2022
    August 2022
    July 2022
    June 2022
    May 2022
    April 2022
    March 2022
    February 2022
    January 2022
    December 2021
    October 2021
    September 2021
    August 2021
    July 2021
    June 2021
    May 2021
    April 2021
    March 2021
    February 2021
    January 2021
    December 2020
    November 2020
    October 2020
    September 2020
    August 2020
    July 2020
    June 2020
    May 2020
    April 2020
    March 2020
    January 2020
    December 2019
    November 2019
    October 2019
    August 2019
    July 2019
    June 2019
    May 2019
    April 2019
    March 2019
    February 2019
    January 2019
    December 2018
    November 2018
    October 2018
    September 2018
    August 2018
    July 2018
    June 2018
    May 2018
    April 2018
    March 2018
    February 2018
    January 2018
    December 2017
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    June 2017
    May 2017

    Categories

    All
    Analytics
    Azure
    Business Applications
    Cloud
    Data Storage
    DevOps
    Monitoring & Reporting

    RSS Feed

www.hcltechsw.com
About HCL Software 
HCL Software is a division of HCL Technologies (HCL) that operates its primary software business. It develops, markets, sells, and supports over 20 product families in the areas of DevSecOps, Automation, Digital Solutions, Data Management, Marketing and Commerce, and Mainframes. HCL Software has offices and labs around the world to serve thousands of customers. Its mission is to drive ultimate customer success with their IT investments through relentless innovation of its products. For more information, To know more  please visit www.hcltechsw.com.  Copyright © 2019 HCL Technologies Limited
  • Home
  • Blogs
  • Forum
  • Resources
  • Events
    • IWA 9.5 Roadshows
  • About
  • Contact
  • What's new