Deep dive into Workload Automation Dynamic Scheduling

This blog addresses some common questions related to Dynamic Scheduling in Workload Automation.

​Make a dynamic agent work
A question that I often hear from Workload Automation customers is:
​
​"I would like to start using Dynamic Scheduling, but I only have FTAs in my environment. Do I have to install at least one dynamic agent for each FTA?"

It is not necessary to install a new dynamic agent if an FTA is already installed on the system, since a dynamic agent component is already installed together with the FTA. However, sometimes it is not working. The reason might be that the dynamic agent component hasn’t been started, or it is not properly configured, or the broker component on the MDM has not been well configured too.

Let’s start investigating the Broker component on the MDM side. We have to make sure that the Broker database contains the correct information. On the MDM run the following commands from <TWA home>/TDWB/bin:

root@EU-HWS-LNX110:/opt/IBM/TWA/TDWB/bin# ./exportserverdata.sh -dbUsr <db user> -dbPwd <db password>
Setting CLI environment variables....
AWKCLI168I Server data successfully exported.
root@EU-HWS-LNX110:/opt/IBM/TWA/TDWB/bin# cat server.properties
https://EU-HWS-LNX110.NPROD.HCLPNP.COM:31116/JobManagerRESTWeb/JobScheduler

The command exportserverdata saves in the file server.properties the list of Broker component addresses, typically the addresses of MDM and all the Backup MDMs. In this case the backup master is not set and EU-HWS-LNX110.NPROD.HCLPNP.COM is the hostname of the MDM while 31116 is WebSphere port for incoming requests.

If addressed are missing, or not correctly set, it is possible to correct them by editing the file server.properties,  importing the new values using the script <TWA home>/TDWB/bin/importserverdata.sh and passing the updated version of server.properties.

It is possible to verify if the import operation was really successful running again the script exportserverdata.sh and checking that the new addresses are now saved in server.properties output file.

​Now, it is necessary to connect the agent system where the FTA and the dynamic agent are installed. The dynamic agent processes are:

tws94fp2  2262     1  0 Sep03 ?        00:00:06 /home/tws94fp2/TWS/ITA/cpa/ita/agent
tws94fp2  2355  2262  0 Sep03 ?        00:55:35 /home/tws94fp2/TWS/bin/JobManager

i.e. agent (the watchdog process) and JobManager.

If they are running, stop them using the script:  <TWA home>/TWS/ShutDownLwa

After they have been stopped, make sure that the agent port to accept requests from the Broker is set. It is written in <TWA home>/TWS/ITA/cpa/ita/ita.ini. In [ITA] section.

​If you find the following settings:

tcp_port = 0
ssl_port = 31114

It means that secure communication on port 31114 is in place. If the port 31114 (the default one at installation time) is not free and cannot be used, it is possible to change this value to the proper port number. Then, edit the file <TWA home>/TWS/ITA/cpa/config/JobManager.ini and check that the following parameters are correctly set in [ResourceAdvisorAgent] section:

FullyQualifiedHostname = EU-HWS-LNX110.NPROD.HCLPNP.COM
ResourceAdvisorUrl = https://EU-HWS-LNX110.NPROD.HCLPNP.COM:31116/JobManagerRESTWeb/JobScheduler/resource
ComputerSystemDisplayName = EU-HWS-LNX110_1

Where:

• FullyQualifiedHostname is the hostname of the system hosting the agent.  
• ResourceAdvisorUrl must point to the hostname (or ip address) and port number that have just been set using importserverdata.
• ComputerSystemDisplayName is the cpu name as we want it to be defined in the database after the agent registration.

Then, it is necessary to delete the lines with the key AgentID and BackupResourceAdvisorUrls: this will force a new agent registration if the agent successfully connects to the Broker component. Restart the dynamic agent with the script <TWA home>/TWS/StartUpLwa.

After the agent has been restarted, check the agent logs, in
<TWA home>/TWS/stdlist/JM/JobManager_message.log,
A sequence of messages like the following should be present:

2018-11-01 18:50:08.187-04:00|2108348224|29408|EU-HWS-LNX110.NPROD.HCLPNP.COM|Start Logger
2018-11-01 18:50:08.188-04:00|INFO|JobManager|2108348224|29408|EU-HWS-LNX110.NPROD.HCLPNP.COM|AWSITA047I Starting subagent "JobManager".
2018-11-01 18:50:08.282-04:00|INFO|JobManager|2108348224|29408|EU-HWS-LNX110.NPROD.HCLPNP.COM|AWSITA090I The Job Status Notification Manager started.
2018-11-01 18:50:08.310-04:00|INFO|JobManager|1786758912|29408|EU-HWS-LNX110.NPROD.HCLPNP.COM|AWSITA050I The agent is open for e-business.
2018-11-01 18:50:08.340-04:00|INFO|JobManager|2108348224|29408|EU-HWS-LNX110.NPROD.HCLPNP.COM|AWSITA257I The Event Driven Workload Automation component was created.
2018-11-01 18:50:08.343-04:00|INFO|JobManager|2108348224|29408|EU-HWS-LNX110.NPROD.HCLPNP.COM|AWSITA257I The Event Driven Workload Automation component was created.
2018-11-01 18:50:14.040-04:00|INFO|JobManager|1778366208|29408|EU-HWS-LNX110.NPROD.HCLPNP.COM|AWSITA083I Resource information was sent to "https://EU-HWS-LNX110.NPROD.HCLPNP.COM:31116/JobManagerRESTWeb/JobScheduler/resource".

The last message, AWSITA083I is the message that certifies that the agent is correctly started and is able to connect to the Broker component. Also, you can verify that the following entries have been added to JobManager.ini and updated in [ResourceAdvisorAgent] section:

AgentID = 2FAC9900CA2711E7BAE78921BEAF999E
BackupResourceAdvisorUrls = https://EU-HWS-LNX110.NPROD.HCLPNP.COM:31116/JobManagerRESTWeb/JobScheduler/resource

Note that BackupResourceAdvisorUrls must contain the same list of Broker server addresses as previously configured with exportserverdata.sh/importserverdata.sh scripts.
It is not necessary to define a dynamic agent cpu in the database. It is automatically defined after a successful registration.

CPUNAME EU-HWS-LNX110_1
  DESCRIPTION "This workstation was automatically created."
  OS UNIX
  NODE EU-HWS-LNX110.NPROD.HCLPNP.COM SECUREADDR 31114
  TIMEZONE America/New_York
  FOR MAESTRO HOST EU-HWS-LNX11_DWB
    AGENTID "2FAC9900CA2711E7BAE78921BEAF999E"
    TYPE AGENT
    PROTOCOL HTTPS
END

The dynamic agent cpu is automatically created with the AGENTID and cpu name that are present in the configuration file JobManager.ini hosted by the Broker workstation. The workstation will be ready for the scheduling activities after next FINAL or after JnextPlan –for 0000.
 ​

​Make the Gateway work
Another question I often hear from customer is: 

"I want to configure a dynamic agent, already installed and running, to communicate with the master domain manager through a gateway. Do I have to install a local or remote gateway?"

If you need to use a remote gateway, it is necessary to install it remotely. Otherwise, you can leverage the installation of the dynamic agent that is currently in use and manually configure the agent to communicate with the master domain manager through the local getaway already installed.

The local gateway configuration file is located in
<TWA installation directory>/TWS/ITA/cpa/config/JobManagerGW.ini
and is used to manage the gateway process JobManagerGW whose lifecycle is handled by the watchdog agent named “agent”.

To configure the local gateway, you need to know the local port of the agent process: it is the value of tcp_port or ssl_port attributes in the configuration file
 <TWA installation directory>/TWS/ITA/cpa/ita/ita.ini, in [ITA] section.

Suppose this value is NNNNN. Then, follow these steps to complete the configuration:
1. Stop the dynamic agent
2. Edit the configuration file JobManagerGW.ini and in [ITA] section make sure that the following settings are in place (consider that GW1 is the gateway ID that has been selected, and EU-HWS-LNX110.NPROD.HCLPNP.COM is the system hostname)

autostart = yes
JobManagerGWID = GW1
JobManagerGWURIs = https://EU-HWS-LNX110.NPROD.HCLPNP.COM:NNNNN/ita/JobManagerGW/JobManagerRESTWeb/JobScheduler/resource

3. Edit the configuration file JobManagerGW.ini and in [ResourceAdvisorAgent] section make sure that the following settings are in place, considering that the port is WebSphere https port

ResourceAdvisorUrl = https://<MDM hostname>:<port>/JobManagerRESTWeb/JobScheduler/resource
BackupResourceAdvisorUrls = https://<MDM hostname>:<port>/JobManagerRESTWeb/JobScheduler/resource
ResourceAdvisorUrl = https://<MDM hostname>:<port>/JobManagerRESTWeb/JobScheduler/resource
BackupResourceAdvisorUrls = https://<MDM hostname>:<port>/JobManagerRESTWeb/JobScheduler/resource

4. Edit the configuration file JobManager.ini and set the following attributes in the following way, considering that EU-HWS-LNX110.NPROD.HCLPNP.COM is, as above, the hostname of the system where the dynamic agent is installed

​ [ResourceAdvisorAgent]
ResourceAdvisorUrl = https://EU-HWS-LNX110.NPROD.HCLPNP.COM:NNNNN/ita/JobManagerGW/JobManagerRESTWeb/JobScheduler/resource
BackupResourceAdvisorUrls = https://EU-HWS-LNX110.NPROD.HCLPNP.COM:NNNNN/ita/JobManagerGW/JobManagerRESTWeb/JobScheduler/resource

5. Restart the dynamic agent.
​After the dynamic agent restart, the following processes are up and running:

tws94fp2 13166     1  0 15:50 ?        00:00:03 /home/tws94fp2/TWS/ITA/cpa/ita/agent
tws94fp2 13180 13166  0 15:50 ?        00:00:05 /home/tws94fp2/TWS/bin/JobManager
tws94fp2 13181 13166  0 15:50 ?        00:00:10 /home/tws94fp2/TWS/bin/JobManagerGW

In JobManager_message.log it is possible to check that JobManager is sending the system resources to the gateway address:

2018-11-25 15:50:17.969-05:00|INFO|JobManager|911644480|13180|EU-HWS-LNX110.NPROD.HCLPNP.COM|AWSITA047I Starting subagent "JobManager".
2018-11-25 15:50:18.190-05:00|INFO|JobManager|595584768|13180|EU-HWS-LNX110.NPROD.HCLPNP.COM|AWSITA050I The agent is open for e-business.
2018-11-25 15:50:24.992-05:00|INFO|JobManager|585017088|13180|EU-HWS-LNX110.NPROD.HCLPNP.COM|AWSITA083I Resource information was sent to "https://EU-HWS-LNX110.NPROD.HCLPNP.COM:3212/ita/JobManagerGW/JobManagerRESTWeb/JobScheduler/resource".

If a job is submitted, it is possible to follow the job lifecycle and the job status update that is sent to the gateway:

2018-11-25 15:52:16.978-05:00|540923648|13180|EU-HWS-LNX110.NPROD.HCLPNP.COM|AWSITA006I The job with ID "AGT94#JOBS.PP1.JNUM-804527197" was successfully created.
2018-11-25 15:52:17.059-05:00|INFO|JobManager|738891520|13180|EU-HWS-LNX110.NPROD.HCLPNP.COM|AWSITA031I The job with ID "AGT94#JOBS.PP1.JNUM-804527197" started.
2018-11-25 15:52:17.074-05:00|INFO|JobManager|738891520|13180|EU-HWS-LNX110.NPROD.HCLPNP.COM|AWSITA034I The job with ID "AGT94#JOBS.PP1.JNUM-804527197" completed successfully.
2018-11-25 15:52:18.437-05:00|INFO|JobManager|729966336|13180|EU-HWS-LNX110.NPROD.HCLPNP.COM|AWSITA109I The notification of the status of the job with ID "AGT94#JOBS.PP1.JNUM-804527197" was succesfully sent to the URI "https://EU-HWS-LNX110.NPROD.HCLPNP.COM:3212/ita/JobManagerGW/JobManagerRESTWeb/JobScheduler/job".

Also, in JobManagerGW_message you can see that system resources are forwarded to the master domain manager:

2018-11-25 18:40:24.518-05:00|INFO|JobManagerGW|1733740288|13181|EU-HWS-LNX110.NPROD.HCLPNP.COM|AWSITA307I Resource information was sent to "https://EU-HWS-LNX110.NPROD.HCLPNP.COM:31116/JobManagerRESTWeb/JobSchedulerGW/resourceGW".

Note that it is always possible to revert this configuration and manually pass from the communication through gateway to the direct connection to the master domain manager. This can be achieved in the following way:
1. Stop the dynamic agent.
2. Edit the configuration file JobManagerGW.ini and in [ITA] section make sure that the following settings are in place:

autostart = no

3. Edit the configuration file JobManager.ini and in [ResourceAdvisorAgent] section make sure that the following settings are in place, considering that the port is the WebSphere https port:

ResourceAdvisorUrl = https://<MDM hostname>:<port>/JobManagerRESTWeb/JobScheduler/resource
BackupResourceAdvisorUrls = https://<MDM hostname>:<port>/JobManagerRESTWeb/JobScheduler/resource

4. Restart the dynamic agent.

After the above steps the dynamic agent is directly connected to the master domain manager without passing through the gateway.
​
If you have any feedback or want to know more about Dynamic Scheduling, contact Paolo Salerno,  paolo.salerno@hcl.com or comment below.
​

​Paolo Salerno
Lead Specialist 

Paolo is a Senior Software Engineer. He has been working in the Workload Automation area since 2011 and in the last 6 years has been covering the position of L3 Support Team Leader.