WORKLOAD AUTOMATION COMMUNITY
  • Home
  • Blogs
  • Forum
  • Resources
  • Events
    • IWA 9.5 Roadshows
  • About
  • Contact
  • What's new

Analyzing Dynamic Agent Job Scheduling Internal Status

8/30/2017

0 Comments

 
Picture
Currently Workload Automation customers are moving increasingly to dynamic agent scheduling managed by Dynamic Broker component (ITDWB enterprise application). Dynamic agent scheduling introduces additional processing to the traditional one.

There are multiple processing steps between job execute notification, via mailman server, to Broker component and the final job dispatch to the remote JobManager. All these steps correspond to an internal job status in the JOB_BROKER_JOBS table in the DWB schema that map to WAIT status in the plan (which could be check using conman or the Dynamic Console). The WAIT status means, as documented, that the job is waiting to fulfill its dependencies. In this blog, it is explained how to retrieve a meaningful detailed status of Jobs.

In particular, in the case of heavy workloads, it could be useful to understand the overall status of jobs grouped by internal state to monitor potential bottlenecks.

It could be implemented with the following query for DB2 database:

    
In spite of the multitudes of internal status, it is very important to monitor some of these in case of apparent internal job hanging.
 
In this study case, the former query has been used for sampling the internal broker status during a scheduling workload scenario in addition to a ‘conman sj …  ‘  sampling. During the peak, there were constantly thousands of jobs in WAIT status; that plan status was associated to internal Broker state “RESOURCE_ALLOCATION_RECEIVED”. The latter status means that all internal processing has been completed, including the logical resources assignment (cpu limit constraint had already been resolved by batchman).  
Picture
Picture
In this study case, the root cause of this enqueuing was found at agent side where kernel process running queue had high occupancy level causing an high service time for the request of job execution coming from Broker.
 
Monitoring WorkManager.JobDispatcherWorkManager threads and transaction service time it could be detected that both had increased and the former was capped during the workload peak.

The capping values are defined in the JobDispatcherConfig.properties, for instance the one related to the thread that submit jobs to remote JobManager:

Queue.actions.4=execute
Queue.size.4=30 (in this case the capping value is 30)

This value could be tuned (increased) in order to have more parallel threads processing the jobs whose internal broker state is “RESOURCE_ALLOCATION_RECEIVED”. It is not advisable to increment this value If the scheduling workload is defined against a limited set of dynamic agents, causing further resource usage overhead at agent side.

​In any case, while changing the Queue.size.# value,  the following formula must be verified:
Picture
While running the above explained query over time it could be verified that the number of jobs in final status could change accordingly with the cleanup policy in the JobDispatcherConfig.properties:

    
Picture
Pier Fortunato Bottan
Software Engineer
Workload Automation

Pier is a Software Engineer with long standing experience in software performance discipline. Pier has a degree in Physics and is currently based in the HCL Products and Platforms Rome software development laboratory.
0 Comments

Your comment will be posted after it is approved.


Leave a Reply.

    Archives

    March 2023
    February 2023
    January 2023
    December 2022
    September 2022
    August 2022
    July 2022
    June 2022
    May 2022
    April 2022
    March 2022
    February 2022
    January 2022
    December 2021
    October 2021
    September 2021
    August 2021
    July 2021
    June 2021
    May 2021
    April 2021
    March 2021
    February 2021
    January 2021
    December 2020
    November 2020
    October 2020
    September 2020
    August 2020
    July 2020
    June 2020
    May 2020
    April 2020
    March 2020
    January 2020
    December 2019
    November 2019
    October 2019
    August 2019
    July 2019
    June 2019
    May 2019
    April 2019
    March 2019
    February 2019
    January 2019
    December 2018
    November 2018
    October 2018
    September 2018
    August 2018
    July 2018
    June 2018
    May 2018
    April 2018
    March 2018
    February 2018
    January 2018
    December 2017
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    June 2017
    May 2017

    Categories

    All
    Analytics
    Azure
    Business Applications
    Cloud
    Data Storage
    DevOps
    Monitoring & Reporting

    RSS Feed

www.hcltechsw.com
About HCL Software 
HCL Software is a division of HCL Technologies (HCL) that operates its primary software business. It develops, markets, sells, and supports over 20 product families in the areas of DevSecOps, Automation, Digital Solutions, Data Management, Marketing and Commerce, and Mainframes. HCL Software has offices and labs around the world to serve thousands of customers. Its mission is to drive ultimate customer success with their IT investments through relentless innovation of its products. For more information, To know more  please visit www.hcltechsw.com.  Copyright © 2019 HCL Technologies Limited
  • Home
  • Blogs
  • Forum
  • Resources
  • Events
    • IWA 9.5 Roadshows
  • About
  • Contact
  • What's new