WORKLOAD AUTOMATION COMMUNITY
  • Home
  • Blogs
  • Forum
  • Resources
  • Events
    • IWA 9.5 Roadshows
  • About
  • Contact
  • What's new

Submitting jobs by using Azure Databricks Plugin with Workload Automation

8/11/2021

0 Comments

 
Picture
​Let us begin with understanding of Azure what it is all about before moving to our Azure Databricks plugin and how it benefits our workload automation users.
Picture
“Azure is an open and flexible cloud platform that enables you to quickly build, deploy and manage applications across a global network of Microsoft-managed datacentres. You can build applications using any language, tool, or framework. And you can integrate your public cloud applications with your existing IT environment.”

Azure is incredibly flexible, and allows you to use multiple languages, frameworks, and tools to create the customised applications that you need. As a platform, it also allows you to scale applications up with unlimited servers and storage.

What is an Azure Databricks?

Azure Databricks provides the latest versions of Apache Spark and allows you to seamlessly integrate with open-source libraries. Spin up clusters and build quickly in a fully managed Apache Spark environment with the global scale and availability of Azure.

Azure Databricks is a data analytics platform optimized for the Microsoft Azure cloud services platform. Azure Databricks offers three environments for developing data intensive applications: Databricks SQL, Databricks Data Science & Engineering, and Databricks Machine Learning.

  • Azure databricks plugin supports the latest analytic services for Azure data lake storage Gen1, Azure data lake storage Gen2, Azure blob storage, and other storage accounts.
 
  • It provides an easy-to-use platform for analysts to read data from multiple data sources and turn it into easily understandable data.
 
  • It provides faster performance with various optimizations at the I/O layer and processing layer (Databricks I/O).

Azure Databricks Plugin
 
Log in to the Dynamic Workload Console and open the Workload Designer. Choose to create a new job and select “Azure Databricks Plugin” job type in the Cloud section.
Picture
Fig 1 : Job Definition

Connection Tab
 
Establishing connection to the Azure Databricks Workspace. 
 
Workspace Instance - A unique instance name (per-workspace URL), that is assigned to each Azure Databricks deployment. It is the fully-qualified domain name used to log into your Azure Databricks deployment and make API requests.
 
Example: adb-<workspace-id>.<random-number>.azuredatabricks.net. The workspace ID appears immediately after adb- and before the “dot” (.). For the per-workspace URL https://adb-5555555555555555.19.azuredatabricks.net/
 
Access Token - Enter the access token that is generated in the Azure cloud, to authenticate to and access Databricks REST APIs.
 
We can generate token under User settings in workspace.
Picture
Fig 2

Test Connection
- Click to verify that the connection to the Azure server works correctly.
Picture
Fig 3 : Connection Tab

Action Tab

 
Use this section to define the operation details.
 
Operation
  • Run the selected job
  • Cancel the pending/running job
 
Select – It gives the list of available jobs present in the workspace to select.
 
Details – It gives more information on selected job.
Picture
Fig 4 : Action Tab

Submitting your job

​ 
It is time to Submit your job into the current plan. You can add your job to the job stream that automates your business process flow. Select the action menu in the top-left corner of the job definition panel and click on Submit Job into Current Plan. A confirmation message is displayed, and you can switch to the Monitoring view to see what is going on.
Picture
Fig 5 : Monitor page with extra properties

Once we submit the job, we can cancel the particular job by Kill option.
Picture
Fig 6 

Job Log

Picture
Fig 7
Picture


​​Are you curious to try out the Azure Storage plugin? Download the integrations from the Automation Hub and get started or drop a line at santhoshkumar.kumar@hcl.com

Picture
Suhas H N – Senior Developer at HCL Software
 
Works as a Plugin Developer in Workload Automation team in HCL Software, Bangalore. Acquired skills on Java, Spring, Spring Boot, Microservices, AngularJS, JavaScript.
View my profile on LinkedIn

Picture
Arka Mukherjee, Quality Analyst at HCL Technologies
 
Working as Quality Analyst for the Workload Automation team in HCL Software, Bangalore. Worked both in manual and automation test scenarios across various domains.
View my profile on LinkedIn
0 Comments

Your comment will be posted after it is approved.


Leave a Reply.

    Archives

    March 2023
    February 2023
    January 2023
    December 2022
    September 2022
    August 2022
    July 2022
    June 2022
    May 2022
    April 2022
    March 2022
    February 2022
    January 2022
    December 2021
    October 2021
    September 2021
    August 2021
    July 2021
    June 2021
    May 2021
    April 2021
    March 2021
    February 2021
    January 2021
    December 2020
    November 2020
    October 2020
    September 2020
    August 2020
    July 2020
    June 2020
    May 2020
    April 2020
    March 2020
    January 2020
    December 2019
    November 2019
    October 2019
    August 2019
    July 2019
    June 2019
    May 2019
    April 2019
    March 2019
    February 2019
    January 2019
    December 2018
    November 2018
    October 2018
    September 2018
    August 2018
    July 2018
    June 2018
    May 2018
    April 2018
    March 2018
    February 2018
    January 2018
    December 2017
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    June 2017
    May 2017

    Categories

    All
    Analytics
    Azure
    Business Applications
    Cloud
    Data Storage
    DevOps
    Monitoring & Reporting

    RSS Feed

www.hcltechsw.com
About HCL Software 
HCL Software is a division of HCL Technologies (HCL) that operates its primary software business. It develops, markets, sells, and supports over 20 product families in the areas of DevSecOps, Automation, Digital Solutions, Data Management, Marketing and Commerce, and Mainframes. HCL Software has offices and labs around the world to serve thousands of customers. Its mission is to drive ultimate customer success with their IT investments through relentless innovation of its products. For more information, To know more  please visit www.hcltechsw.com.  Copyright © 2019 HCL Technologies Limited
  • Home
  • Blogs
  • Forum
  • Resources
  • Events
    • IWA 9.5 Roadshows
  • About
  • Contact
  • What's new