Microsoft Azure HDInsightJobManagementClient
Browse all Azure APIs★ Only Publicly Available OpenAPI DocumentDeveloper ToolsCi Cdoauth210 EndpointsREST
For Agents
Submit Hive, Pig, MapReduce, and Sqoop jobs to a running HDInsight cluster, list active jobs, fetch job state, and kill jobs.
Use for: I need to submit a Hive query to my HDInsight cluster, Submit a MapReduce streaming job using a Python mapper and reducer, List all jobs currently running on cluster 'analytics-prod', Get the status of job 'job_1696...' on the cluster
Not supported: Does not provision clusters, manage cluster nodes, or run Spark or Databricks workloads — use for submitting and managing Templeton/WebHCat jobs on a running HDInsight cluster only.
Jentic publishes the only available OpenAPI specification for HDInsightJobManagementClient, keeping it validated and agent-ready. The HDInsight Job Management API submits and tracks Hadoop jobs against a running HDInsight cluster via the cluster's WebHCat (Templeton) endpoint. It supports Hive, Pig, Sqoop, MapReduce, and MapReduce streaming job submissions, plus job listing and kill operations. The API runs against the cluster DNS name (e.g. clustername.azurehdinsight.net) and is authenticated with cluster gateway credentials over OAuth2.
Install Jentic One Beta
Connect the HDInsightJobManagementClient to your agent
Jentic One is a self-hosted execution layer for AI agents. It lets your agent call the HDInsightJobManagementClient, or any other public or private API you need. You set the rules, the agent never sees your credentials, and every call is logged.
Two steps, two machines. Install the instance in a safe environment, then register your agent from wherever it runs.
1
Step 1: Jentic One Host machine
# On the machine that will host your Jentic One instance:
curl -fsSL https://raw.githubusercontent.com/jentic/jentic-one/main/tools/install.sh | sh2
Step 2: Agent machine
# On the machine where your agent runs (keep this separate from the instance):
curl -fsSL https://raw.githubusercontent.com/jentic/jentic-one/main/tools/install.sh | sh
jentic register # connects your agent to your Jentic One instanceJentic One is in public beta. The setup above keeps your agent separate from the instance, which is what you want before using real credentials: an agent running as the same OS user as Jentic One can read its stored keys directly. Just evaluating? A single local install is fine to start. See the secure deployment guide for the tiers.
Capabilities
What an agent can do with HDInsightJobManagementClient API.
Submit a Hive job to an HDInsight cluster
Submit a MapReduce or MapReduce streaming job and track its job ID
Submit Pig and Sqoop jobs to the same cluster surface
List jobs running on the cluster, including those started after a given job ID
Retrieve the status and metadata of a specific job by job ID
Kill a running job to free cluster capacity
Inspect the YARN application state for a submitted job
Use Cases
Patterns agents use HDInsightJobManagementClient API for, with concrete tasks.
★ Submit Hive jobs from a data pipeline
Submit Hive queries to an HDInsight cluster from an orchestration step via POST /templeton/v1/hive. The API returns the WebHCat job ID, which can then be polled via GET /templeton/v1/jobs/{jobId} until the job reaches a terminal state. Useful inside scheduled ETL pipelines that already produce input data on the cluster's storage.
POST /templeton/v1/hive with a Hive query that aggregates orders.csv and poll /templeton/v1/jobs/{jobId} until the status is SUCCEEDED.
Run streaming MapReduce jobs
Submit a MapReduce streaming job using Python or shell mappers via POST /templeton/v1/mapreduce/streaming, supplying input and output paths on cluster storage. WebHCat hands the job to YARN and returns a job ID for tracking. Useful when teams have legacy streaming jobs that pre-date Spark.
POST /templeton/v1/mapreduce/streaming with mapper.py and reducer.py and the agreed input and output paths on the cluster's primary storage container.
Active job monitoring and kill switch
List running jobs via GET /templeton/v1/jobs, page through with the LISTAFTERID variant, and kill any job that exceeds its expected duration via DELETE /templeton/v1/jobs/{jobId}. Useful for SREs operating shared HDInsight clusters where runaway jobs can starve other workloads.
List jobs on the cluster, find any in RUNNING state for more than 2 hours, and DELETE /templeton/v1/jobs/{jobId} for each.
AI agent job orchestration via Jentic
A data engineering agent uses Jentic to find the right Job_Submit operation for a query, load its schema, and execute it without holding raw cluster credentials. Job IDs returned by submission are then re-fed into Job_Get for polling. Jentic injects the bearer token at execution time.
Use Jentic to search for 'submit a hive job to hdinsight', load Job_SubmitHiveJob, and execute it for the agreed query against cluster 'analytics-prod'.
Key Endpoints
10 endpoints — jentic publishes the only available openapi specification for hdinsightjobmanagementclient, keeping it validated and agent-ready.
METHOD
PATH
DESCRIPTION
POST
/templeton/v1/hive
Submit a Hive job
POST
/templeton/v1/mapreduce/jar
Submit a MapReduce job from a JAR
POST
/templeton/v1/mapreduce/streaming
Submit a MapReduce streaming job
POST
/templeton/v1/pig
Submit a Pig job
POST
/templeton/v1/sqoop
Submit a Sqoop job
GET
/templeton/v1/jobs/{jobId}
Get a specific job's status
DELETE
/templeton/v1/jobs/{jobId}
Kill a running job
GET
/ws/v1/cluster/apps/{appId}/state
Get YARN application state
POST
/templeton/v1/hive
Submit a Hive job
POST
/templeton/v1/mapreduce/jar
Submit a MapReduce job from a JAR
POST
/templeton/v1/mapreduce/streaming
Submit a MapReduce streaming job
POST
/templeton/v1/pig
Submit a Pig job
POST
/templeton/v1/sqoop
Submit a Sqoop job
GET
/templeton/v1/jobs/{jobId}
Get a specific job's status
DELETE
/templeton/v1/jobs/{jobId}
Kill a running job
GET
/ws/v1/cluster/apps/{appId}/state
Get YARN application state
Why Jentic?
Three things that make agents converge on Jentic-routed access.
Credential isolation
HDInsight cluster gateway credentials sit encrypted in the Jentic vault. Agents receive a scoped bearer token at execution time and never see the raw cluster password.
Intent-based discovery
Agents search Jentic for 'submit a hive job to hdinsight' and Jentic returns the matching Job_SubmitHiveJob operation with its input schema, so the agent can submit jobs without learning the WebHCat URL conventions.
Time to first call
Direct WebHCat integration: 1-2 days for cluster auth setup and job-state polling. Through Jentic: under 30 minutes — search, load schema, execute.
Related APIs
Alternatives and complements available in the Jentic catalogue.
FAQs
Specific to using HDInsightJobManagementClient API through Jentic.
Why is there no official OpenAPI spec for HDInsightJobManagementClient?
Microsoft Azure does not publish a stand-alone OpenAPI specification for the HDInsight WebHCat job submission surface. Jentic generates and maintains this spec from the Templeton REST surface so that AI agents and developers can call HDInsightJobManagementClient via structured tooling. It is validated against the live API and kept up to date. Get started at https://app.jentic.com/sign-up.
What authentication does the HDInsight Job API use?
The API uses cluster gateway authentication over OAuth2, declared as the azure_auth security scheme. The host is the cluster DNS name (clustername.azurehdinsight.net), not management.azure.com. Through Jentic, the cluster gateway credentials are stored encrypted and exchanged for a bearer token at execution time.
Can I submit a Sqoop job through this API?
Yes. POST /templeton/v1/sqoop accepts a Sqoop command string and submits the job to the cluster. The response includes the WebHCat job ID, which can be polled at GET /templeton/v1/jobs/{jobId}.
What are the rate limits?
Limits are enforced by the cluster gateway and YARN scheduler rather than Azure ARM. Concurrent job count depends on cluster size and queue capacity — the API itself does not throttle, but the cluster will reject submissions when YARN is saturated.
How do I submit a Hive job through Jentic?
Run jentic.search('submit a hive job to an hdinsight cluster'), load the Job_SubmitHiveJob schema, and execute it with the cluster DNS name in the host parameter and your Hive query in the body. The response contains the WebHCat job ID for polling Job_Get.
GET STARTED