This documentation explains how to configure Databricks JDBC authentication using Azure Managed Identity from an Azure Kubernetes Service (AKS) cluster. The setup leverages Workload Identity to enable AKS pods to seamlessly authenticate with Databricks without managing secrets. 


Reference: Deploy and configure an AKS cluster with workload identity - Azure Kubernetes Service | Microsoft Learn


Prerequisites

  • User-Managed Identity (UMI) is created in Azure. 
  • The User-Managed Identity is added to the Databricks workspace as a service account. 

  • Data-Reader access is granted to the User-Managed Identity for the required catalog in Databricks.


Steps to Configure:


1. Create a Service Account in AKS 


    Define a Kubernetes Service Account that references the User-Managed Identity Client ID via annotations.

apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
azure.workload.identity/client-id: "<USER_ASSIGNED_IDENTITY_CLIENT_ID>"
name: <SERVICE_ACCOUNT_NAME>
namespace: <NAMESPACE>

    This service account will be linked to your AKS pods that require Databricks JDBC authentication. 


2. Get the OIDC Issuer URL of the AKS Cluster


    Run the following command to retrieve the OIDC Issuer URL:

az aks show \
--name "<AKS_CLUSTER_NAME>" \
--resource-group "<RESOURCE_GROUP>" \
--query "oidcIssuerProfile.issuerUrl" \
--output tsv

    This URL will be required to configure federated identity credentials. 


3. Create Federated Identity Credentials

    

    Link the AKS service account with the User-Managed Identity by creating federated identity credentials.

az identity federated-credential create \
--name <FEDERATED_IDENTITY_CREDENTIAL_NAME> \
--identity-name "<USER_ASSIGNED_IDENTITY_NAME>" \
--resource-group "<RESOURCE_GROUP>" \
--issuer "<AKS_OIDC_ISSUER>" \
--subject system:serviceaccount:"<SERVICE_ACCOUNT_NAMESPACE>":"<SERVICE_ACCOUNT_NAME>" \
--audience api://AzureADTokenExchange

This allows the AKS pod to exchange tokens securely with Azure AD. 


4. Annotate AKS Engine Pods


Update your engine pods to use workload identity authentication. Add the following:

  • Label: azure.workload.identity/use: "true"

  • Service Account Reference: Attach the service account created 



JDBC Authentication Flow:

  1. Pod in AKS uses the annotated service account.

  2. Workload Identity integrates with Azure AD to issue a token via the User-Managed Identity.

  3. The token is used for Databricks JDBC authentication.

  4. Databricks validates the managed identity against the configured service account.

Architecture Diagram:

    Below is a high-level representation of the flow:

graph TD
A[AKS Pod with Service Account] -->|OIDC Token| B[Azure AD]
B -->|Exchange for Access Token| C[User-Managed Identity]
C -->|Token| D[Databricks JDBC Endpoint]
D -->|Data Access| E[Databricks Catalog]