Description
Key Responsibilities Azure & Databricks Platform Engineering
- Engineer, operate, and evolve enterprise‑grade Azure Databricks platforms across multiple subscriptions and environments (dev, test, prod).
- Design and implement Databricks workspace configurations, cluster policies, pools, and job orchestration with a strong focus on scalability, reliability, and cost efficiency.
- Engineer and maintain Azure platform components supporting Databricks, including networking, identity, storage, and security services.
- Monitor platform health, performance, and usage; proactively design improvements and resolve operational issues.
Databricks‑Centric Data & Analytics Enablement
- Engineer and optimize Databricks integrations with Azure services, including:
- Azure Data Lake Storage (ADLS Gen2)
- Delta Lake and Unity Catalog
- Azure SQL, Azure Synapse, Event Hubs, and other data sources
- Support batch and streaming data pipelines and analytics workloads running on Databricks.
- Partner with Data Engineering, Analytics, and AI teams to ensure performant, reliable, and secure production workloads.
Security, Governance & Compliance Engineering
- Design and implement security and governance controls across Databricks and Azure, including:
- Azure Active Directory integration, RBAC, and SCIM provisioning
- Unity Catalog, data access policies, and cluster governance
- Engineer secure integrations using Azure Key Vault, Private Endpoints, and network security constructs.
- Ensure platform designs comply with organizational security, governance, and audit requirements.
Infrastructure as Code & Platform Automation
- Design, develop, and maintain Terraform‑based IaC for Databricks and supporting Azure infrastructure (workspaces, permissions, networking, storage).
- Build and evolve CI/CD pipelines using Azure DevOps YAML pipelines for platform and infrastructure deployments.
- Promote engineering best practices for IaC, including modularization, versioning, automated testing, and environment consistency.
Collaboration & Platform Leadership
- Act as a subject matter expert for Databricks on Azure, guiding teams on platform usage, architecture, and operational best practices.
- Work cross‑functionally with Cloud, Architecture, CICD, Data, AI, and Application teams.
- Create and maintain platform documentation, reference architectures, and operational runbooks.
Required Qualifications
- 5+ years of hands‑on experience engineering solutions on Microsoft Azure, with a strong focus on Azure Databricks platforms.
- Deep expertise in:
- Azure Databricks (clusters, jobs, pools, Delta Lake, Unity Catalog, workspace engineering)
- Azure core services (Storage, Networking, Identity & Access Management)
- Cost optimization, monitoring, and observability for data platforms
- Strong experience with Terraform, including multi‑subscription and multi‑environment deployments.
- Proficiency with CI/CD pipelines, preferably Azure DevOps.
- Solid understanding of cloud security, governance, and platform engineering practices.
Preferred Certifications Azure & Databricks Certifications
- Microsoft Certified: Azure Administrator Associate
- Microsoft Certified: Azure Solutions Architect Expert
- Databricks Certified Data Platform Engineer or Administrator
HashiCorp Terraform Certifications
- HashiCorp Terraform Associate (003 or later)
- HashiCorp Terraform Engineer
Soft Skills
- Strong communication and documentation skills.
- Ability to explain complex Azure and Databricks platform designs to diverse technical audiences.
- Engineering mindset with a strong focus on automation, reliability, and continuous improvement.
- Collaborative, proactive, and adaptable in a fast‑evolving cloud and data landscape.
Solliciteren