[Remote] Senior Databricks Engineering Lead

Note: The job is a remote job and is open to candidates in USA. Pacific Northwest National Laboratory (PNNL) is a world-class research institution focused on scientific research and innovation. They are seeking a Senior Azure Databricks Engineer to lead the design, build, and operation of a data lakehouse that powers analytics and reporting across PNNL Enterprise Systems, ensuring data governance and enabling self-service analytics.

Responsibilities

• Lead the design, build, and operation of our data lakehouse that powers analytics and reporting across PNNL Enterprise Systems
• Deliver governed, performant, and reliable data products—especially for ERP and other enterprise—and enable self-service analytics with Power BI and AI/ML
• Design and evolve a Databricks‑based architecture that moves data with confidence from source systems to curated gold tables
• Build robust pipelines that transform raw data into analytics‑ready assets for Power BI and AI/ML, balancing pragmatic MVP delivery with a future‑focused architecture
• Lead modernization from legacy data warehouses and ETL tools into Azure Databricks—refactoring brittle jobs into scalable patterns
• Shape CI/CD for Databricks (e.g., DAB, Azure DevOps, GitHub Actions) and standardize deployment practices across environments
• Configure and operate workspaces, clusters, jobs, and workflows; tune for performance and reliability; and embed data quality, monitoring, and observability to keep critical pipelines healthy
• Implement role‑based access controls, data masking, and fine‑grained models with Unity Catalog to enable secure, compliant data sharing and ensure proper classification, lineage, and auditability
• Guide engineers and analysts in adopting lakehouse best practices and modern data engineering standards—coding, testing, version control, and documentation
• Stay current with Azure and Databricks capabilities, recommending and piloting features like Delta Live Tables and Unity Catalog enhancements to build a secure, reliable, and future‑ready data platform

Skills

• PhD and 3 years of Software Engineering experience -OR-
• MS/MA and 5 years of Software Engineering experience -OR-
• BS/BA and 7 years of Software Engineering experience -OR-
• AA and 16 years of Software Engineering experience in designing, architecting, programming, deploying, and automating software solutions in support of scientific research or consumer digital product development -OR-
• HS/GED and 18 years of Software Engineering experience in designing, architecting, programming, deploying, and automating software solutions in support of scientific research or consumer digital product development
• 7+ years of professional data engineering or platform engineering experience, with 3–5+ years focused on cloud data platforms
• 5+ years of experience operating production Azure Databricks, including Delta Lake, SQL, notebooks, Jobs/Workflows, and cluster management
• Production experience (3-5+ years) designing and operating ingestion-to-gold pipelines (medallion architecture) for ERP or other complex transactional sources
• Experience with Azure Data Factory and/or Fabric Data Pipelines for orchestration and integration
• Familiarity with core Azure services: ADLS Gen2, Key Vault, Azure DevOps or GitHub
• Strong proficiency in Python and SQL in a Spark/Databricks environment
• Experience implementing Databricks Asset Bundles (DAB) or equivalent for CI/CD and standardizing deployment workflows
• Experience using GenAI / LLM-based tools (e.g., GitHub Copilot, Azure OpenAI, Databricks Genie, or similar) to accelerate and automate engineering tasks such as code generation, test creation, documentation, and troubleshooting
• Exposure to agentic AI / AI agents (e.g., orchestrating multi-step AI workflows for data quality checks, pipeline monitoring, or support automation) is a plus

Benefits

• Medical insurance
• Dental insurance
• Vision insurance
• Robust telehealth care options
• Several mental health benefits
• Free wellness coaching
• Health savings account
• Flexible spending accounts
• Basic life insurance
• Disability insurance 6
• Employee assistance program
• Business travel insurance
• Tuition assistance
• Relocation
• Backup childcare
• Legal benefits
• Supplemental parental bonding leave
• Surrogacy and adoption assistance
• Fertility support
• Company-funded pension plan 6
• 401 (k) savings plan with company match 6
• Up to 120 vacation hours per year
• Ten paid holidays per year

Company Overview