Senior Databricks Engineer- Irving, TX

Overview

We are seeking a Senior Databricks Engineer to support a Deloitte project team in a data modernization effort. This role requires expertise in PySpark/Spark SQL to refactor legacy SAS/SQL code while developing and configuring ETL and DevOps pipelines for Azure Databricks environments.

This position is on-site in Irving, TX, four days per week and requires candidates to be legally authorized to work in the U.S. without employer sponsorship.

Key Responsibilities:
  • Refactor Legacy Code – Interpret and refactor SAS/SQL scripts with complex joins and data transformations across diverse data sources.
  • Data Ingestion & Transformation – Develop PySpark frameworks for ingesting data from multiple sources (CSV, Excel, on-prem servers) using lakehouse federation.
  • Azure Infrastructure Development – Create and manage storage containers, compute resources, and Azure pipelines for Databricks environments.
  • Security & Compliance – Guide infrastructure teams in configuring Unity Catalog/Schemas, workspaces, clusters, and storage containers to meet project requirements.
  • CI/CD Implementation – Develop CI/CD pipelines using Databricks asset bundles and Azure DevOps for streamlined deployment and automation.
Required Skills:
  • 2-5 years of experience developing ETL pipelines using Databricks, preferably in Azure Databricks.
  • Strong programming skills in PySpark, Python, and SQL.
  • Deep understanding of data structures and transformations.
  • Strong grasp of Azure Databricks architecture.
Preferred Skills:
  • Databricks certifications, such as Databricks Engineering Professional or Apache Spark certifications.