Databricks Engineer

Regulated Position: Candidates may need to meet specific work authorization requirements if the project falls under ITAR, NERC CIP, NRC, or other applicable regulations. Strong experience developing data pipelines using Azure Data Factory (ADF) .

Role: Databricks Engineer
Duration: 6 months
Experience Required: 8–10 years
Regulated Position: Candidates may need to meet specific work authorization requirements if the project falls under ITAR, NERC CIP, NRC, or other applicable regulations.

Must-Have Technical/Functional Skills

• Strong experience developing data pipelines using Azure Data Factory (ADF).
• Experience ingesting data from multiple sources using Azure Data Factory.
• Hands-on experience transforming data using Azure Databricks.
• Strong knowledge of Delta Lake for data transformation and management.
• Experience with Azure DevOps CI/CD.
• Experience managing Stories, Bugs, and Issues in Azure DevOps.
• Experience implementing and working with Prefect for workflow orchestration.
• Strong SQL skills with the ability to write complex queries.
• Experience with database querying, data analysis, joins, and duplicate-record identification.
• Ability to troubleshoot and resolve data-related issues.

Role Descriptions

• Design and implement data integration solutions using Microsoft Azure.
• Develop scalable and reliable data pipelines using Azure Data Factory.
• Ingest and process data from various sources.
• Transform and manage data using Azure Databricks and Delta Lake.
• Implement workflow orchestration using Prefect.
• Perform detailed data analysis to troubleshoot data-related issues.
• Assist in identifying, analyzing, and resolving data quality issues.
• Work effectively as an individual contributor and problem solver.
• Support end-to-end data engineering and integration activities.

Roles & Responsibilities

• Work in Agile development environments.
• Participate in sprint planning, development, testing, and sprint delivery.
• Collaborate with cross-functional teams to deliver data engineering solutions.
• Implement integration solutions within the Microsoft Azure ecosystem.
• Analyze data to identify root causes of data-related problems.
• Develop and optimize SQL queries for data validation and analysis.
• Identify duplicate records and data inconsistencies.
• Troubleshoot pipeline and data processing failures.
• Contribute to continuous improvement of data engineering processes.

Generic Managerial Skills

• Experience working in Agile development projects.
• Experience participating in and delivering sprints.
• Strong communication and collaboration skills.
• Ability to work independently as an individual contributor.
• Strong analytical and problem-solving skills.

Key Skills / Keywords to Search in Resume

• Azure Databricks
• Azure Data Factory (ADF)
• Delta Lake
• Prefect
• Azure DevOps
• CI/CD
• Data Pipelines
• Data Ingestion
• Data Transformation
• Data Integration
• SQL
• Database Querying
• Data Analysis
• Data Quality
• Agile
• Sprint Delivery

Regulated Positions

• ITAR (International Traffic in Arms Regulations): May require U.S. Citizens or U.S. Persons.
• NERC CIP (NERC Critical Infrastructure Protection): May impose specific personnel and access requirements.
• NRC (Nuclear Regulatory Commission): May impose specific work authorization or security requirements.
• Other Applicable Regulations: Additional work authorization requirements may apply depending on the project and governing regulations.

Education

• Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field.
• Equivalent professional experience may be considered.

Certifications

• Optional.
• Azure, Databricks, or related data engineering certifications are desirable.

Desirable Skills

• Experience with additional Azure data services.
• Experience building enterprise-scale data integration solutions.
• Knowledge of data lakehouse architectures.
• Experience with data quality and validation frameworks.
• Experience optimizing Databricks workloads and SQL queries.
• Knowledge of CI/CD best practices for data engineering projects.
• Experience with workflow orchestration and scheduling tools.

Essential Skills

• Databricks
• Azure Data Factory
• Delta Lake
• Azure
• SQL
• Data Pipelines
• Data Integration
• Prefect
• Azure DevOps
• CI/CD
• Agile Development

Keyword

• Azure Databricks

Skills

• Digital: Databricks