Data Engineer (Microsoft Fabric and Databricks)

Job Title: Data Engineer (Microsoft Fabric and Databricks)
Location: Reston, VA 20190
Duration: 12 Months
Job Type: Temporary Assignment
Work Type: Onsite

Job Summary:

• The Senior Data Engineer is responsible for the design, development, administration, governance, and operational support of enterprise data platforms built on Microsoft Fabric.
• This role combines deep data engineering expertise with platform administration capabilities to ensure scalable, secure, reliable, and cost-effective data solutions that support business operations, analytics, AI initiatives, and project delivery.
• The position requires 10-15 years of overall relevant experience in data engineering, data platforms, cloud technologies, and enterprise data management.
• The role will design, build, and support data ingestion, ETL/ELT pipelines, lakehouse architectures, and analytical datasets using Microsoft Fabric, Databricks, Python, PySpark, SQL, Spark, Delta Lake, OneLake, and Data Pipelines.
• In addition, the role will govern Databricks and Fabric environments, including security, access management, monitoring, performance tuning, capacity management, and platform operations.
• The Senior Data Engineer will collaborate with business stakeholders, architects, and analytics teams to deliver robust data solutions while ensuring platform reliability, compliance, security, and operational excellence.
• This role will also drive continuous improvement in data quality, governance, automation, performance, and cost optimization.

Major Responsibilities:

Data Engineering & Solution Delivery:

• Own end-to-end technical design, development, and delivery of enterprise data engineering solutions.
• Design and implement scalable ETL/ELT, data integration, and data transformation pipelines for structured and unstructured data sources.
• Develop and maintain batch, near real-time, and streaming data ingestion pipelines from databases, APIs, files, cloud storage, and enterprise applications.
• Build reusable notebooks, data transformations, and processing frameworks using Databricks, Microsoft Fabric, Python, PySpark, and SQL.
• Design and implement Lakehouse architectures using Delta Lake, OneLake, and medallion architecture (Bronze, Silver, Gold layers).
• Create trusted datasets, curated data products, and semantic-ready data structures supporting analytics, reporting, AI, and machine learning initiatives.
• Implement data quality frameworks, validation rules, reconciliation controls, and exception management processes.
• Design scalable data models, dimensional schemas, and data marts to optimize reporting and analytical workloads.
• Optimize Spark workloads, SQL queries, storage strategies, partitioning, and data processing performance.

Databricks & Microsoft Fabric Administration:

• Administer Databricks workspaces, Fabric capacities, Fabric workspaces, Lakehouses, Data Warehouses, Dataflows, and Pipelines.
• Configure and maintain user access, security groups, RBAC permissions, service principals, and identity integrations.
• Manage secrets, credentials, key vault integrations, and secure access to enterprise data sources.
• Monitor platform health, resource utilization, workload performance, and capacity consumption.
• Perform platform upgrades, feature enablement, release validation, and environment maintenance activities.
• Implement capacity planning, workload management, and resource allocation strategies.
• Support governance initiatives by enforcing workspace standards, naming conventions, tagging, and compliance controls.

Operations, Governance & Reliability:

• Develop operational runbooks, support procedures, and disaster recovery documentation.
• Monitor data pipelines and platform services to ensure availability and reliability.
• Establish and maintain metadata management, data lineage, cataloging, and governance processes.
• Implement observability, logging, monitoring, and alerting across the data platform ecosystem.
• Drive cost optimization initiatives across compute, storage, Fabric capacities, and Databricks workloads.
• Establish Git-based source control and branching strategies for notebooks, pipelines, and infrastructure components.
• Implement CI/CD pipelines for automated deployment, testing, environment promotion, and release management.
• Develop reusable deployment templates and engineering accelerators for platform consistency and scalability.
• Communicate technical solutions, trade-offs, risks, dependencies, and delivery plans to both technical and non-technical stakeholders.

Education and Experience Requirements:

• B.Tech in Computer Science, Information Technology, Engineering, or an equivalent degree is required.
• 10-15 years of overall relevant experience in data engineering, data platforms, analytics, and cloud technologies.
• Minimum 5+ years of hands-on experience with Databricks and 2+ years Microsoft Fabric environments.
• Proven experience designing and supporting production-grade ETL/ELT and data integration solutions.
• Experience administering enterprise-scale Databricks and Microsoft Fabric platforms.
• Strong experience working with Azure data services and cloud-native architectures.
• Demonstrated experience in stakeholder engagement, solution ownership, and technical leadership.
• Experience mentoring engineers and supporting enterprise platform operations is preferred.

Required Knowledge and Skills:

Data Engineering:

• Strong expertise in Databricks, Spark, Delta Lake, Microsoft Fabric, OneLake, Lakehouse, and Data Warehouse architectures.
• Advanced proficiency in Python, PySpark, SQL, and Notebook-based development.
• Expertise in ETL/ELT frameworks, data ingestion patterns, incremental processing, CDC, and data transformation techniques.
• Strong understanding of dimensional modeling, data warehousing, and semantic data structures.
• Experience with batch processing, streaming architectures, and real-time data integration.

Platform Administration:

• Strong knowledge of Databricks Workspace Administration and Microsoft Fabric Administration.
• Experience managing capacities, workspaces, governance controls, security policies, and platform operations.
• Expertise in Microsoft Entra ID (Azure AD), RBAC, access controls, service principals, and security integration.
• Experience with secrets management, key vaults, credential management, and secure data access.
• Knowledge of platform monitoring, observability, troubleshooting, incident management, and operational excellence.

Governance & DevOps:

• Strong understanding of data governance, metadata management, lineage, data cataloging, and compliance controls.
• Experience with Git, Azure DevOps, CI/CD pipelines, release management, and automated testing.
• Knowledge of Infrastructure as Code, automation frameworks, and deployment standards.
• Experience implementing enterprise data quality and reliability frameworks.

Performance & Cost Optimization:

• Experience tuning Spark applications, SQL workloads, storage structures, and distributed compute environments.
• Knowledge of Fabric capacity optimization, Databricks cluster optimization, workload management, and FinOps practices.
• Ability to balance performance, scalability, reliability, security, and cost considerations.
• Ability to work effectively in a fast-paced enterprise environment supporting multiple business priorities.

Preferred Certifications:

• Microsoft Certified: Fabric Analytics Engineer Associate
• Microsoft Certified: Azure Data Engineer Associate
• Databricks Certified Data Engineer Professional
• Databricks Certified Data Engineer Associate