Senior Data Engineer –Plano, TX | Full-Time / HybridPosition OverviewWe are seeking a highly skilled and experienced
Senior Data Engineer to help design, develop, and optimize a modern cloud-based data platform.This role will focus heavily on
Azure Databricks, PySpark, Spark SQL, scalable ETL/ELT pipelines, and data lakehouse architecture. The ideal candidate will have experience working in high-volume data environments and collaborating with data architects, data scientists, business stakeholders, and cloud infrastructure teams to deliver reliable, secure, and high-performance data solutions.
Key ResponsibilitiesData Architecture & Engineering- Design and implement scalable ETL/ELT pipelines using Azure Databricks, PySpark, and Spark SQL.
- Build robust data ingestion workflows from diverse sources, including APIs, event-streaming platforms, relational databases, and cloud storage.
- Develop and maintain Bronze, Silver, and Gold data layers using a modern medallion architecture.
- Establish best practices for Delta Lake, schema evolution, data versioning, and governance.
- Optimize large-scale data pipelines for performance, reliability, and cost efficiency through partitioning, caching, clustering, and Spark configuration tuning.
Cloud Data Platform- Integrate Databricks with Azure cloud services, including cloud data lakes, data integration services, event-streaming platforms, data warehouses, secrets management, and CI/CD tooling.
- Implement secure and governed access to enterprise data using managed identities, role-based access controls, and data protection techniques.
- Support the development and evolution of a scalable cloud-native data platform.
Collaboration & Technical Leadership- Partner with data analysts, data scientists, business stakeholders, and other technical teams to understand data requirements.
- Build trusted datasets, reporting structures, and business-critical KPIs.
- Provide technical leadership and mentorship to junior and mid-level engineers.
- Conduct code reviews and contribute to architectural and engineering standards.
- Collaborate with architecture, security, and infrastructure teams to ensure data solutions meet organizational security and compliance requirements.
Data Quality & Monitoring- Implement data quality checks, validation frameworks, alerts, and lineage tracking.
- Develop monitoring and observability solutions for mission-critical data pipelines.
- Establish automated failure detection, alerting, and recovery processes.
- Continuously improve data reliability, scalability, and operational performance.
Required Qualifications- Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or a related technical field.
- 5+ years of professional data engineering experience, including 3+ years working extensively with Azure Databricks and Apache Spark.
- Strong expertise with Delta Lake, Unity Catalog, Databricks Notebooks, and SQL-based data warehousing.
- Hands-on experience with Azure data services and cloud infrastructure.
- Strong programming skills in Python and SQL;familiarity with Scala is a plus.
- Solid understanding of data modeling, including OLTP/OLAP environments and star/snowflake schemas.
- Experience with performance optimization and tuning of large-scale data workloads.
- Experience implementing CI/CD for data engineering environments using tools such as Azure DevOps, Terraform, or Databricks CLI.
- Strong understanding of data security, governance, and access-control practices.
Preferred Qualifications- Databricks certification or comparable professional certification.
- Experience building real-time or streaming data pipelines using technologies such as Structured Streaming, Event Hubs, or Kafka.
- Experience supporting machine learning workflows or ML platforms.
- Familiarity with business intelligence and visualization platforms.
- Experience with data cataloging, lineage, and enterprise governance tools.
- Knowledge of data privacy, security, and compliance frameworks.
- Experience working in high-volume, production data environments.
Work Arrangement- Location: Plano, TX
- Schedule: Full-Time
- Work Arrangement: Hybrid — 4 days per week on-site
- Remote: 1 day per week
- Relocation Assistance: Not Provided
- Work Authorization: Candidates must be authorized to work in the United States. Sponsorship is not available for this position.
- Local Requirement: Candidates must be able to commute to the Plano/Dallas–Fort Worth area and work on-site four days per week.
Compensation & BenefitsThe organization offers competitive compensation and a comprehensive benefits package. Additional details regarding compensation and benefits will be discussed during the interview process.