Senior Data Engineer

Senior Data Engineer

The Senior Data Engineer is a highly experienced data professional responsible for leading the design, development, and optimization of complex data pipelines and platforms that support enterprise analytics, reporting, and advanced data use cases. This role serves asa technical leader within the data engineering team, owning moderately large initiatives, guiding architectural decisions, and mentoring Data Engineers.

Reporting to the Director of Data Intelligence and Decision Science (or a designated senior leader), the Senior Data Engineer partners closely with data scientists, analysts, software engineers, informaticists, and business stakeholders. The role ensures scalable, secure, and high-quality data solutions while supporting organizational priorities in clinical, operational, financial, and research domains. The Senior Data Engineer plays a key role in preparing the organization for advanced analytics, automation, and AI/ML adoption, without holding full enterprise-wide ownership reserved for the Principal Data Engineer.

Responsibilities

Leads Design and Optimization of Data Pipelines

• Designs, builds, and maintains complex, scalable ETL/ELT pipelines for structured and unstructured data.
• Leads integration of data from EHRs, financial systems, registries, and external data sources.
• Optimizes pipelines for performance, reliability, fault tolerance, and cost efficiency.
• Implements batch and near–real-time data processing patterns as needed.
• Ensures pipelines meet regulatory, privacy, and security requirements (e.g., HIPAA).

Owns Key Data Platforms and Architecture Components

• Serves as technical owner for specific data platforms, domains, or subject areas (e.g., clinical analytics, operational reporting).
• Designs and maintains data lake, warehouse, and data mart structures using cloud platforms
• Develops and enforces data modeling standards, schema design, and partitioning strategies.
• Partners with IT and cloud teams to ensure availability, scalability, and disaster recovery readiness.

Enables Advanced Analytics and Data Science

• Builds curated, analytics-ready datasets and reusable data assets for analysts and data scientists.
• Collaborates with data science teams to support feature engineering, model training, and deployment workflows.
• Develops frameworks and patterns that improve self-service analytics and reduce ad hoc data requests.
• Supports experimentation and proof-of-concept work for predictive analytics and AI/ML use cases.

Drives Process Improvement and Engineering Best Practices

• Leads initiatives to improve data engineering workflows, including automation, monitoring, and CI/CD for data pipelines.
• Refactors legacy pipelines and infrastructure to improve maintainability and scalability.
• Establishes best practices for code quality, documentation, testing, and version control.
• Evaluates new tools and technologies and recommends adoption where appropriate.

Mentors and Provides Technical Leadership

• Serves as a technical mentor to Data Engineer staff.
• Reviews code, pipeline designs, and architecture artifacts to ensure quality and consistency.
• Provides guidance on complex technical problems and helps unblock team members.
• Contributes to onboarding, internal training, and knowledge-sharing activities.

Collaborates with Stakeholders and Leads Medium-to-Large Initiatives

• Partners with business, clinical, research, and operational stakeholders to translate requirements into technical solutions.
• Leads data engineering workstreams within cross-functional projects or agile squads.
• Communicates technical concepts, trade-offs, and risks to non-technical audiences.
• Supports planning, estimation, and prioritization of data engineering initiatives.

Knowledge / Skills / Abilities

Technical Expertise

• Advanced proficiency in SQL and Python and related languages for data engineering.
• Strong experience with distributed data processing frameworks (e.g., Spark).
• Hands-on expertise with workflow orchestration tools (e.g., Airflow).
• Deep familiarity with cloud-based data platforms and services (AWS, GCP, or Azure/Fabric).
• Experience designing and optimizing data models for analytics and reporting.

Data Governance and Compliance

• Strong understanding of data governance, data quality, and security best practices.
• Experience working with regulated data, particularly healthcare or clinical data.
• Familiarity with healthcare data standards (e.g., HL7, FHIR) preferred.

Problem Solving and Decision Making

• Analyzes complex systems to identify root causes and scalable solutions.
• Balances short-term delivery with long-term architectural sustainability.
• Makes sound technical decisions with limited ambiguity.

Collaboration and Leadership

• Effectively collaborates across technical and non-technical teams.
• Provides constructive feedback and technical guidance to peers.
• Demonstrates ownership, accountability, and initiative.

Required Qualifications

• Bachelor's Degree in Computer Science, Data Engineering, Information Systems, or a related field.
• At least 6 years of experience in data engineering, analytics engineering, or data platform development.
• Demonstrated experience designing and leading complex data pipelines and data platforms.
• Relevant education and experience may be substituted as appropriate.

Preferred Qualifications

• Master's Degree in Data Science, Data Engineering Computer Science, Informatics, or related field.
• Experience in healthcare data engineering or regulated data environments.
• Exposure to AI/ML infrastructure, feature stores, or model operationalization.
• Experience leading technical initiatives or acting as a team lead.

Licenses, Registrations or Certifications

• REQUIRED: None
• PREFERRED: Cloud Certification
• Microsoft Certified: Azure Data Engineer Associate
• Google Cloud Professional Data Engineer
• AWS Certified Data Analytics – Specialty

Salary Range

$138,000+ depending on qualifications

Working Conditions

• Standard office equipment
• Repetitive use of a keyboard
• May be exposed to healthcare-related occupational hazards depending on assignment

Required Materials

• Resume/CV
• 3 work references with their contact information; at least one reference should be from a supervisor
• Letter of interest