Request ID: 101192-1
Title: Senior Data Engineer
Locations: Raleigh/Phoenix/Dallas
Duration: 06+ Months with Possible Extension.
Pay Range: $65-$68/Hour on W2/C2C (All inclusive)
We are seeking a highly skilled Senior Data Engineer with 8 12+ years of experience in designing and building enterprise-scale data platforms. The ideal candidate will have deep expertise in Snowflake, DBT, Python, PySpark, Qlik Replicate, Astronomer Airflow, and CI/CD to develop scalable, secure, and high-performance data pipelines supporting modern analytics and business intelligence initiatives.
The successful candidate will play a key role in architecting data ingestion frameworks, implementing data governance and security, managing schema evolution, and optimizing cloud-based data platforms.
Key Responsibilities
- Design, develop, and maintain scalable, resilient data pipelines using Snowflake features, including Snowpipe, Streams, Tasks, Dynamic Tables, and advanced SQL.
- Build and maintain DBT models with testing, documentation, lineage, and reusable transformations.
- Develop Python and PySpark-based ingestion frameworks for files, APIs, and enterprise data sources.
- Build ingestion pipelines for structured and semi-structured data formats, including CSV, JSON, XML, Excel, fixed-width files, and multi-record layouts.
- Design ingestion solutions for Mainframe VSAM/EBCDIC data sources.
- Implement metadata-driven schema evolution and schema drift detection strategies.
- Configure and manage Qlik Replicate for Change Data Capture (CDC) and database replication from Oracle, SQL Server, and DB2.
- Develop robust, auditable, and recoverable data replication pipelines.
- Implement Snowflake data governance using RBAC, masking policies, and security best practices.
- Apply Protegrity (or similar tokenization platforms) for sensitive data protection.
- Develop and orchestrate workflows using Astronomer Airflow with dependency management, scheduling, retries, and monitoring.
- Integrate data pipelines with GitLab and Azure DevOps CI/CD pipelines.
- Optimize Snowflake performance, compute resources, storage utilization, and query execution.
- Implement monitoring, alerting, logging, and operational dashboards for data pipelines.
- Collaborate with Data Architects, Analysts, Data Scientists, and business stakeholders to deliver reliable data solutions.
Required Skills
- 8 12+ years of experience in Data Engineering.
- Strong expertise in Snowflake architecture and development.
- Hands-on experience with:
- Snowpipe
- Streams
- Tasks
- Dynamic Tables
- Advanced SQL
- Data Masking
- RBAC
- Performance Tuning
- Strong experience with DBT (Data Build Tool).
- Proficiency in Python and PySpark.
- Experience with Qlik Replicate for CDC and database replication.
- Hands-on experience with Astronomer Airflow workflow orchestration.
- Experience implementing schema drift detection and schema evolution.
- Knowledge of GitLab, Git workflows, and CI/CD automation.
- Experience with REST APIs and enterprise data integration.
- Strong understanding of data governance, metadata management, and data quality.
- Excellent analytical, troubleshooting, and problem-solving skills.
- Strong communication and collaboration skills.
Preferred Qualifications
- Experience with Protegrity or similar data security/tokenization platforms.
- Experience working with Oracle, SQL Server, and DB2 databases.
- Knowledge of Azure DevOps and cloud-native data engineering practices.
- Experience with enterprise-scale data lake and data warehouse implementations.
- Exposure to Agile/Scrum development methodologies.
Experience Required
- 8 12+ years of hands-on Data Engineering experience.