Data Engineer III
Remote- CST/EST Hrs.
4-month contract
Pay Rate: $70.00 - $76.00per hr.
Our client is seeking a Data Engineer III who can work independently with minimal supervision and build complex data ingestion pipelines. This role will partner closely with the Digital team to develop ingestion solutions that move data into Snowflake, while following established SDLC, CI/CD, QA/QC, and data security standards.
Responsibilities
- Partner with the Digital team to design, develop, and maintain data ingestion pipelines into Snowflake.
- Build pipelines supporting Kafka, relational databases, mainframe flat files, and other data sources.
- Develop ETL processes using Qlik or comparable data integration technologies.
- Implement data obfuscation within the ETL process to protect sensitive information.
- Move processed data through S3 into Snowflake.
- Load data into appropriate DDL/schema structures using Spark within Snowflake.
- Develop controls to validate data ingestion and quality.
- Monitor streaming data, including Kafka topics, to ensure data is loading and processing correctly.
- Troubleshoot pipeline, data-quality, and ingestion issues.
- Perform QA/QC throughout the development lifecycle.
- Follow established software development, CI/CD, and SDLC processes.
- Document development and technical changes.
Required Skills
- 8 years of professional experience
- Senior-level Data Engineering experience.
- Strong software development and SDLC knowledge.
- Ability to work independently with minimal supervision.
- Git and experience working collaboratively within a shared repository.
- CI/CD experience.
- Strong data ingestion and pipeline development experience.
- ETL tool experience required.
- Specific ETL platform is not required.
- Experience with tools such as Databricks or similar platforms is transferable.
- Qlik Replicate is not required and can be learned on the job.
- Experience working with multiple data sources, including:
- Kafka
- Relational databases/views
- Mainframe flat files
- Change Data Capture (CDC)
- AWS Lambda
- AWS S3
- Snowflake
- Snowflake stored procedures (Sprocs)
- Spark within Snowflake
- User-defined functions (UDFs)
- Data obfuscation and protection of highly sensitive data.
Preferred Background
- Experience in banking, financial services, or another regulated environment is helpful but not required. Experience working with complex, sensitive, or highly regulated data is beneficial.
Frontier AI / Data Security
- This role supports Frontier AI applications and requires handling highly sensitive data. The engineer will be responsible for implementing appropriate data obfuscation and ensuring sensitive data is protected throughout the ingestion process.
Key Data Flow
- Data Sources ETL/Qlik Obfuscation S3 Snowflake DDL/Schema Data Controls