Sr Data Engineer

Job Title: Sr Data Engineer

Location: Mclean, VA 5 days onsite

Duration: Contract

Role Summary

• Lead the design and development of cloud-native data pipelines on AWS and Snowflake.
• Drive Spark-on-EKS, advanced Snowflake capabilities, and AI/Agentic AI proofs of concept (PoCs).
• Mentor mid-level developers and collaborate with stakeholders.

Key Responsibilities

• Architect end-to-end pipelines using AWS, Snowflake, IICS, and PySpark.
• Design and deploy containerized PySpark workloads on EKS.
• Build and tune EMR clusters on EC2 or EKS for large-scale processing.
• Orchestrate workflows using AWS Step Functions and Control-M.
• Manage metadata using AWS Glue Data Catalog and integrate it with Snowflake Iceberg.
• Use DynamoDB for pipeline state, checkpoints, and configuration.
• Lead implementation of Snowflake capabilities, including Iceberg Tables, Dynamic Tables, Views, Materialized Views, Tasks and Streams, Data Governance, Horizon Catalog, Semantic Views, Cortex (Cortex AI), Snowflake Agents, and token-usage optimization strategies.
• Build RAG and Agentic AI PoCs, covering vector search, BM25 hybrid retrieval, evaluation frameworks, human-in-the-loop controls, and security controls.
• Develop PySpark and Snowpark pipelines for ingestion and transformation.
• Lead IICS file-based ingestion design.
• Conduct code reviews, improve performance, and mentor engineers.
• Leadership and Collaboration Expectations
• Demonstrates ownership and accountability for design quality, delivery commitments, and production outcomes.
• Communicates complex technical decisions clearly to engineering teams, product partners, and stakeholders.
• Collaborates across teams, constructively challenges assumptions, and drives alignment on scalable solutions.
• Provides practical coaching and feedback that strengthens engineering capability and promotes knowledge sharing.
• Applies sound judgment, balances speed with engineering rigor, and proactively manages risk and dependencies.
• Shows curiosity and adaptability when evaluating emerging cloud, data, and AI capabilities.

Skills Required

• Strong hands-on experience with AWS data services, including EKS, EMR, EC2, Step Functions, DynamoDB, AWS Glue Data Catalog, and S3.
• Advanced Snowflake expertise.
• Expertise in PySpark and Snowpark.
• Hands-on experience with IICS for file-based ETL ingestion.
• Experience building RAG and Agentic AI PoCs.
• Strong SQL and data modeling skills.
• Leadership and mentoring capability.
• Knowledge of Control-M orchestration.

Must-Have Skills

• AWS: EKS, EMR, EC2, Step Functions, DynamoDB, AWS Glue Data Catalog, S3, and IAM.
• Snowflake: Iceberg, Dynamic Tables, Views, Materialized Views, Tasks, Governance, Horizon, Cortex, and Agents.
• PySpark: PySpark on EKS, including containerized Spark.
• Snowpark for Python.
• IICS for file-based ETL ingestion.
• Strong SQL and data modeling.
• Experience with RAG or Agentic AI PoCs.
• Awareness of reporting tools such as Power BI or Tableau.
• Control-M scheduling.

anky@purplehires.com