Role: Lead AWS Data Engineer
Location: Philadelphia, PA- Onsite
Contract position
Role Description
We are seeking a Lead Data Engineer to design, build, and optimize large-scale data solutions on AWS. This role combines hands-on technical expertise with leadership responsibilities, ensuring adherence to engineering best practices, mentoring team members, and collaborating with architects and product owners to deliver high-quality, secure, and scalable data platforms.
Key Responsibilities
- Architect and maintain data pipelines using AWS native services (Glue, Kinesis, Lambda, S3, Redshift).
- Design and optimize data models on AWS Cloud leveraging Redshift, RDS, and S3.
- Implement ETL/ELT workflows and PySpark jobs for data ingestion, transformation, and storage.
- Operationalize self-service data preparation tools (e.g., Trifacta) on AWS.
- Conduct performance engineering for large-scale data lakes in production environments.
- Participate in design workshops, provide trade-offs and recommendations for solution architecture.
- Mentor engineers on coding best practices, problem-solving, and AWS service utilization.
- Define code review processes, deployment strategies, and ensure compliance with security standards.
- Collaborate with System Architect and Scrum Master to manage dependencies, risks, and blockers.
- Support test strategy, defect resolution, and root cause analysis during warranty periods.
- Maintain documentation in Confluence and ensure team alignment on standards and practices.
Technical Skills & Experience
- 5+ years in designing and deploying big data applications and ETL jobs using PySpark APIs/SparkSQL.
- Strong experience with AWS services across multiple domains:
o Collection: Kinesis, DMS
o Storage: S3, RDS, Redshift, DynamoDB
o Analytics & ML: Glue, EMR, Athena, SageMaker, Bedrock
o Compute: EC2, Lambda, ECS
o Security: IAM, KMS, SSE
- Proficiency in SQL and relational databases (Oracle, SQL Server, Teradata); expert-level query tuning.
- Hands-on experience with Python development, REST APIs (AWS API Gateway, Node.js), and CI/CD pipelines using GitHub.
- Familiarity with file formats (JSON, Parquet, Avro) and Linux/Unix shell scripting.
- Exposure to Docker/Kubernetes, Delta Lake APIs, and data quality frameworks.
- AWS certification (Developer Associate or higher) preferred.
Leadership & Collaboration
- Proven ability to mobilize and motivate teams, set technical direction, and resolve conflicts.
- Experience working in Agile environments, managing dependencies, and delivering under ambiguity.
- Strong communication skills for stakeholder engagement and cross-functional collaboration.
Preferred Skills
- Experience with BI tools (QuickSight, Tableau).
- Knowledge of data governance, security, and compliance standards.
- Familiarity with performance testing and observability tools for data pipelines.
Skills: Digital :AWS services~Digital:PySpark APIs/SparkSQL
Experience Required: 8-10
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.