Design, build, and maintain scalable batch and streaming data pipelines using Apache Spark that ingest, transform, and deliver data from a wide range of internal sources. Develop robust ETL/ELT workflows and orchestrate them with tools such as Airflow, Dagster, or similar, ensuring pipelines are observable, idempotent, and recoverable.
Numbers & Facts
Location
Chandler, AZ
Description
Required Skills & Qualifications (Must-have qualifications that candidates must meet to be considered)
7 years experience in data engineering with a strong track record of building production data pipelines.
Deep hands-on experience with Apache Spark, including writing, debugging, and performance-tuning large-scale jobs.
Strong experience with data modeling, performance optimization, and cost governance.
Expert-level SQL and strong programming skills in Java and/or Python.
Experience with workflow orchestration tools such as Apache Airflow, Dagster, or Prefect.
Experience working in a cloud environment.
Experience with version control (Git), CI/CD pipelines, and software engineering best practices.
Experience with streaming technologies like Kafka.
Prior work experience at client or in client's Industry.
Applicants must be able to work directly for Artech on W2.
Preferred Skills & Qualifications (Nice-to-have skills but are not required)
Experience with microservices architecture.
Experience with cloud implementation, data on cloud, and integration with cloud.
Experience mentoring junior and mid-level engineers.
Day-to-Day Responsibilities (key tasks and expectations for the role)
Design, build, and maintain scalable batch and streaming data pipelines using Apache Spark that ingest, transform, and deliver data from a wide range of internal sources.
Optimize Snowflake/Databricks data warehouse, including schema design, data modeling, warehouse sizing, clustering, and cost management.
Develop robust ETL/ELT workflows and orchestrate them with tools such as Airflow, Dagster, or similar, ensuring pipelines are observable, idempotent, and recoverable.
Tune Spark jobs and Snowflake/Databricks queries for performance and cost, identifying bottlenecks in partitioning, shuffles, joins, and compute usage.
Implement data quality checks, testing, monitoring, and alerting so issues are caught before they reach downstream consumers.
Establish and enforce best practices for data governance, security, access control, and compliance.
Partner with product teams to understand requirements and deliver datasets that are accurate, well-documented, and easy to use.
Contribute to CI/CD, IaC, and deployment practices for the data platform.
Lead technical design discussions, write design documents, review code to raise the quality bar across the team.
Mentor junior and mid-level engineers and help shape the long-term roadmap for our data platform.
For immediate consideration please click APPLY to begin the screening process with Alex.