Airflow Architect - Big Data Systems
Austin, TX
Gross $135K including $5K variable
Contracting $60/hr,
Job Summary
Need an experienced Airflow Architect to design, architect, and lead enterprise-scale data orchestration solutions for modern big data platforms. The ideal candidate will have deep expertise in Apache Airflow , Big Data technologies , cloud-native architectures , and data engineering best practices to build scalable, reliable, and high-performance data pipelines.
Key Responsibilities
- Architect and implement large-scale data orchestration frameworks using Apache Airflow.
- Design, develop, and Eligible to workimize ETL/ELT pipelines for batch and near real-time processing.
- Build scalable workflows integrating Hadoop, Spark, Kafka, Databricks, data lakes, and data warehouses.
- Define Airflow deployment strategies, DAG standards, security, monitoring, and governance.
- Implement workflow scheduling, dependency management, error handling, retries, alerting, and SLA monitoring.
- Lead cloud-based data platform implementations on AWS
- Eligible to workimize pipeline performance, scalability, reliability, and cost efficiency.
- Establish CI/CD processes and DevOps practices for Airflow environments.
- Collaborate with business stakeholders, data engineers, architects, and platform teams to translate requirements into technical solutions.
- Mentor engineering teams on Airflow architecture, data orchestration patterns, and best practices.
Required Skills
Airflow & Data Engineering
- Expert knowledge of Apache Airflow DAG design, scheduling, operators, sensors, and executors.
- Strong experience in workflow orchestration, metadata management, and pipeline monitoring.
- Hands-on expertise in ETL/ELT development and data pipeline automation.
Big Data Technologies
- Apache Spark / PySpark
- Hadoop ecosystem
- Apache Kafka
- Data Warehousing concepts
- Data Lakes and Lakehouse architectures
- NoSQL databases
Programming
- Python (advanced)
- SQL (advanced)
- Shell scripting
- Core Java (preferred)
Cloud Platforms
- AWS (EMR, Glue, S3, Redshift)
Preferred Qualifications
- 10+ years of experience in data engineering and big data platforms.
- 3+ years of architecture experience designing enterprise data solutions.
- Experience with Kubernetes, Docker, and cloud-native deployments.
- Knowledge of Data Mesh, Data Fabric, and Modern Data Platforms.
- Experience leading architecture reviews and technology strategy discussions.
Success Metrics
- Pipeline reliability and SLA adherence
- Workflow performance Eligible to workimization
- Reduced operational incidents
- Improved data quality and observability
- Scalable and reusable orchestration frameworks
Disclaimer: Diverse Lynx LLC is an Equal Opportunity Employer. All applicants and employees are evaluated without discrimination, based solely on their qualifications, ability, competence and performance. This email and its attachments may contain confidential or proprietary information and is intended only for the recipient(s). If you received this message in error, please disregard it and notify the sender. If you no longer wish to receive our communications, you may unsubscribe at any time.
Security Notice: Our official website is www.diverselynx.com We do not operate or authorize any other websites representing Diverse Lynx LLC.