Introduction
As a key figure in our data engineering team, you will play a critical role in designing and implementing scalable data pipelines and architectures. You will leverage your expertise in Python and PySpark to enhance our data processing capabilities, with a particular focus on migrating data from Snowflake to Databricks. This position requires a strong background in big data technologies and excellent leadership skills.
Required Skills & Qualifications
- Minimum of 8 years of experience in data engineering, with a strong focus on Databricks and big data technologies.
- Proficient in Python and PySpark, with a proven track record of building data pipelines and ETL processes.
- Experience with Snowflake and a demonstrated ability to migrate data solutions to Databricks.
- Strong understanding of data modeling, data warehousing concepts, and distributed computing.
- Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and their data services.
- Excellent problem-solving skills and the ability to work in a fast-paced environment.
- Strong communication and interpersonal skills, with the ability to collaborate effectively with technical and non-technical stakeholders.
- Bachelor’s degree in Computer Science, Engineering, or a related field; advanced degree preferred.
- Prior work experience at client or in client's Industry.
- Applicants must be able to work directly for the company on W2.
- Previous experience working with large datasets in a high-volume environment.
- Knowledge of machine learning frameworks and analytics tools is a plus.
- Experience in Agile methodologies and DevOps practices.
Day-to-Day Responsibilities
- Lead the design, development, and optimization of data pipelines and ETL processes in Databricks.
- Collaborate with cross-functional teams to gather requirements and translate them into robust data solutions.
- Migrate existing data solutions from Snowflake to Databricks, ensuring minimal disruption and high data integrity.
- Utilize Python and PySpark to build and maintain scalable data processing frameworks.
- Implement best practices for data governance, security, and performance optimization.
- Mentor and guide junior data engineers, fostering a culture of continuous learning and improvement.
- Monitor system performance and troubleshoot issues to ensure optimal data flow and availability.
- Stay updated with industry trends and emerging technologies related to data engineering and analytics.
Company Benefits & Culture
- Inclusive and diverse work environment.
- Opportunities for professional growth and development.
- Flexible work arrangements to support work-life balance.
For immediate consideration please click APPLY to begin the screening process with Alex.