Senior Data Engineer Essential Duties and Responsibilities:Actively work with business and other BI stakeholders to understand their needs and solution enhancements to the data warehouseBe available for addressing issues or bugs in the present implementationDesign, build, and maintain data infrastructure, ETL processes, and data pipelines.Design, build, and maintain data pipelines using DatabricksConfigure and implement data sourcing events on KafkaBuild and optimize python/Pyspark frameworkMaintain Snowflake and AWS data infrastructureWork with Unstructured/Semi-Structured data and build reporting tablesAssist the BI Analyst to manage client reporting requirementsBuild Data models and assist the BI Analyst to offer data-driven insightsDesign data models and automate manual processes.Create and execute a test strategy to ensure robustness of data pipelinesEnsure effectiveness in infrastructure consumption by implementing solutions that are optimized and scalableFoster an environment that emphasizes trust, open communication, creative thinking, and cohesive team effort, across the business, IT and vendor teamsDeep Expertise in Databricks Spark:Proficient in Databricks Spark: Exceptional skills in Databricks Spark for sophisticated data processing. Proven experience in leveraging Spark for complex ETL tasks, surpassing traditional data processing methods.ETL Pipeline Mastery: Demonstrated excellence in designing and implementing ETL pipelines specifically within Databricks. Ability to utilize Spark's full capabilities to create efficient, scalable data pipelines.Data Transformation and Analysis: Expert in data transformation using Databricks Spark, skilled in performing advanced data analytics and processing large datasets with high efficiency.Optimization Techniques: Deep understanding of optimizing Databricks Spark applications for maximum performance, including fine-tuning Spark configurations, and memory management.Integration with Confluent Kafka:Kafka Exposure: Solid background in working with Confluent Kafka, particularly in integrating it with Spark-based systems for real-time data streaming and processing.Efficient Data Pipelines: Proficiency in creating and managing data pipelines that seamlessly integrate Kafka with Databricks Spark, ensuring efficient data flow and processing.Supporting integrations with business applications, using KafkaDataOps and Agile Methodologies:DataOps Principles: Strong grasp of DataOps methodologies, with a focus on improving the efficiency and quality of data analytics via automation, collaboration, and process optimization.Agile Development: Experienced in Agile software development practices, adept at implementing Agile methodologies like Scrum or Kanban in data-centric projects for improved collaboration and rapid delivery.Proficiency in SQL and Platform Integrations:Advanced SQL Skills: Expertise in SQL, particularly for querying and managing tables/data warehouses within Snowflake. Ability to seamlessly integrate these with Databricks Spark.Platform Adaptability: Skilled in adapting to and integrating various data platforms and technologies, aligning them with strategic organizational goals.Collaborative and Best Practice-Oriented:Adherence to High Standards: Committed to maintaining high standards in code quality, documentation, and adhering to DataOps and Agile best practices.Team Collaboration and Leadership: Ability to work collaboratively in a team, fostering a culture of continuous learning and improvement.Qualifications:Bachelor's or master's degree in computer science, statistics, or analytics.Over 8 years of experience in the field of data engineering with capabilities on working through the entire development lifecycleAbility to work with senior business stakeholders to be able to ascertain data needs out of business opportunities or challengesMinimum of 5 years of cloud experience on any of the major cloud platforms preferably AWSHands-on Knowledge of Data Modeling, Warehousing and Power BI (or equivalent) as a Reporting ToolAdvanced proficiency in SQL with the ability to read and write queries is requiredDemonstrable experience with Databricks, Snowflake, Python, and PySparkPreferably to have experience with Kafka and Power BI
Job Posted by ApplicantPro