Job Title: Database engineer
Location: Louisville KY 40202 (Remote)
Hours: Monday - Friday, 8am-5pm with limited OT
Duration: 4 Months with extension on W2
Summary of Duties & Job Description
We are looking for a Data Engineer to work with our data science teams to collect, store, process, and analyze huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them. You will also be responsible for integrating them with the architecture used across teams in our group.
• Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities.
• Implementing ETL process and data pipelines.
• Monitoring performance and advising any necessary infrastructure changes.
• Collaborating with Data Scientists to build Machine Learning systems and run experiments.
Skills and Qualifications
• Strong algorithm and data structures knowledge.`
• Knowledge of various databases (RDBMS, noSQL, HDFS, Cassandra, Redis).
• Experience building cloud data lakes and data warehouses is highly desirable.
• Experience with security and authentication in cloud platforms (Azure preferred).
• Bachelor’s Degree or above in a technical field (Computer Science preferred).
• 5+ years of experience in data engineering.
• Experience building stream-processing systems, using solutions such as Spark-Streaming or Flink.
• Familiar with Spark ecosystem (e.g. Databricks).
• Good knowledge of Big Data querying tools.
• Understanding of Data Catalog, Data Governance, Data Lineage.
• Knowledge of various ETL techniques and data pipelines.
• Knowledge of messaging systems, such as Kafka or RabbitMQ.
• Familiarity with Machine Learning, Deep Learning and Natural Language Processing. workflows and libraries (scikit-learn, Spark MLlib, TensorFlow/Keras/PyTorch).
• Running Machine Learning tests and experiments.