The ideal candidate will have extensive experience with data integration, processing, and management, specifically utilizing Apache Nifi, RESTful APIs, Hadoop, Accumulo, and Elasticsearch.
Key Responsibilities:
- Design, develop, and maintain data pipelines to support C2IE deployments.
- Utilize Apache NiFi for data ingestion, transformation, and routing.
- Develop and maintain RESTful APIs for seamless data exchange between systems.
- Implement and manage data storage solutions, including Hadoop, Accumulo, and Elasticsearch.
- Collaborate with cross-functional teams to understand data requirements and implement effective solutions.
- Ensure data quality, integrity, and security throughout the data lifecycle.
- Monitor and optimize data workflows to ensure high performance and reliability.
- Troubleshoot and resolve issues related to data integration and processing.
- Stay updated with the latest trends and technologies in data engineering and apply them to improve existing processes.
Required Skills and Qualifications:
- Bachelor's degree in Computer Science, Information Technology, or related field.
- Proven experience as a Data Engineer or similar role, with a focus on supporting complex data environments.
- Strong proficiency in Apache NiFi for data integration and processing.
- Experience in developing and managing RESTful APIs.
- In-depth knowledge of data storage solutions, including Hadoop, Accumulo, and Elasticsearch.
- Proficiency in programming languages such as Python, Java, or Scala.
- Strong understanding of data security and compliance requirements.
- Excellent problem-solving skills and attention to detail.
- Strong communication and collaboration skills.
Preferred Qualifications:
- Experience with Elastic Search and the Elastic Stack
- Experience with cloud platforms (e.g., AWS, Azure, Google Cloud) for data engineering.
- Knowledge of DevOps practices and tools like Jenkins, Docker, and Kubernetes.
- Experience with data visualization tools and techniques.
- Relevant certifications in data engineering or related fields.