GCP Cloud/Data Engineer

Location: Charlotte, NC - Hybrid

Responsibilities

• Platform Management: Deploy, configure, and maintain OpenShift clusters or GCP projects to support containerized Spark Applications for both GPU and CPU Compute.
• Data Pipeline Development: Design and implement large-scale data processing workflows using Apache Spark.
• Optimization: Tune Spark jobs for performance, leveraging OpenShift's resource management capabilities (e.g., Kubernetes orchestration, auto-scaling).
• Integration: Integrate Spark with other data sources (e.g., Kafka, S3, cloud storage) and sinks (e.g., databases, data lakes).
• CI/CD Implementation: Build and maintain CI/CD pipelines for deploying Spark applications in OpenShift or GCP using tools like GitHub Actions, Sonar, Harness.
• Monitoring & Troubleshooting: Monitor cluster health, Spark job performance, and resource utilization using OpenShift tools (e.g., Prometheus, Grafana) and resolve issues proactively.
• Security: Ensure compliance with security standards, implementing role-based access control (RBAC) and encryption for data in transit and at rest.
• Collaboration: Work with cross-functional teams to define requirements, architect solutions, and support production deployments.

Qualifications

• Experience:
  • 5 years working on Apache Spark for big data processing.
  • 3 years of Django development experience.
  • 2 years of creating and maintaining conda environments.
  • 4 years managing containerized environments with OpenShift or Kubernetes.
  • 2 years managing GPU-based AIML applications.
• Technical Skills:
  • Proficiency in Spark frameworks (Python/PySpark, Scala, or Java).
  • Hands-on experience with OpenShift administration (e.g., cluster setup, networking, storage).
  • Proficiency in creating and maintaining conda environments and dependencies.
  • Familiarity with Docker and Kubernetes concepts (e.g., pods, deployments, services, and images).
  • Knowledge of distributed systems, cloud platforms (AWS, GCP, Azure), and data storage solutions (e.g., S3, HDFS).
  • Programming: Strong coding skills in Python, Scala, or Java; experience with shell scripting is a plus.
  • Tools: Experience with Git Actions, Helm, Harness, and CI/CD tools.
  • Problem-Solving: Ability to debug complex issues across distributed systems and optimize resource usage.
  • Education: Bachelor's degree in Computer Science, Engineering, or related field.