Sign upLog in
Information Technology and Computer Jobs

Data Architect - Palo Alto, CA (Hybrid)

Georgia Tek Systems

  • Palo Alto, CA
  • 30+ days ago

    Highlights

    Task Management: Develop and maintain distributed task processing using Celery and ensure robust task queue management with Redis or RabbitMQ. Data Orchestration: Design, implement, and manage data workflows using Airflow to automate and orchestrate data processing tasks.

    Numbers & Facts

    LocationPalo Alto, CA

    Description

    Data Architect
    Location: Palo Alto, CA (Hybrid)
    Duration: Long Term
    Rate: DOE

    Key Responsibilities:

    Data Orchestration:
    • Design, implement, and manage data workflows using Airflow to automate and orchestrate data processing tasks.
    • Optimize Airflow DAGs (Directed Acyclic Graphs) for performance and scalability.
    Task Management:
    • Develop and maintain distributed task processing using Celery and ensure robust task queue management with Redis or RabbitMQ.
    Database Management:
    • Design and manage databases using Cosmos DB, MongoDB, and PostgreSQL.
    • Develop and maintain efficient data models and ensure data consistency and integrity.
    API and Webhooks:
    • Implement and manage FastAPI webhooks to handle data ingestion and integration tasks.
    • Develop and maintain Azure Functions to support webhook operations and integrate with cloud services.
    Streaming Data:
    • Implement and manage Kafka Streams to handle real-time data processing and streaming requirements.
    Data Lake Management:
    • Work with Iceberg to manage and optimize large-scale data lake storage and querying.
    Collaboration and Communication:
    • Collaborate with data scientists, engineers, and business analysts to understand data requirements and provide technical solutions.
    • Document processes, architectures, and configurations to ensure knowledge sharing and compliance with best practices.
    Required Skills and Qualifications:

    Experience and Knowledge:
    • Proven experience with Airflow for data orchestration and workflow management.
    • Hands-on experience with Celery for task management and Redis or RabbitMQ for messaging.
    • Proficiency with Cosmos DB, MongoDB, and PostgreSQL for data storage and management.
    • Experience developing and managing webhooks using FastAPI and integrating with Azure Functions.
    • Knowledge of Kafka Streams for real-time data processing.
    • Familiarity with Iceberg for data lake management and optimization.
    • Healthcare domain experience is good to have .
    Technical Skills:
    • Strong understanding of data pipelines, ETL processes, and data integration.
    • Proficient in Python, with experience in building and maintaining data-oriented applications.
    • Ability to work with large datasets and optimize performance across distributed systems
    Soft Skills:
    • Excellent problem-solving and analytical skills.
    • Strong communication and collaboration skills.
    • Ability to work independently and manage multiple priorities in a fast-paced environment.

    Similar Job Searches

    Data Engineer Jobs in Palo Alto, CAData Manager Jobs in Palo Alto, CABusiness Intelligence Manager Jobs in Palo Alto, CADatabase Engineer Jobs in Palo Alto, CADirector Of Information Systems Jobs in Palo Alto, CAData Analyst Jobs in Palo Alto, CAData Architect Jobs in Redwood City, CAData Architect Jobs in Mountain View, CAData Architect Jobs in Sunnyvale, CAData Architect Jobs in Santa Clara, CA

    Similar Jobs