Build robust data pipelines for ETL/ELT processes using tools like Dataflow , Cloud Composer , or other GCP services. Data Engineering: Design, develop, and maintain a Data Lake architecture using GCP services, particularly BigQuery and AlloyDB .
Numbers & Facts
Location
Parsippany, NJ
Industry
Other/Not Classified
Company Size
100 to 499 employees
Description
GCP Data Engineer with DBA
Roles and Responsibilities: Data Engineering:
Design, develop, and maintain a Data Lake architecture using GCP services, particularly BigQuery and AlloyDB.
Build robust data pipelines for ETL/ELT processes using tools like Dataflow, Cloud Composer, or other GCP services.
Integrate data from various sources (structured and unstructured) into the Data Lake, ensuring consistency and reliability.
Optimize BigQuery data models and queries for high performance and scalability.
Database Administration:
Administer and maintain AlloyDB instances, ensuring high availability, security, and performance.
Perform backup, recovery, and disaster recovery planning for AlloyDB and BigQuery.
Monitor and optimize database performance, query execution, and resource utilization.
Manage schema design, indexing, partitioning, and clustering to enhance database efficiency.
Apply database governance and compliance best practices to ensure data security and regulatory adherence.
Data Governance and Management:
Implement data governance policies, including role-based access controls, data masking, and encryption.
Manage metadata, data cataloging, and data lineage tracking to support audit and compliance requirements.
Conduct regular health checks on the Data Lake to ensure data quality and integrity.
Collaboration and Reporting:
Collaborate with data scientists, analysts, and application developers to define data requirements and deliver solutions.
Design and deploy real-time and batch data analytics solutions using GCP's BigQuery and Looker.
Provide documentation and training to teams on using Data Lake features effectively.
Automation and Monitoring:
Automate database and data pipeline tasks.
Set up monitoring and alerting for BigQuery and AlloyDB using Cloud Monitoring and Cloud Logging.
Resolve incidents and troubleshoot performance issues in real time.