Job Title: Data Engineer - Databricks / Customer 360
Location: Marlboro, MA - Hybrid (3 days in office: Tuesday, Wednesday & Thursday)
Bill Rate: $70/hr
ROLE_DESCRIPTION
We are seeking a skilled Data Engineer to design, build, Eligible to workimize, and govern scalable data pipelines and Customer 360 data solutions within the Databricks platform.
RESPONSIBILITIES
- Design, develop, and deploy batch and streaming data pipelines using PySpark, Spark SQL, and Lakeflow to ingest data from POS, e-commerce platforms, loyalty systems, and marketing clouds.
- Implement data transformations and business logic within the Medallion Architecture (Bronze, Silver, and Gold layers).
- Build and Eligible to workimize Gold dimension tables as the single source of truth.
- Define governed business KPIs using Unity Catalog Metric Views.
- Implement data quality frameworks and cleansing routines to ensure accurate and trustworthy Customer 360 data.
- Monitor, debug, and Eligible to workimize Databricks jobs and Spark clusters for performance and cost efficiency.
- Implement partitioning, caching, and Delta Lake data layout best practices.
- Manage Databricks environments, clusters, and deployments using Terraform and AWS DevOps/GitHub Actions.
- Implement CI/CD best practices for data pipelines.
- Implement Unity Catalog governance capabilities, including data lineage, access controls, tags, and data masking.
- Collaborate with Functional Consultants, Data Scientists, and Analytics Engineers to deliver consumption-ready datasets.
- Build and govern Unity Catalog Business Semantics, including Metric Views, Domains, Pages, Glossary terms, and certified assets.
- Configure and curate AI/BI Genie Spaces and the Genie Ontology to enable conversational analytics and AI-driven data consumption.
SKILLS_REQUIRED
- PySpark
- Spark SQL
- Lakeflow
- Lakeflow Connect
- Lakeflow Declarative Pipelines
- Terraform
- AWS DevOps
- GitHub Actions
Disclaimer: Diverse Lynx LLC is an Equal Opportunity Employer. All applicants and employees are evaluated without discrimination, based solely on their qualifications, ability, competence and performance. This email and its attachments may contain confidential or proprietary information and is intended only for the recipient(s). If you received this message in error, please disregard it and notify the sender. If you no longer wish to receive our communications, you may unsubscribe at any time.
Security Notice: Our official website is www.diverselynx.com We do not operate or authorize any other websites representing Diverse Lynx LLC.