MuleSoft Integration Engineer Intellibee
Durham, NC8 days ago
Highlights
The candidate will own the end-to-end technical design of the data and AI layers, build reference implementations for the engineering team, and ship production-grade Python, Scala, and PySpark code every week. The platform ingests 150+ inbound data feeds, distributes to 35+ downstream systems, and is organized as a medallion architecture (Bronze / Silver / Gold).
Numbers & Facts
| Location | Cary, NC |
| Job Type | Full-time |
| Industry | Computer/IT Services |
| Salary | $140,000–$145,000 Per Year |
| Year Founded | 1997 |
| Headquarters | Princeton, New Jersey, US |
| Website | http://www.innovativeinfotech.com |
Description
ABOUT THE ENGAGEMENT
A centralized, AI-first enterprise Data Hub for a global insurance and financial services client on Azure Databricks. The platform ingests 150+ inbound data feeds, distributes to 35+ downstream systems, and is organized as a medallion architecture (Bronze / Silver / Gold). AI is embedded in ingestion, canonical mapping, data quality, reconciliation, and business user access from day one.
This is a senior hands-on leadership role. The candidate will own the end-to-end technical design of the data and AI layers, build reference implementations for the engineering team, and ship production-grade Python, Scala, and PySpark code every week. Candidates who have not written or reviewed production code in the past year are not a fit.
Job Responsibilities
WHAT THE ROLE OWNS
- Data platform architecture and engineering: lakehouse architecture (Bronze / Silver / Gold contracts, ADLS Gen2 zone layout, Delta Lake table design, partitioning, schema evolution, retention).
- Metadata-driven, parameterized ingestion frameworks for batch files, database extracts, CDC feeds and streaming (Azure Event Hubs / Kafka, Spark Structured Streaming).
- Canonical PySpark and Scala Spark jobs, coding and testing standards, PR reviews, production incident debugging, Spark cluster tuning and cost guardrails.
- CI/CD for Databricks and ADF in Azure DevOps using Databricks Asset Bundles and Terraform; observability with Azure Monitor and Log Analytics.
- AI-augmented ingestion and canonical mapping: auto-generated bridge documents, DML, canonical table definitions; AI-assisted source-to-canonical mapping with human review gate.
- AI-driven data quality, anomaly detection (data drift, schema drift, volume shifts, reconciliation breaks), automated reconciliation, and synthetic privacy-preserving test data.
- Semantic layer and knowledge graph, plus a GPT-powered conversational interface (text-to-SQL / semantic-layer retrieval) with row- and column-level security.
- Governance and leadership: Unity Catalog (lineage, access control,
PII standards), Architecture Review Boards and AI governance forums, mentoring engineers, documentation.
Qualifications
MUST-HAVE SKILLS & EXPERIENCE
- Expert-level Python, Scala and PySpark: production-ready, modular, well-tested solutions; Spark workload troubleshooting; optimizing large-scale batch and streaming pipelines using Delta Lake.
- Strong SQL and data modelling (dimensional and normalised), schema design, data contracts.
- Databricks expertise: Delta Lake, Unity Catalog, Jobs & Workflows, cluster and pool management, performance tuning, Model Serving.
- Azure data stack: ADLS Gen2 (zone design, ACLs, lifecycle), Azure Data Factory (parameterized / metadata-driven frameworks), Azure Event Hubs.
- 3+ years designing and shipping LLM-based systems in production: RAG pipelines, agentic / tool-calling workflows, chunking and embedding strategy, vector and hybrid retrieval, prompt engineering.
- Evaluation discipline: golden datasets, regression suites, accuracy and hallucination tracking, human-in-the-loop feedback.
- Hands-on with LangChain, LlamaIndex or LangGraph, plus at least one provider stack (Azure OpenAI, OpenAI, or Databricks Model Serving).
- Metadata-driven frameworks: schema inference, data profiling, lineage, catalogs.
- 12-18 years of total experience in data engineering / data platform delivery.
- Proven enterprise-scale delivery of a medallion / lakehouse architecture.
- Azure security and governance: Entra ID, managed identities, RBAC, POSIX ACLs, Key Vault, private endpoints, PII handling.
- CI/CD and IaC: Azure DevOps, Terraform, Databricks Asset Bundles, automated testing of data pipelines.
- Clear technical writing and ability to present and defend designs to engineers and non-technical stakeholders.
STRONGLY PREFERRED
- Knowledge graphs and ontologies (RDF/SPARQL, Neo4j, graph modelling over a lakehouse).
- Text-to-SQL or semantic-layer-backed natural-language query systems at enterprise scale.
- ML-based anomaly detection on time-series or transactional financial data.
- Financial services or insurance domain (finance close, GL, subledger, reconciliation, actuarial data).
- LLMOps / MLOps: model and prompt versioning, cost governance, observability.
- Databricks Data Engineer Professional, Azure DP-203 / DP-700, or AZ-305 certification.
- dbt, Great Expectations or similar; Workday, Prism or Accounting Center exposure.
About Company
Innovative Information Technologies, Inc. is a leading provider of information technology services and solutions. Our mission is to deliver high-quality technological solutions that enable companies to meet their business goals more effectively. With offices located in the US and India, we offer our clients an array of flexible, cost-effective solutions that range from staff supplementation through complete project outsourcing. We offer onsite, offsite and offshore delivery models that are tailored to the needs of our clients. Specializing in specific technological competencies, our solutions are intentionally designed to allow our clients to maximize the power of technology for the ongoing success of their business.
Similar Job Searches
Similar Jobs
- New!
*Hybrid Remote* Structural Forensic Engineer Jobot
Raleigh, NC1 day agoRemote - New!
Senior Mechanical Engineer (PE) – Building Systems Jobot
Raleigh, NC2 days ago Mechanical Engineer, Senior Management Jobot
Raleigh, NC8 days ago- New!
Sr. Electrical Engineer Jobot
Durham, NC3 days ago - New!
Electrical Engineer Jobot
Raleigh, NC2 days ago