Senior Data Engineer - IoT

Senior Data Engineer - IoT

Location: Fort Mill, SC

Schedule: Monday-Friday | Primarily On-Site with flexibility for 1-2 days remote

Compensation: $185,000-$190,000 base salary + 7.5%-15% bonus

Position Overview

Our client is seeking a Senior Data Engineer - IoT to take a leading role in the development and long-term direction of its IoT data platform. This is a newly created position for a highly hands-on engineer who brings significant experience working with large-scale IoT, machine, or telemetric data.

This individual will help drive the continued migration of IoT data into Databricks, while designing and building a scalable data platform that supports real-time and near-real-time reporting, analytics, data science, AI/ML, customer-facing applications, and sustainability initiatives.

This is an opportunity to have meaningful ownership over the organization's IoT data strategy-not simply implement an existing specification. The ideal candidate will be comfortable moving between hands-on engineering, architecture, technical guidance, and business-facing discussions.

Key Responsibilities

• Design, build, and maintain end-to-end IoT data pipelines from edge devices into Databricks using Bronze/Silver/Gold medallion architecture.
• Lead the migration of IoT data from a third-party application into a scalable Databricks environment.
• Develop and optimize streaming ingestion pipelines using technologies such as Apache Kafka, Azure Event Hubs, and Databricks Structured Streaming.
• Establish data engineering best practices, including pipeline design patterns, data quality frameworks, schema governance, and scalable development standards.
• Architect and optimize Delta Lake tables, including partitioning, Z-ordering, and data lifecycle management for performance and cost efficiency.
• Collaborate with infrastructure and cloud teams on Databricks clusters, Unity Catalog, Delta Live Tables, and related cloud technologies.
• Develop solutions capable of supporting multi-terabyte, high-volume telemetry and time-series data.
• Partner with data scientists, analysts, and business stakeholders to translate requirements into scalable data solutions.
• Ensure IoT data is structured and optimized for downstream analytics, AI/ML, reporting, and customer-facing use cases.
• Mentor junior and mid-level data engineers through code reviews, pair programming, technical guidance, and architectural direction.
• Troubleshoot production pipeline issues, perform root-cause analysis, and implement long-term solutions.
• Evaluate emerging technologies and recommend improvements to the organization's IoT and data engineering platform.
• Represent the data engineering team in technical and business discussions and help shape the organization's long-term IoT data strategy.

Required Qualifications

• 5+ years of hands-on experience working with IoT, machine, telemetric, or similarly large-scale device data.
• Strong senior-level data engineering experience, preferably in environments involving high-volume or real-time data.
• Strong proficiency in Python and SQL, with significant experience developing production-grade Spark/PySpark solutions.
• Experience designing and implementing data pipelines for high-frequency telemetry, time-series data, or device-generated data.
• Experience with Databricks and/or other large-scale cloud data platforms.
• Strong understanding of data architecture, pipeline design, data quality, and schema governance.
• Experience working with streaming or near-real-time data architectures.
• Ability to lead technical discussions and translate business requirements into scalable technical solutions.
• Strong communication and interpersonal skills, with the ability to work effectively with engineers, data scientists, analysts, and business stakeholders.
• Ability to provide technical guidance and mentorship to other engineers.

Preferred Qualifications

• Hands-on experience with Databricks, PySpark, Delta Lake, Delta Live Tables, Unity Catalog, and Databricks Workflows.
• Experience with Apache Kafka, Azure Event Hubs, or similar streaming technologies.
• Experience with fleet telematics platforms such as Geotab or similar IoT/connected-vehicle platforms.
• Experience working with Azure cloud infrastructure and infrastructure-as-code concepts.
• Familiarity with Azure Key Vault and Azure Data Factory.
• Experience with dbt or other modern data transformation tools.
• Experience supporting multi-terabyte data environments.
• Databricks Certified Data Engineer Associate or Professional certification.
• Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related field, or equivalent professional experience.
• Experience within the rental, transportation, fleet, logistics, automotive, manufacturing, or connected-device industries is a plus.

What You'll Bring

The successful candidate will be more than a data pipeline developer. You'll bring the technical depth to work hands-on with complex IoT data while also providing architectural direction and helping shape how the organization approaches IoT data for years to come.

You are a hands-on technical leader who enjoys solving complex data challenges, mentoring engineers, collaborating with stakeholders, and turning large volumes of machine-generated data into solutions that drive meaningful business outcomes.

Why This Opportunity?

• Own the direction of IoT data engineering within a growing enterprise environment.
• Help design a scalable, modern IoT data platform rather than simply implementing an established specification.
• Play a key role in the continued migration and expansion of IoT data within Databricks.
• Work at the intersection of IoT, data engineering, analytics, AI/ML, customer-facing solutions, and sustainability.
• Provide technical leadership while remaining hands-on with the technology.
• Join a collaborative data organization with opportunities to influence architecture, engineering standards, and future technology decisions.

Category Code: JN008

#LI-DM1

#CRGSearchJobs

#LI-Onsite

Seniority Level: Mid-Senior Level