Data Engineer

TechDigital

  • Fort Worth, TX
  • 30+ days ago

    Highlights

    Experience in data ingestion, cleansing, and transformation processes from various structured and unstructured data sources like Cassandra & Mark Logic and on-prem Mainframe sources using Databricks/PySpark supporting batch and near-real-time ingestion, transformation, and processing. Responsibilities: Design develop and optimize scalable data pipelines leveraging Databricks (Spark, PySpark, SQL, Delta Lake) to support enterprise level data processing and analytics.

    Numbers & Facts

    LocationFort Worth, TX
    IndustryOther/Not Classified
    Company Size100 to 499 employees

    Description

    Job Summary:
    Skills
    • 5+ years' experience in SQL (Expert)
    • 4+ years of experience in Azure Databricks with PySpark
    • 4+ years of experience in Azure Cloud platform
    • 3+ years of experience in ADF (Azure Data Factory), ADLS Gen 2 and Azure SQL
    • 2+ years of experience in Databricks workflow & Unity catalog
    • 2+ years of experience in Python programming & package builds
    • Experience in data ingestion, cleansing, and transformation processes from various structured and unstructured data sources like Cassandra & Mark Logic and on-prem Mainframe sources using Databricks/PySpark supporting batch and near-real-time ingestion, transformation, and processing.
    • Manage job scheduling, orchestration, and monitoring (e.g., using Azure Data Factory, Airflow, or Databricks Workflows).
    • Ability to design modular, reusable workflows using tasks, triggers, and dependencies.
    • Skilled in using dynamic expressions, parameterized pipelines, custom activities, and triggers.
    • Familiarity with integration runtime configurations, pipeline performance tuning, and error handling strategies.
    • Strong understanding of ETL/ELT design patterns, data warehousing, and data lakehouse architectures.
    • Good to have Azure Entra/AD skills and GitHub Actions
    • Good to have experience working on event-driven architectures using Kafka, Azure Event Hub
    Responsibilities:
    • Design develop and optimize scalable data pipelines leveraging Databricks (Spark, PySpark, SQL, Delta Lake) to support enterprise level data processing and analytics
    • Write clean maintainable and efficient PySpark and Python code to support data ingestion transformation
    • Integrate Azure Databricks with various Azure data services to build robust and scalable data platforms
    • Implement and maintain ETL workflows for scalability cost effective and operational efficiency
    • Collaborate with data analysts and stakeholders to gather requirements and deliver scalable data solutions
    • Participate actively in design discussions code reviews and agile ceremonies to foster a collaborative and high performing team environment.

    Similar Jobs