DataBricks Specialist

Job Title: DataBricks Specialist

Location: New York, NY

Duration: Fulltime

Skills: Python | SQL

Salary: 130K-150K/Year

Roles and Responsibilities:

Must Have Technical/Functional Skills

• Experience: 5 to 10+ years in data engineering, software development, or solution architecture, with hands-on Databricks production experience.
• Technical Stack: Strong proficiency in Python, SQL, or Scala; deep knowledge of Apache Spark runtime internals and Delta Lake.
• Cloud Ecosystems: Working knowledge of major cloud platforms like Microsoft Azure, AWS, or GCP.
• Certifications: Preferred credentials include Databricks Certified Data Engineer or Solutions Architect credentials.

Roles & Responsibilities

• Architecture & Development: Design end-to-end data pipelines, ETL/ELT processes, and lakehouse storage patterns using Apache Spark, Delta Lake, and Python or SQL.
• Performance Tuning: Optimize cluster sizing, query execution, shuffles, partitions, and storage layouts for large-scale data workflows. [1]
• AI & Machine Learning Integration: Implement MLOps workflows, manage model lifecycles, or deploy generative AI and vector search solutions using tools like MLflow and Unity Catalog. [1]
• Collaboration & Consulting: Partner with enterprise stakeholders, data scientists, and software engineers to translate business requirements into technical solutions. [1]
• Administration & Governance: Oversee security configurations, access controls, workspace administration, and CI/CD deployment pipelines

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.