Data Migration Consultant

Job Title: Data Migration Consultant
Location: Onsite Indianapolis
Industry: Life Sciences

Job Overview:
Theoris is assisting our client in their search for a Data Migration Consultant to be a part of a multiyear Safety Case Management transformation which will address many business challenges. This will include Portfolio Complexity and Growth, Interoperability Across independent systems and Adoptability to regulatory and technological change. In this role you will be working with a group of others to Design, and build metadata driven, config based raw-to-refine ingestion pipelines for large-scale Veeva Vault object sets using Databricks on AWS with use of a Unity catalog medallion architecture. You will be developing and maintaining Databricks Lakeflow Declarative Pipelines, writing and optimizing python based code, and authoring advanced SQL for databricks.
 
Requirements

• Must be Eligible to work on W2 without requiring sponsorship now or in the future
• Willing to work on site for a minimum of 3 days a week or additional as required
• 5+ Years experience in Data Engineering or Cloud Engineering including migrations
• 5+ years of experience with AWS
• Minimum 2 years’ experience with Veeva Vault
• Experience Developing and maintaining Databricks Lakeflow Declarative Pipelines (formerly DLT), including CDC decorator patterns for insert/update/delete tracking and materialized views for relationship expansion.
• Experience with writing and optimizing Python-based code generators and automation utilities that translate metadata exports into production pipeline definitions (e.g., SQL/MV templates).
• Ability to author advanced SQL for Databricks/Delta and Iceberg-compatible tables, resolving platform-specific issues (e.g., LATERAL VIEW/JOIN chaining, correlated subqueries, deletion vectors, REORG PURGE).
• Experience designing manifest driven CI/CD approval and deployment workflows.
• Experience with compliance tractability and data quality standards within a regulated environment.

• Design and build metadata-driven, config-based raw-to-refine ingestion pipelines for large-scale Veeva Vault object sets (e.g., 200+ objects) using Databricks on AWS with a Unity Catalog medallion (raw/refine) architecture.

• Develop and maintain Databricks Lakeflow Declarative Pipelines (formerly DLT), including CDC decorator patterns for insert/update/delete tracking and materialized views for relationship expansion.
• Integrate with the Veeva Direct Data API (DDAPI) and/or Databricks Lakeflow Veeva Vault Connector, including managing incremental file timing, schema evolution, and field-rename edge cases.
• Write and optimize Python-based code generators and automation utilities that translate metadata exports into production pipeline definitions (e.g., SQL/MV templates).
• Author advanced SQL for Databricks/Delta and Iceberg-compatible tables, resolving platform-specific issues (e.g., LATERAL VIEW/JOIN chaining, correlated subqueries, deletion vectors, REORG PURGE).
• Define and provision cloud and platform infrastructure using Terraform, including Databricks workspaces, Unity Catalog objects, service principals, and IAM/permissions.
• Build and maintain AWS infrastructure using CloudFormation (CFN), integrating with services such as S3, Glue, Lambda, Step Functions, EventBridge, and DynamoDB where applicable.
• Design manifest-driven CI/CD approval and deployment workflows (e.g., GitHub Actions) leveraging the Databricks Statement Execution API for controlled production releases.
• Ensure pipelines meet the compliance, traceability, and data-quality standards required in regulated pharma domains, understanding Veeva Vault as the system of record for transactional data.
• Partner with Safety/Regulatory IT teams, data architects, and business stakeholders to translate Veeva data domain requirements into scalable engineering solutions.
• Troubleshoot and resolve complex data pipeline, schema evolution, and platform compatibility issues; drive root-cause fixes and long-term architectural improvements.
• Document architecture decisions, naming conventions, and operational runbooks; mentor other engineers on Veeva/Databricks best practices.

• Health insurance
• Health savings account
• Dental insurance
• Vision insurance
• Flexible spending accounts
• Life insurance
• Retirement plan