Design and implement scalable data pipelines and architectures on Azure Databricks.
Optimize ETL/ELT workflows, ensuring efficiency in data processing, storage, and retrieval.
Leverage Apache Spark, Delta Lake, and Azure-native services to build high-performance data solutions.
Ensure best practices in data governance, security, and compliance within Azure environments.
Troubleshoot and fine-tune Spark jobs for optimal performance and cost efficiency.
Lead the migration of Azure SQL to Azure Databricks, ensuring a seamless transition of data workloads.
Design and implement scalable data pipelines to extract, transform, and load (ETL/ELT) data from Azure SQL into Databricks Delta Lake.
Optimize Azure SQL queries and indexing strategies before migration to enhance performance in Databricks