Highlights
Responsibilities include: Lead the technical migration from legacy SQL Server stored procedures and ADF pipelines to Databricks Lakehouse (Delta Lake), ensuring best practice Lakehouse design. They are responsible for designing and creating systems that collect, manage, and convert raw data into usable information for data scientists and business analysts to interpret and enables the agency to make smarter decisions and optimize operations.
Numbers & Facts
| Location | NY |
| Industry | Staffing/Employment Agencies |
| Salary | $150,000–$160,000 Per Year |
| Company Size | 50 to 99 employees |
| Year Founded | 2000 |
| Website | http://www.ecs-federal.com/ |
Description
ECS is seeking a Senior Databricks Migration Engineer to work in our Arlington, VA (hybrid) office. Please Note: This position is contingent upon additional funding.
The individual serves as the authoritative resource for the agency who specializes in preparing big data infrastructure for analytical or operational uses. They are responsible for designing and creating systems that collect, manage, and convert raw data into usable information for data scientists and business analysts to interpret and enables the agency to make smarter decisions and optimize operations.
Responsibilities include:
- Lead the technical migration from legacy SQL Server stored procedures and ADF pipelines to Databricks Lakehouse (Delta Lake), ensuring best practice Lakehouse design.
- Translate traditional relational data warehousing paradigms into scalable, distributed Lakehouse frameworks (Bronze, Silver, Gold).
- Design robust, reusable ETL/ELT frameworks using PySpark, Delta Live Tables (DLT), and Databricks Workflows.
- Architect and refine the Gold Layer (dimensional models, star schemas) specifically to maximize Power BI performance.
- Optimize Databricks SQL Warehouses to support high-concurrency, low-latency Power BI queries (DirectQuery and Import modes).
- Implement advanced optimization techniques, including Z-Ordering, data skipping, liquid clustering, and materialized views.
- Define and enforce governance standards for cluster sizing, auto-scaling policies, and serverless SQL compute to balance performance with cost.
- Implement proactive monitoring dashboards to track Databricks Unit (DBU) consumption and identify cost-saving opportunities.
- Establish best practices for partition strategies and file size management within Delta Lake.
- Design and implement a robust data security model using Unity Catalog for centralized governance.
- Enforce row-level and column-level security policies to ensure compliant data access for Power BI consumers and internal analysts.
- Align the Lakehouse security architecture with existing enterprise Azure Active Directory (Microsoft Entra ID) and RBAC standards.
- Act as the primary technical lead, conducting dedicated pair-programming sessions, workshops, and code reviews to transition the team from SQL-centric to Spark-centric thinking.
- Create comprehensive technical documentation, including architecture diagrams, design patterns, and optimization playbooks.
- Build a foundational knowledge transfer framework to ensure the internal team is fully self-sufficient post-migration.
- Communicate effectively verbally and in written form to both technical and non-technical audience.
- Work in an organized fashion, completing tasks timely while paying close attention to details.
Salary Range: $150,000-$160,000
General Description of Benefits
About Company
ECS was founded in 2001 by experienced IT professionals with a commitment to quality processes, people and performance. Led by our Chairman, Roy Kapani, and an experienced executive leadership team, ECS provides our customers with solutions and services that support their critical needs and further mission objectives. This commitment has paved the way for expansive growth, year over year.
ECS gained market share in 2011 in the Department of Defense and Federal spaces through both organic and acquisition growth. In May, ECS completed its first strategic acquisition with the purchase of OAK Management, Inc., a leading provider of marine environmental services, ship systems engineering, maritime consulting and platform acquisition management. The OAK acquisition kicked off ECS’ intention to add tactical acquisitions as a part of its long term strategy to supplement and expand upon organic growth and to build enterprise value. ECS closed out 2011 with the acquisition of Paradigm Technologies, Inc. The Paradigm transaction added approximately 200 employees to ECS’ existing 900+ employees. Paradigm also added new Defense clients for ECS, including the Missile Defense Agency, the Navy’s Program Executive Officer for Integrated Warfare Systems, the United States Marine Corps, and the U.S. Marshals Service.
In 2012, ECS completed the acquisition of iLuMinA Solutions, Inc. iLuMinA brings large-scale Enterprise Resource Planning (ERP) software implementation and infrastructure design and development to ECS’ expanding capabilities.
ECS will continue to invest in corporate infrastructure and quality processes as we grow and enhance our ability to offer professional excellence to both our customers and our employees.