Sr. Data Engineer

Required 5 Years Experience with version control systems such as Git and CI/CD pipelines Required 5 Years Knowledge of data modeling techniques including dimensional modeling, data vault, and star/snowflake schemas. Create data tools for analytics and data scientist team members that assist them in building and optimizing data products needed to support ongoing operations and data-informed decision-making.

Title: Sr. Data Engineer
Duration: 11/23/2026 - 12/31/2026
Work Hours: 40hrs/week
Location: 400 North Street, 5th Floor, Harrisburg, PA 17120

Responsibilities:

• Create and maintain optimal data pipeline architecture.
• Design and implement cloud-based data solutions using AWS services such as S3, Glue, Redshift, EMR, Athena, Kinesis, and Lambda. 
• Build and maintain data orchestration workflows using tools such as Apache Airflow, AWS Step Functions, or similar services. 
• Assemble large, complex data sets that meet functional / non-functional business area requirements.
• Engineer analytical data environments that support advanced statistical analysis, predictive modeling, machine learning, and evidence-building activities.
• Partner with data scientists, research analysts, and policy analysts to ensure enterprise data assets support reproducible research and statistical modeling.
• Identify, design, and implement internal process improvements: e.G. Automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and 'big data'technologies.
• Build analytics tools that utilize the data pipeline to provide actionable insights into education and workforce outcomes, operational efficiency, and other key business performance metrics.
• Develop and maintain data pipeline monitoring, alerting, and logging frameworks. 
• Implement data quality checks and validation frameworks across pipelines.   
• Develop dashboards, reports, analytical models, and/or other decision-support products. 
• Work with stakeholders including the Executive, Data, Design, and Support teams to assist with data-related technical issues and support their data infrastructure needs.
• Ensure compliance with applicable Commonwealth policies, data governance standards, privacy requirements, security controls, and interagency data-sharing agreements.
• Create data tools for analytics and data scientist team members that assist them in building and optimizing data products needed to support ongoing operations and data-informed decision-making.
• Work with data and analytics professionals to strive for greater functionality in our data systems.

 
Required Qualifications:

• Advanced SQL knowledge and proficiency working with relational databases, query authoring (SQL), as well as working familiarity with a variety of databases.
• Experience building and optimizing 'big data'data pipelines, architectures, and data sets.
• Advanced knowledge and proficiency designing, building, and refining dashboards, reports, analytical models, and decision-support products for executive and non-technical users. 
• Experience working with student-level higher education data, including establishing linkages to other sources (e.G. Employment and wage data) to measure outcomes and/or generate insights for decision-making). 
• Experience performing root-cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Strong analytic skills related to working with unstructured datasets.
• Experience building processes supporting data transformation, data structures, metadata, dependency, and workload management.
• A successful history of manipulating, processing, and extracting value from large, disconnected datasets.
• Working knowledge of message queuing, stream processing, and highly scalable 'big data'data stores.
• Strong project management and organizational skills.
• Experience supporting and working with cross-functional teams in a dynamic environment.

Required / Desired Skills Skill Required / Desired Amount of Experience   Hands-on experience with cloud-based data services such as AWS (e.G. S3, EC2, Glue, EMR, RDS, Redshift, Athena, Lambda, DynamoDB) Required 5 Years   Proficiency in object-oriented/object function scripting languages: e.G. Python, R, Java, C++, Scala Required 5 Years   Proficiency in relational SQL and NoSQL databases, including Oracle, MS SQL Server, Postgres, Cassandra. Required 5 Years   Experience with version control systems such as Git and CI/CD pipelines Required 5 Years   Knowledge of data modeling techniques including dimensional modeling, data vault, and star/snowflake schemas. Required 5 Years   Familiarity with data cataloging and metadata management tools such as AWS Glue Data Catalog. Required 5 Years   Understanding of data governance, data lineage, and data quality frameworks. Required 5 Years   Knowledge of REST API design and integration patterns. Required 5 Years   Experience with big data tools: e.G. Hadoop, Spark, Kafka. Required 5 Years   Experience with data pipeline and workflow management tools: e.G. Azkaban, Luigi, Airflow. Required 5 Years   Experience with data integration services solutions from vendors such as Informatica, MuleSoft, Talend, and TIBCO. Required 5 Years   Experience with stream-processing systems: e.G. Storm, Spark-Streaming. Required 5 Years