Validation Engineer, AI & Data
Location: Indianapolis, IN
Summary: Validate the accuracy, reliability, completeness, and consistency of business context and AI generated responses against approved business requirements and ground truth answers.
Responsibilities:
- Design and implement the testing strategy for the Domain Context Model, Business Context Repository, ontology mappings, and AI response evaluation.
- Develop representative ground truth question and answer test sets with defined expected results and tolerances, validated by data owners.
- Build and maintain automated evaluation tests for AI responses generated through the client’s Context Injection toolkit.
- Evaluate AI responses with and without domain context to establish baseline accuracy and measure improvements.
- Validate numeric answers, SQL and query results, business logic, aggregations, join relationships, and narrative responses.
- Identify and classify failures, including hallucinations, stale information, incorrect schemas, faulty joins, missing business context, and persona mismatches.
- Develop repeatable evaluation scripts, scoring rubrics, regression tests, and accuracy reporting dashboards or scorecards.
- Verify context repository data completeness, classification consistency, access control behavior, and changes introduced through automated context updates.
- Document test evidence, defects, remediation results, AI readiness assessments, and validation outcomes.
- Support release readiness, weekly demonstrations, and handover of reusable evaluation frameworks and documentation.
Requirements
Required Technical Expertise: Data quality engineering, SQL, Python, automated testing, PyTest or equivalent frameworks, ETL and data validation, AI and LLM evaluation, ground truth testing, regression testing, and test automation.