AI Test Manager

AI Test Manager

Job Title: AI Solutions Evaluation & Experience Manager – Contractor

Location: [Greenwood Village, CO]

Contract Duration: [6 months, with potential for extension]

Job Overview:

• We are seeking a highly technical AI Solutions Evaluation & Experience Manager to architect the framework that determines the real-world intelligence and accuracy of our AI and analytic products and perform ongoing solution testing and monitoring. In this role, you will move beyond traditional QA to implement LLM-as-a-Judge and SLM patterns to evaluate performance at scale.
• You will be responsible for managing a team that will calculate and interpret critical technical metrics, such as RAGas scores (Faithfulness, Answer Relevance), Perplexity, and Hallucination rates, to determine model accuracy.
• Additionally, you will analyze adoption and utilization metrics to assess the health of our product portfolio and trigger safe product retirements, when appropriate.
• 5+ years of experience in management and 3+ years of experience in technology or AI are required.
• A thorough understanding of test script development, requirements traceability, and testing execution for technical solutions is required.