Analysis Skills, Artificial Intelligence (AI), Automation, Computer Science, Continuous Deployment/Delivery, Continuous Integration, Data Analysis, Data Management, Data Science, Fellowship, Information Technology Software, Mentoring, Model Validation, Performance Analysis, Performance Metrics, Performance Modeling, Quality Assurance, Quality Metrics, Regression Testing, Software Agents, Software Engineering, System Integration (SI), Systems Reliability, Test Automation, Test Strategy, Testing, Time Management, Trend Analysis, Webinar
The AI QA Analyst Associate will support the Testing and Quality Assurance team within the AI Technology Platform (AITP) program. The incumbent will focus on validating AI-enabled solutions, including LLM agents, and intelligent automation solutions. Responsibilities include evaluating model behavior, prompt effectiveness, response accuracy, system reliability, as well as testing AI workflows, data pipelines, and system integrations, assist in developing automated testing strategies, supporting CI/CD pipelines, and improving regression test coverage.
This Fellowship Program position is a full-time, temporary, limited term assignment not to exceed three years from the date of hire. It is intended for applicants that have graduated with a bachelor''s degree or a master''s degree from an accredited college or university in Computer Science, Information Technology, Software Engineering, Data Science, or a closely related field awarded within the last 12 months.
This position requires onsite work at our Austin TX headquarters.
WHAT YOU WILL DO:
AI Quality Assurance & Test Automation
- Assists in designing, developing, and executing automated test strategies for AI-enabled applications.
- Supports testing of AI workflows, data pipelines, and system integrations.
- Assists with prompt testing, hallucination detection, and evaluation of AI model performance using defined metrics.
- Supports basic bias and safety validation activities and participates in human-in-the-loop review processes to ensure accurate and appropriate AI outputs.
- Identifies and documents defects, inconsistencies, and risks during testing cycles.
- Assists with validating AI model inputs, outputs, and data flows to ensure accuracy and reliability.
- Supports testing of AI-driven features, including conversational agents and workflow automation.
Support Functions
- Supports data analysis related to AI system performance, including reviewing evaluation outputs, identifying trends,
- Assisting with reporting model quality metrics.
Fellowship Curriculum
- Participates in trainings and projects related to the TRS Fellowship Program including webinars, quarterly check-ins, classes or other learning activities, book group, projects, and the mentoring program.
- Meets TRS Fellowship Program deadlines and goals for completion of trainings in deliverables in a timely manner.
Performs related work as assigned.
T
Teacher Retirement System of Texas