AI Quality Assurance Analyst

Description

Responsibilities and Duties:

• Validate AI-driven outputs related to loan processing, underwriting, document recognition, and decisioning.
• Design and execute test cases for AI models, automation workflows, and system enhancements.
• Identify, document, track defects, inconsistencies, and performance issues.
• Perform regression, functional, and user acceptance testing (UAT) for mortgage systems.
• Collaborate with product, engineering, and data science teams to improve AI accuracy and reliability.
• Analyze AI model behavior and flag potential risks, biases, or compliance concerns.
• Ensure alignment with mortgage industry regulations and investor guidelines (e.g., Non-QM with the future agency).
• Support data validation efforts, including OCR (optical character recognition) and document classification tools.
• Monitor production performance and provide feedback for ongoing AI model training and optimization.
• Assist in developing QA standards and best practices for AI implementations.

Education and Experience:

• 3+ years of experience in QA, software testing, or mortgage operations 
• Strong understanding of the mortgage lifecycle, including origination, processing, underwriting, and closing 
• Experience with Loan Origination Systems (LOS), such as Vesta, LauraMac, or similar platforms 
• 2+ years of ServiceNow ticketing experience required
• Familiarity with QA methodologies, including test planning, execution, and defect tracking 
• Strong analytical and problem-solving skills with a high level of attention to detail 
• Ability to analyze system behavior and validate expected versus actual results 

• Experience with AI/ML tools, automation platforms, or intelligent document processing 
• Understanding of OCR, NLP, or decisioning engines 
• Knowledge of mortgage compliance and regulatory requirements 
• Experience with SQL, data validation, or basic scripting is a plus 
• Exposure to Agile or Scrum development environments 
• Ability to work in a hybrid office setting in Bloomington, MN

• AI & QA Testing Expertise: Designs and executes test cases to validate AI models, automation workflows, and system functionality. 
• Analytical & Critical Thinking: Evaluates AI outputs, identifies discrepancies, and assesses model behavior for accuracy and risk. 
• Mortgage Domain Knowledge: Applies understanding of loan lifecycle, documentation, and regulatory requirements to testing scenarios. 
• Attention to Detail & Data Validation: Ensures precision in testing, defect tracking, and validation of OCR and document classification outputs. 
• Cross-Functional Collaboration: Works effectively with product, engineering, and data science teams to improve AI performance.

• Integrity: Ensures AI-driven decisions and outputs are accurate, compliant, and aligned with regulatory and investor requirements. 
• Collaboration: Partners with cross-functional teams to enhance AI solutions and support seamless system performance. 
• Excellence: Maintains high standards in testing, validation, and continuous improvement of AI-driven processes. 
• Critical Curiosity: Investigates AI behavior, identifies potential risks or biases, and seeks opportunities to improve model accuracy and efficiency.

Benefits

• Competitive compensation package, including base salary and performance-based bonus opportunities
• 401(k) plan with 100% company match up to 4%
• Comprehensive health coverage: medical, dental, vision, HSA, and FSA options
• Generous paid time off: 20 days PTO, company holidays, and sick time
• Paid parental leave
• Company-paid life insurance and disability coverage
• Employee Assistance Program (EAP): mental health, financial, and wellness support
• Professional development: tuition reimbursement and growth opportunities
• Commuter and transit benefits