AI QA Engineer

AI Quality EngineerLocation Charlotte NCCompensation 90000-105000 annuallyEmployment Type Full-timeProject Deloitte Wealth Management ClientPosition SummaryWe are seeking an AI Quality Engineer to join a high-impact team delivering AI-enabled capabilities within enterprise systems for a leading wealth management client. This role will help ensure that AI-powered applications-including LLM-based assistants retrieval-augmented generation RAG solutions machine learning models and intelligent workflows-are reliable accurate secure and ready for enterprise use.The AI Quality Engineer will define and execute quality strategies across the AI solution lifecycle from data preparation and model evaluation through integration testing production monitoring and continuous improvement. The ideal candidate brings strong quality engineering discipline hands-on experience testing complex enterprise applications and a practical understanding of AIML or LLM-driven systems.This is an opportunity to shape repeatable AI quality practices in a visible regulated environment where accuracy user trust operational reliability and measurable business outcomes are critical. The role aligns with established AI risk-management expectations that emphasize ongoing testing evaluation verification and validation throughout the AI lifecycle.nist1Key ResponsibilitiesDesign and execute test strategies for AI-powered applications including functional integration regression usability performance and non-functional testing.Validate LLM-based solutions for accuracy relevance consistency groundedness safety latency and end-to-end task completion.Test retrieval-augmented generation solutions including document ingestion retrieval quality prompt behavior response quality citations or source alignment and orchestration workflows.Develop test cases evaluation datasets benchmark scenarios and expected outcomes for prompts retrieval pipelines model-driven workflows and end-user interactions.Identify model failure modes hallucinations edge cases bias concerns workflow breakdowns and integration issues document findings and recommend corrective actions.Perform data validation activities including data preprocessing checks feature validation test-data quality assessment and data-flow testing.Support validation of machine learning and AI models using relevant metrics business scenarios and acceptance criteria.Test APIs data pipelines system integrations workflows and dependencies supporting AI-enabled business processes.Partner with AI engineers software engineers product owners business analysts and stakeholders to define quality standards acceptance criteria and release-readiness requirements.Contribute to automation of testing AI evaluations regression checks monitoring and quality reporting across development and production environments.Monitor production feedback user behavior operational metrics and model performance to identify opportunities for improvement.Document defects test results risks quality decisions methodologies and release recommendations.Support quality governance including traceability defect management root-cause analysis and evidence for release approvals.Help establish scalable AI quality engineering practices controls and reusable testing assets for enterprise delivery.Required QualificationsBachelors degree in Computer Science Engineering Information Systems Data Science or a related technical field.6 years of experience in quality engineering QA automation software testing test engineering or a related technical discipline.Experience testing enterprise applications APIs data pipelines workflow-based systems or integrated technology platforms.Experience evaluating AI- machine learning- or LLM-enabled solutions including prompt behavior output quality model-driven workflows or automated decision-support capabilities.Strong knowledge of software-testing methodologies including functional integration regression end-to-end performance and user-acceptance testing.Experience creating test plans test cases test data defect reports quality metrics and release-readiness documentation.Understanding of defect management root-cause analysis risk assessment and quality reporting.Familiarity with QA automation frameworks test-management tools CICD-aligned testing and Agile delivery practices.Ability to work effectively with both technical teams and business stakeholders.Strong written and verbal communication skills with the ability to clearly communicate quality risks trade-offs and recommendations.Preferred QualificationsExperience testing LLM applications AI assistants generative AI solutions intelligent agents or RAG-based systems.Experience with LLM evaluation prompt testing hallucination testing retrieval validation AI observability or automated evaluation frameworks.Familiarity with Python SQL REST APIs JSON scripting or automation tools used for test automation and data validation.Experience creating synthetic datasets golden datasets benchmark scenarios adversarial test cases or AI evaluation frameworks.Exposure to model monitoring production observability drift detection model-performance metrics or feedback-loop design.Familiarity with responsible AI AI governance privacy security model risk or compliance controls.Experience in financial services wealth management consulting insurance healthcare or another regulated environment.Experience working with cloud-based AI services or platforms such as Azure AI AWS Bedrock Google Vertex AI OpenAI or similar tools.