AI Quality Analyst, AI & Innovation

The AI Quality Analyst, AI & Innovation safeguards the quality, reliability, and measurable performance of the team's AI data extraction processes, which convert commercial risk insurance documents - such as policies, endorsements, submissions, schedules of values, and loss runs - into structured data. Rather than building the extraction systems, this role exists to support the organization's Enterprise AI Evaluation Strategies - the standards, methods, and processes used to determine whether AI systems are accurate, reliable, and ready for use. The role spans both the foundation and the practice of evaluation: building and curating the ground-truth ("golden") datasets against which AI performance is measured, and contributing directly to the evaluation processes those datasets power. Reporting to the Director, AI & Innovation, the AI Quality Analyst creates, curates, and maintains ground-truth files, helps design and run evaluations, reviews AI extraction outputs against ground truth, and delivers clear, data-driven feedback that the development team uses to advance the extraction process.

A central part of the role is engaging directly with business users: training them on the ground-truth application, guiding them through the process of producing and reviewing ground-truth data, and collecting the associated source files and validated outputs. The AI Quality Analyst maintains a comprehensive, working understanding of the extraction processes to define what "correct" looks like and to safeguard extraction quality end to end. The role is well suited to someone who brings a working knowledge of commercial risk insurance, is familiar with AI/ML and large language model concepts, and is skilled at quality review, data stewardship, and stakeholder engagement - without requiring hands-on AI/ML development experience.

Key Responsibilities

• Own, create, and maintain the curated ground-truth ("golden") datasets and ground-truth labels that developers rely on to measure, benchmark, and improve AI extraction quality - the foundation of the team's evaluation work.
• Contribute to the design, execution, and continuous improvement of the organization's Enterprise AI Evaluation Strategies, making sure ground-truth data is built to serve the evaluations it supports rather than simply collected and stored.
• Help design evaluation plans and test sets for AI extraction use cases, selecting representative and edge-case samples across commercial risk insurance document types, lines of business, and levels of complexity.
• Participate in evaluation cycles - baseline, regression, and pre-release testing - whenever models, prompts, or processes change, and help define the acceptance thresholds that determine whether changes are ready for production.
• Help calibrate automated evaluation methods, such as LLM-as-judge scoring, against human-reviewed ground truth to confirm those methods can be trusted at scale.
• Contribute to enterprise evaluation standards, templates, and playbooks so evaluation practices are consistent and reusable across AI initiatives beyond data extraction.
• Define, document, and uphold clear standards and annotation guidelines for what constitutes a correct extraction, so quality can be measured consistently and reproducibly.
• Apply commercial risk insurance knowledge to define what a correct extraction looks like for key policy and risk data - such as named insureds, lines of coverage, limits, deductibles and retentions, premiums, exclusions, and loss history - and to resolve ambiguous or complex cases.
• Engage with and train business users on the ground-truth application, guide them through creating and reviewing ground-truth data, and collect the associated source files and validated outputs needed to build and expand the ground-truth library.
• Review AI extraction outputs against ground truth, identify and categorize discrepancies and error types, and deliver actionable diagnostics and feedback to the team advancing the extraction process.
• Coordinate human-in-the-loop review and annotation workflows, including inter-annotator agreement, to keep ground-truth data accurate, consistent, and trustworthy.
• Manage the versioning and lifecycle of ground-truth datasets so quality results remain reproducible and comparable as models, prompts, and processes change.
• Partner with developers to define and interpret quality metrics and acceptance criteria - accuracy, precision, recall, F1, completeness, groundedness/faithfulness, hallucination rate, and schema conformance - and help translate results into clear findings.
• Monitor quality trends across extraction runs to surface drift, degradation, and anomalies before they reach downstream products, escalating issues to the team.
• Maintain a comprehensive, working understanding of the existing AI data extraction processes the team leads, partnering closely with the Manager and team in support of their continuous improvement.
• Support responsible-AI review of AI outputs, including robustness, bias, safety, and compliance considerations at the output level.
• Define and maintain quality metrics, dashboards, and reporting that give the team and stakeholders clear visibility into extraction performance and ground-truth coverage.
• Document ground-truth methodology, metric definitions, annotation standards, and quality practices to establish and mature the team's AI quality practice.
• Collaborate with the Analytics Engineering function, which consumes validated AI-extracted data into Gold-layer assets, and the Applied AI Analyst, which evaluates AI from a business and product perspective, to align quality standards with downstream needs.