Product Analyst

Required Qualifications 3+ years experience drafting detailed Jira user stories, maintaining product backlogs, and defining acceptance criteria based on business requirements in an Agile product team or equivalent environment. Preferred Qualifications Experience in HR technology, learning technology, content creation platforms, or knowledge management solutions as a Business Analyst, Product Analyst, Product Owner, or equivalent role.

Role: Product Analyst
Location - Phoenix, AZ is preferred but remote can be considered
Duration - 6+ months

Skills -

Required Qualifications

• 3+ years experience drafting detailed Jira user stories, maintaining product backlogs, and defining acceptance criteria based on business requirements in an Agile product team or equivalent environment.
• 2+ years experience in data analysis with the ability to interpret AI performance metrics and translate findings into actionable product recommendations.
• 1+ year experience working with LLM-based applications, including understanding of hallucination patterns, output quality issues, grounding techniques, bias detection, and mitigation strategies.
• 1+ year hands-on experience executing structured AI agent evaluation plans, tracking iterations, maintaining golden datasets, and performing regression testing.
• 1+ year experience supporting the development, evaluation, and continuous improvement of AI-powered products, AI agents, copilots, or generative AI solutions.
• Experience translating AI product requirements into prompts, agent workflows, business rules, and measurable success criteria.
• Experience defining AI product success metrics and measuring improvements in output quality, user satisfaction, task completion, or productivity.
• Experience partnering with engineering, UX, data science, and business stakeholders to prioritize, evaluate, and refine AI product capabilities.

Preferred Qualifications

• Experience in HR technology, learning technology, content creation platforms, or knowledge management solutions as a Business Analyst, Product Analyst, Product Owner, or equivalent role.
• Hands-on experience building and executing specialized AI test cases, evaluating LLM outputs, prompt variations, and agentic workflows.
• Hands-on experience with prompt engineering, prompt optimization, and evaluation of AI-generated content quality, accuracy, completeness, and consistency.
• Experience designing evaluation rubrics, benchmark datasets, golden datasets, and quality measurement frameworks for AI-generated outputs.
• Experience conducting side-by-side model comparisons, experimentation, or A/B testing to improve AI product performance.
• Familiarity with Retrieval Augmented Generation (RAG), grounding strategies, hallucination mitigation, and content sourcing concepts.
• Familiarity with automated AI evaluation tools, frameworks, and testing methodologies.
• Experience with human-in-the-loop workflows and designing review, validation, and approval processes for AI-generated outputs.
• Experience working with AI governance, responsible AI practices, bias detection, content safety, and model quality evaluation.
• Experience evaluating content quality against defined business, compliance, instructional design, or organizational standards.
• Strong written and verbal communication skills, with the ability to document evaluation methodologies, findings, recommendations, and trade-offs for both technical and business audiences.