Sign upLog in

AI Business Data Analyst

UNICON International, Inc.

Torrance, CA

Apply
JOB DETAILS
SALARY
$103,000–$108,000 Per Year
SKILLS
A/B Testing, Acceptance Testing, Adobe Product Family, Agile Programming Methodologies, Amazon Web Services (AWS), Analysis Skills, Artificial Intelligence (AI), Artificial Intelligence (AI) Agents, Backlog Prioritization, Benchmark Testing, Business Analysis, Business Case, Business Skills, Business impact analysis (BIA), Change Management, Cloud Computing, Cognitive Science, Competitive Research, Computer Science, Consumer Branding, Content Indexing, Continuous Improvement, Conversation Engine, Cross-Functional, Customer Experience, Customer Support/Service, Customer/Client Research, Customer/Consumer Behavior, Data Analysis, Data Science, Database Programming Languages, Decision Support, Dental Insurance, Detail Oriented, Documentation, Economics, Experiment Design, Identify Issues, Information Retrieval, Information Science, Instrumentation, Leadership, Marketing, Metadata, Metrics, Microsoft Excel, Needs Assessment, Operational Audit, Operational Improvement, Operational Support, Performance Analysis, Performance Metrics, Power BI, Process Flow, Product Backlog, Product Demonstration, Product Planning, Product Support, Product Testing, Production Control, Production Systems, Python Programming/Scripting Language, Qualitative Research, Quality Assurance, Quality Monitoring, Reporting Dashboards, Requirements Management, Risk, Risk Analysis, SQL (Structured Query Language), Scorecarding, Scrum Project Management and Software Development, Sprint Planning, Sprint Retrospective, Statistics, Systems Analysis, Tableau, Taxonomies, Team Lead/Manager, Team Player, Technical Support, Telemetry, Testing, Trend Analysis, Use Cases, User Experience Design (UXD), User Interface/Experience (UI/UX), Validation Plan, Vision Plan, Willing to Travel
LOCATION
Torrance, CA
POSTED
2 days ago
Job Title: AI Business Data Analyst
Location: Torrance, CA
Work Model: Hybrid
Compensation: $103-108k

Job Title: AI Business Data Analyst
Reports To: CX AI Product Owner
Division: CX & UX Design Division
Department: CX Product

3rd Party Suppliers: Not accepting 3rd party suppliers for this role
Work Type: Remote (Onsite 2 days a week)
Travel: 5%
Overtime: Yes

POSITION OVERVIEW
The AI Business Data Analyst will support American client Motor s Customer Experience digital transformation by providing day-to-day analytics, quality monitoring, KPI definition, evaluation, and insight generation for production generative AI and conversational AI applications.

This role will help ensure AI-powered customer experiences are accurate, reliable, measurable, and continuously improving. The analyst will monitor production LLM application performance, evaluate response quality, build dashboards, identify risks and opportunities, and synthesize insights that inform product roadmap decisions, feature requirements, backlog prioritization, and operational improvements.
The ideal candidate combines strong analytical skills, product sense, technical curiosity, and practical understanding of generative AI applications, including conversational AI, RAG, prompt-based systems, AI evaluation metrics, and LLM-as-a-judge methodologies.

WHAT YOU WILL BE DOING
AI Product Performance, Evaluation & Insights 50%
Establish, monitor, and report on KPIs related to AI product performance, customer experience, user behavior, operational efficiency, and business value.
Analyze production AI application data, including conversation logs, usage patterns, customer feedback, quality signals, escalation trends, and operational telemetry.
Identify issues and opportunities related to response quality, accuracy, hallucination risk, retrieval relevance, content drift, deflection, satisfaction, coverage, and task completion.
Design and support AI evaluation processes, including response quality reviews, evaluation rubrics, benchmark test sets, human-in-the-loop review, and LLM-as-a-judge approaches.
Build dashboards, scorecards, and recurring business reviews that communicate AI application health, trends, risks, and improvement opportunities.
Translate findings into actionable recommendations, product requirements, backlog items, acceptance criteria, and release validation plans.

Product Analytics, Discovery & Roadmap Support 25%
Conduct exploratory analysis to understand user behavior, customer intent, conversation topics, feature usage, and AI-assisted customer journeys.
Identify friction points, emerging needs, product gaps, and opportunities to improve AI-powered customer experiences.
Partner with Product Managers and CX stakeholders to translate data-driven insights into roadmap recommendations, feature concepts, and experimentation priorities.
Support product discovery through analysis of customer feedback, behavioral data, qualitative research, competitive trends, and emerging AI capabilities.
Design and evaluate experiments or A/B tests related to prompts, workflows, retrieval strategies, response patterns, and feature changes.

Agile Product Delivery & Cross-Functional Partnership 25%
Lead or participate in the gathering, analysis, and documentation of business requirements, user stories, acceptance criteria, use cases, process flows, and technical dependencies for AI product enhancements.
Participate in Agile ceremonies including backlog refinement, sprint planning, daily syncs, demos, sprint reviews, and retrospectives.
Support UAT, release validation, and post-release monitoring for prompt updates, indexed content changes, retrieval logic changes, model updates, workflow enhancements, and dashboard/reporting releases.
Partner with Product Owners, AI engineers, data teams, QA, CX stakeholders, vendors, and business leaders to ensure AI products meet customer, operational, and business requirements.
Translate technical AI behavior and data findings into clear business implications for technical and non-technical stakeholders.
Support change management, user support, training materials, and stakeholder communications related to AI product performance and enhancements.

WHAT S NEEDED
Minimum Educational Qualifications
BA/BS degree in Business, Economics, Marketing, Analytics, Statistics, Information Science, Computer Science, Cognitive Science, Data Science, or a related field; or equivalent professional experience supported by relevant subject matter training.

Minimum Experience
3+ years of experience in business analysis, product analytics, data analytics, product operations, AI operations, or related analytical roles.
Experience supporting digital products, customer-facing applications, conversational AI, AI agents, chatbots, or generative AI systems.
Experience defining KPIs, measuring product performance, building dashboards, and communicating trends, risks, and opportunities to stakeholders.
Experience analyzing user behavior, usage data, conversation logs, customer journeys, operational metrics, or product performance data.
Good understanding of generative AI applications, including LLMs, prompt-based systems, conversational AI, AI agents, and retrieval-augmented generation.
Familiarity with AI evaluation concepts such as groundedness, relevance, factuality, hallucination risk, retrieval quality, response accuracy, human evaluation, and LLM-as-a-judge methods. Knowledge of RAGAS or similar frameworks helpful.
Experience with or knowledge of LLM observability platforms (e.g., LangSmith, Arize, etc.) beneficial but not required.
Technical & Tool Skills
Experience with SQL or similar query languages for data extraction and analysis. Knowledge of Python and Pandas helpful.
Strong Excel skills, including creation of automations.
Experience with Power BI, Tableau, Adobe Analytics, or similar analytics and visualization tools.
Experience working with content metadata, taxonomies, or categorical data.
Experience building dashboards, recurring reports, operational scorecards, and performance monitoring views.
Familiarity with AWS technologies beneficial, especially DynamoDB and CloudWatch.
Desired Traits
Strong data, analytical, and technical aptitude.
Excellent product sense and ability to connect user behavior, AI system performance, and business outcomes.
Solid understanding of which data is needed to understand how the product is used, and uses data to understand how customers use the product.
Strong at independently identifying insights from data and the UX to refine and innovate the product, makes recommendations to Product Owner based on data.
Rigorous attention to detail and strong judgment around quality, customer experience, and brand risk.
Curiosity about why AI systems behave the way they do and persistence in diagnosing root causes.
Ability to tell clear stories with data and make complex AI performance topics accessible to non-technical audiences.
Comfortable working in ambiguity and operating in a live, evolving production environment.
Self-motivated, execution-oriented, and able to manage multiple stakeholders and competing priorities.

Key Performance Indicators
Timeliness and usefulness of dashboards, reports, scorecards, and business reviews.
Effectiveness of AI evaluation frameworks, test sets, rubrics, and monitoring processes.
Ability to detect, diagnose, and communicate quality degradation, drift, hallucination risk, retrieval issues, or emerging failure patterns.
Clarity and completeness of requirements, user stories, acceptance criteria, and analysis artifacts.
Stakeholder satisfaction with analytical support, insight synthesis, and product recommendations.

Communications & Working Relationships
Internal Contacts
CX AI Product Owner and Product Team: Support product performance analysis, backlog refinement, requirements definition, UAT, release validation, dashboarding, and roadmap recommendations.
Engineering, Data, and QA Teams: Collaborate on evaluation methods, instrumentation, data availability, product behavior analysis, release validation, defect diagnosis, and performance monitoring.
Customer Experience Business and Functional Teams: Identify business needs, communicate AI product performance, gather feedback, and support adoption of AI-powered customer experience improvements.
Digital Product Planning, CX Leadership, and Division/Department Managers: Provide executive-ready insights, business impact analysis, decision support, demos, and recommendations.
Operations and Support Teams: Understand production issues, customer experience impacts, escalation patterns, and improvement opportunities.
External Contacts
Application Managed Support teams, including onshore and offshore partners.
External consultants, agencies, vendors, and technology partners supporting AI, analytics, cloud infrastructure, or product delivery.

Job Dimensions
Number of Direct Reports: 0
Number of Indirect Reports: Peer team model; works closely with 5 7 Agile scrum team members and cross-functional business/technical partners.

Financial Dimensions
Align AI product KPIs to organizational objectives and key results.
Establish and track metrics that demonstrate impact to customer experience, operational efficiency, retention, revenue, cost avoidance, and business value.
Support the Product Owner responsible for innovation, optimization, and ongoing maintenance of AI-powered digital products.
Support executive approval, governance, and deployment of product increments through clear business cases, impact analysis, and performance measurement.

Decisions Expected
Recommend AI product KPI definitions, dashboard views, evaluation metrics, and monitoring priorities.
Recommend backlog priorities based on production AI performance, customer impact, business value, quality risk, and stakeholder needs.
Identify and escalate critical AI quality issues, hallucination risks, retrieval gaps, drift, or customer experience concerns.
Recommend product, prompt, RAG, workflow, or defect remediation actions based on data analysis and evaluation findings.
Recommend experiment designs, release validation approaches, and post-release monitoring plans.

Competencies
Communicating with Impact and Influence
Creating Teamwork and Valuing Relationships
Critical and Innovative Thinking
Developing Self and Others
Purpose Driven
Results Orientation
Technical and Business Acumen

COMPENSATION & BENEFITS
" Medical, dental, and vision insurance
" 401(k)
" Other standard benefits as applicable

ABOUT UNICON
UNICON is a workforce solutions provider with over 35 years of experience supporting organizations across a variety of industries. We partner with our clients to deliver skilled professionals while maintaining a strong focus on quality, compliance, and long-term relationships. UNICON is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or veteran status.

About the Company

U

UNICON International, Inc.