AI Quality Engineer/Operator

Recruiting From Scratch

  • San Francisco, California
  • 30+ days ago
  • Remote

    Highlights

    If you enjoy operational rigor, debugging complex systems, improving AI reliability, and working closely with cutting-edge AI infrastructure in a fast-scaling startup, this role offers exceptional ownership and growth. This is an opportunity to join a rapidly scaling AI healthcare startup at an early stage, where the team has already secured major enterprise contracts and is operating at the frontier of agentic AI systems in regulated healthcare environments.

    Numbers & Facts

    LocationSan Francisco, California (
    Remote
    )
    Websiterecruitingfromscratch.com

    Description

     
    Recruiting from Scratch is a premier talent firm that focuses on placing the best product managers, software, and hardware talent at innovative companies. Our team is 100% remote and we work with teams across the United States to help them hire.

    AI Quality Engineer / Operator

    Location: San Francisco, CA (On-site)
    Company Stage of Funding: Seed
    Office Type: On-site
    Salary: $93,000 – $186,000 + Equity (0.06% – 0.36%)
    Visa: No sponsorship (TN considered)

    Company Description

    Our client is building enterprise-grade AI automation systems for healthcare operations, focused on accelerating patient access to critical treatments and specialty medications.

    Their platform automates workflows including benefit verification, prior authorization, patient onboarding, financial assistance enrollment, and adherence management using AI agents deployed in real enterprise healthcare environments.

    This is an opportunity to join a rapidly scaling AI healthcare startup at an early stage, where the team has already secured major enterprise contracts and is operating at the frontier of agentic AI systems in regulated healthcare environments.

    The company is highly execution-focused, profitable, and building foundational infrastructure for enterprise AI operations in healthcare.

    What You Will Do

    Review and debug AI agent outputs across transcripts, workflows, recordings, and operational systems

    Identify quality issues, edge cases, and failure patterns in production AI systems

    Write, refine, and test prompts for conversational AI and agent workflows

    Run systematic root-cause analysis and quality investigations

    Document issues rigorously with clear categorization, reproduction steps, and escalation context

    Build and maintain lightweight scripts, regex patterns, and automation workflows using AI-assisted coding tools

    Operate continuous quality improvement loops for production AI agents

    Validate fixes and ensure issue resolution across workflows end-to-end

    Collaborate closely with engineers, operations, customer teams, and leadership

    Work directly inside IDEs and AI tooling environments

    Evaluate LLM outputs critically rather than relying blindly on AI-generated responses

    Improve operational reliability and quality processes for enterprise healthcare AI systems

    Contribute to the development of scalable AI quality infrastructure

    Operate in a highly ambiguous, fast-moving startup environment with strong ownership expectations

    Ideal Candidate Background

    0–7 years of experience in QA engineering, operations, legal, technical program management, AI operations, or related precision-oriented roles

    Strong written communication and analytical thinking skills

    Experience working with AI systems, LLMs, prompt engineering, or conversational AI

    Comfortable working inside IDEs and technical tooling environments

    Ability to learn lightweight scripting and debugging workflows quickly

    Strong attention to detail and operational rigor

    Experience debugging complex workflows or systems

    Ability to think systematically about root causes and edge cases

    Comfortable operating in ambiguous startup environments

    Strong learning mindset and technical curiosity

    Able to work autonomously with minimal process structure

    Interest in AI systems, healthcare operations, or enterprise automation

    Strong Signals

    Experience with LLMs, AI agents, prompt engineering, or AI evaluations

    QA engineering or quality operations experience

    Strong written communication and documentation ability

    Experience with regex, scripting, or lightweight automation

    Healthcare, biopharma, or regulated-industry experience

    Operations-heavy startup background

    Legal, compliance, or analytical writing backgrounds

    Strong debugging and root-cause analysis mindset

    Experience reviewing AI outputs at scale

    Technical curiosity combined with operational rigor

    Comfortable learning new tools independently

    Ability to critically evaluate AI-generated outputs

    High attention to detail under pressure

    Strong execution orientation

    Compensation and Benefits

    Base salary: $93,000 – $186,000

    Equity package: 0.06% – 0.36%

    Founding-stage ownership and responsibility

    Direct collaboration with founders and leadership

    High-impact role shaping production AI quality systems

    Exposure to frontier AI infrastructure and healthcare automation

    Rapid career growth opportunity inside an early-stage startup

    Fast-moving and highly collaborative environment

    Why Join

    This is an opportunity to work at the intersection of AI agents, healthcare operations, and enterprise automation.

    You’ll help build and operate quality systems for production AI agents handling critical healthcare workflows in real enterprise environments.

    If you enjoy operational rigor, debugging complex systems, improving AI reliability, and working closely with cutting-edge AI infrastructure in a fast-scaling startup, this role offers exceptional ownership and growth.

    Similar Jobs